question

Upvotes
Accepted
1 0 0 1

How to resolve RTH API - Historical criteria search - Regex codes errors

Hi Team,

Client is unable to get data in Tick history API Historical criteria search using below regex codes. Can you help correcting if there is any mistake.

^CMCU[FGHJKMNQUVXZ][0-9]$|TYO|FUT

^CMAL[FGHJKMNQUVXZ][0-9]$|TYO|FUT

^CMNI[FGHJKMNQUVXZ][0-9]$|TYO|FUT

^CMZN[FGHJKMNQUVXZ][0-9]$|TYO|FUT

Example for CMAL.

2022-05-19 00:35:50 INFO: https://selectapi.datascope.refinitiv.com/RestApi/v1/Search/HistoricalCriteriaSearch

2022-05-19 00:35:51 INFO: Using standard symbol parser

2022-05-19 00:35:51 INFO: Building report config from: /home/sb/arbtick/release/q/tickHistory/templates/hk_futures/instrumentSearch.template

{

"Request":

{ "RicPattern": "^CMAL[FGHJKMNQUVXZ][0-9]$",

"BondTypeCodes": null,

"ContributorIds": null,

"CountryCodes": null,

"CurrencyCodes": null,

"ExchangeCodes": [],

"FutureMonthCodes": null,

"InstrumentTypeCodes": ["114"],

"OptionMonthCodes": null,

"OptionTypeCodes": null,

"CouponRate": null,

"StrikePrice": null,

"ExpiryDate": null,

"MaturityDate": null,

"Range": {

"Start": "2022-05-17T16:00:00.000Z",

"End": "2022-05-18T15:59:59.999Z"

}

}

}


@manikandan.bhachumohan@citi.com - FYI

tick-history-rest-api
icon clock
10 |1500

Up to 2 attachments (including images) can be used with a maximum of 512.0 KiB each and 1.0 MiB total.

Hello @shyamala.k0 ,

Historical Criteria Search regex implements limited regex matching- it is not the complete regex spec.

It would be helpful to share an example of one or two RICs, that the client and you are looking to match?

Please also see this previous discussion thread for the approach ideas.

Upvotes
Accepted
32.2k 39 11 19

Hello @shyamala.k0 ,

Let us consider the first of the example - CMCU

In your Postman, please run the pattern as I have suggested:

{
"Request":
{ "RicPattern": "^CMCU[F-Z]{1}[0-9]{2}$",
"BondTypeCodes": null,
"ContributorIds": null,
"CountryCodes": null,
"CurrencyCodes": null,
"ExchangeCodes": null,
"FutureMonthCodes": null,
"InstrumentTypeCodes": null,
"OptionMonthCodes": null,
"OptionTypeCodes": null,
"CouponRate": null,
"StrikePrice": null,
"ExpiryDate": null,
"MaturityDate": null,
"Range": {
"Start": "2022-05-17T00:00:00.000Z",
"End": "2022-05-18T23:59:59.999Z"
}
}
}

Please observe CMCU instruments in the results, do you see them?

If they are getting CMCU back, but dates are nor working, please widen the Start and End dates, as we are out of April 22 now and out of May as well.

However, with their pattern, I do not get these results on my side either, please verify on your side?


icon clock
10 |1500

Up to 2 attachments (including images) can be used with a maximum of 512.0 KiB each and 1.0 MiB total.

Hi @zoya faberov , using the above API code, I was able to get the results. And I had suggested the same API code to client. Initially client cameback stating it is working for the RICs they are looking for.


Now they cameback with the below follow up question:


"Thanks for helping out.

Just to confirm, should use the pattern [F-Z]{1}[0-9]{2}$ instead of [FGHJKMNQUVXZ][0-9]$

For example : ^CMCU[F-Z]{1}[0-9]{2}$|TYO|FUT instead of ^CMCU[FGHJKMNQUVXZ][0-9]$|TYO|FUT

If so, should we apply the new pattern for the attached list please?"

Please advise further.


Regards,

Shyamala K

Hello @shyamala.k0 ,

I suggest, to cover the requirement as you have discussed completely, to use 4 requests, with patterns:

"RicPattern": "^CMAL[F-Z]{1}[0-9]{2}$"
 "RicPattern": "^CMCU[F-Z]{1}[0-9]{2}$"
 "RicPattern": "^CMNI[F-Z]{1}[0-9]{2}$"
 "RicPattern": "^CMZN[F-Z]{1}[0-9]{2}$"

And to combine the results from the 4 requests.

Please note, that ors ("|TYO|FUT ") that client has introduced intro regex do not concatenate to the regex that is defined before, rather, they replace the previously defined regex, as in "either or". If this is ok with them- this is up to them, but from the requirement that you have shared with us- they are not looking for this to happen.

Upvotes
32.2k 39 11 19

Hello @shyamala.k0 ,

If you just looking for "unary" instruments like "CMALF23" please try

{
                {protocol}}{
                {host}}{
                {api}}Search/HistoricalCriteriaSearch 
{
"Request":
{ "RicPattern": "^CMAL[F-Z]{1}[0-9]{2}$",
"BondTypeCodes": null,
"ContributorIds": null,
"CountryCodes": null,
"CurrencyCodes": null,
"ExchangeCodes": null,
"FutureMonthCodes": null,
"InstrumentTypeCodes": null,
"OptionMonthCodes": null,
"OptionTypeCodes": null,
"CouponRate": null,
"StrikePrice": null,
"ExpiryDate": null,
"MaturityDate": null,
"Range": {
"Start": "2022-05-17T00:00:00.000Z",
"End": "2022-05-18T23:59:59.999Z"
}
}
}

, if for ranges as well, for example "CMALZ24-63PD", which I notice is also a valid instrument, try:

{
                 {protocol}}{
                 {host}}{
                 {api}}Search/HistoricalCriteriaSearch 
{
"Request":
{ "RicPattern": "^CMAL[F-Z]{1}[0-9]{2}",
"BondTypeCodes": null,
"ContributorIds": null,
"CountryCodes": null,
"CurrencyCodes": null,
"ExchangeCodes": null,
"FutureMonthCodes": null,
"InstrumentTypeCodes": null,
"OptionMonthCodes": null,
"OptionTypeCodes": null,
"CouponRate": null,
"StrikePrice": null,
"ExpiryDate": null,
"MaturityDate": null,
"Range": {
"Start": "2022-05-17T00:00:00.000Z",
"End": "2022-05-18T23:59:59.999Z"
}
}
}

Let me know if this is what you are looking for, otherwise please share a couple of example RICs that would match the desired criteria?

icon clock
10 |1500

Up to 2 attachments (including images) can be used with a maximum of 512.0 KiB each and 1.0 MiB total.

Upvotes
32.2k 39 11 19

Hello @shyamala.k0 ,

Having confirmed that you are looking for singular instruments, for example "CMCUM22", and because HistoricalCriteriaSearch implements limited regex, I would run 4 separate RicPattern searches:

 "RicPattern": "^CMAL[F-Z]{1}[0-9]{2}$"
 "RicPattern": "^CMCU[F-Z]{1}[0-9]{2}$"
 "RicPattern": "^CMNI[F-Z]{1}[0-9]{2}$"
 "RicPattern": "^CMZN[F-Z]{1}[0-9]{2}$"

Let us know how this works for you



icon clock
10 |1500

Up to 2 attachments (including images) can be used with a maximum of 512.0 KiB each and 1.0 MiB total.

Hello @zoya faberov ,

Client has comeback stating that "We tried to use the below recommended codes, however we could not receive the required data set.

^CMAL[F-Z]{1}[0-9]{2}$[0-9]$|TYO|FUT

^CMCU[F-Z]{1}[0-9]{2}$[0-9]$|TYO|FUT

^CMNI[F-Z]{1}[0-9]{2}$[0-9]$|TYO|FUT

^CMZN[F-Z]{1}[0-9]{2}$[0-9]$|TYO|FUT

^CMPB[F-Z]{1}[0-9]{2}$[0-9]$|TYO|FUT

Sample RIC prefix we find missing:

Ticker

RIC prefix

ISIN

Name

Description

Exchange

Type

Currency

LPA Comdty

CMCU

GB00DV8VP374

LME COPPER FUTURE Apr22

Copper

LME 3rd Wednesday Prices & Options:LMF

Future

USD

LAA Comdty

CMAL

GB00DV8PQ366

LME PRI ALUM FUTR Apr22

Primary Aluminum

LME 3rd Wednesday Prices & Options:LMF

Future

USD

LNA Comdty

CMNI

GB00GRXTCH88

LME NICKEL FUTURE Apr22

Nickel

LME 3rd Wednesday Prices & Options:LMF

Future

USD

LXA Comdty

CMZN

GB00GRY1R126

LME ZINC FUTURE Apr22

Zinc

LME 3rd Wednesday Prices & Options:LMF

Future

USD

SCOA Comdty

SZZF

SGXDB0017614

SGX Iron Ore 62% May22

SGX TSI Iron Ore CFR China (62% Fe Fines) Index Futures

Singapore Exchange (was SIMEX):SGX

Future

USD

LLA Comdty

CMPB

GB00GRXTCG71

LME LEAD FUTURE Apr22

Lead

LME 3rd Wednesday Prices & Options:LMF

Future

USD

Kindly advice."


Request you to look into the above issue and advise.


Regards,

Shyamala K

Hi @zoya faberov ,


Please find the attached screenshot for details on "Sample RIC prefix we find missing"

Regards,

Shyamala KSample RIC prefix.p

Upvotes

subscriptionListWithReceivedRICs_new.txt

We were able to get only 148 symbols for the date 6th of Jun, with the suggested pattern (Example "RicPattern": "^LCO[F-Z]{1}[0-9]{2}$")

We gave another round of try, now combining both the pattern for 7th of Jun 2022 and we are able to get bigger range of products(Example : ^LCO[FGHJKMNQUVXZ][0-9]$|TYO|FUT and ^LCO[F-Z]{1}[0-9]{2}$


icon clock
10 |1500

Up to 2 attachments (including images) can be used with a maximum of 512.0 KiB each and 1.0 MiB total.

@zoya faberov - Hi Zoya,


Client has directly posted the follow up query on this issue above. Request you to look into the issue and advise. Thanks


Client's query again for reference:

"We were able to get only 148 symbols for the date 6th of Jun, with the suggested pattern – attached details.

Are we missing something?

Should we be combining both the patterns?"


@ashok.kumar.murahari@citi.com - fyi


Regards,

Shyamala K


Any updates on this please?

@zoya faberov , Hi Zoya,


The above request is pending from a long time, request you to prioritize and provide an update at the earliest.

Please note client is chasing for updates.


Regards,

Shyamala K

@ashok.kumar.murahari@citi.com - FYI

Upvotes
32.2k 39 11 19

Hello @shyamala.k0 and @ashok.kumar.murahari.1 ,

Apologies for the belated response, I was out of the office.

You may have had a chance to observe, that if you apply criteria as:

{
"Request":
{ "RicPattern": "^LCO[FGHJKMNQUVXZ][0-9]$|TYO|FUT",
"BondTypeCodes": null,
"ContributorIds": null,
"CountryCodes": null,
"CurrencyCodes": null,
"ExchangeCodes": null,
"FutureMonthCodes": null,
"InstrumentTypeCodes": null,
"OptionMonthCodes": null,
"OptionTypeCodes": null,
"CouponRate": null,
"StrikePrice": null,
"ExpiryDate": null,
"MaturityDate": null,
"Range": {
"Start": "2022-05-17T00:00:00.000Z",
"End": "2022-05-18T23:59:59.999Z"
}
}
}

You get more hits, but please examine the results, are these the hits that you require:

{
    "@odata.context": "https://selectapi.datascope.refinitiv.com/RestApi/v1/$metadata#Collection(DataScope.Select.Api.Search.HistoricalSearchResult)",
    "value": [
        {
            "Identifier": "FUT",
            "IdentifierType": "Ric",
            "Source": "",
            "Key": "VjF8MHgzMDAwMDAwMDAwMDAwMDAwfDB4MzAwMDAwMDAwMDAwMDAwMHx8fHx8fHxGVVR8",
            "Description": "Historical Instrument",
            "InstrumentType": "Unknown",
            "Status": "Valid",
            "DomainCode": "6",
            "FirstDate": "2004-03-30T00:00:00.000Z",
            "LastDate": "2022-05-03T00:00:00.000Z",
            "History": []
        },
        {
            "Identifier": "FUT.A",
            "IdentifierType": "Ric",

I believe they are not.

If you are looking to retrieve results as per requirement communicated by @shyamala.k0 , I suggest running 4 requests with patterns as above, for clarity, and next, combining the 4 results. This should produce the results per your requirements.

The regex that is supported by service is not the full regex spec . In my observation, "|" inside stanzas is not fully supported, and when used, leads to random results. Which is why I would not combine the patterns in the request, via "|", but combine the results.

I hope that this information helps.

I would not be able to confirm why there were only 148 results on a specific date. Refintiiv content experts will be able to help with this type of question about content, and if posted to Refinitiv Helpdesk Online, the expert for type of content in question will be assigned to help.

icon clock
10 |1500

Up to 2 attachments (including images) can be used with a maximum of 512.0 KiB each and 1.0 MiB total.