How to resolve RTH API - Historical criteria search - Regex codes errors

Options

Hi Team,

Client is unable to get data in Tick history API Historical criteria search using below regex codes. Can you help correcting if there is any mistake.

^CMCU[FGHJKMNQUVXZ][0-9]$|TYO|FUT

^CMAL[FGHJKMNQUVXZ][0-9]$|TYO|FUT

^CMNI[FGHJKMNQUVXZ][0-9]$|TYO|FUT

^CMZN[FGHJKMNQUVXZ][0-9]$|TYO|FUT

Example for CMAL.

2022-05-19 00:35:50 INFO: https://selectapi.datascope.refinitiv.com/RestApi/v1/Search/HistoricalCriteriaSearch

2022-05-19 00:35:51 INFO: Using standard symbol parser

2022-05-19 00:35:51 INFO: Building report config from: /home/sb/arbtick/release/q/tickHistory/templates/hk_futures/instrumentSearch.template

{

"Request":

{ "RicPattern": "^CMAL[FGHJKMNQUVXZ][0-9]$",

"BondTypeCodes": null,

"ContributorIds": null,

"CountryCodes": null,

"CurrencyCodes": null,

"ExchangeCodes": [],

"FutureMonthCodes": null,

"InstrumentTypeCodes": ["114"],

"OptionMonthCodes": null,

"OptionTypeCodes": null,

"CouponRate": null,

"StrikePrice": null,

"ExpiryDate": null,

"MaturityDate": null,

"Range": {

"Start": "2022-05-17T16:00:00.000Z",

"End": "2022-05-18T15:59:59.999Z"

}

}

}


manikandan.bhachumohan@citi.com - FYI

Best Answer

  • zoya faberov
    zoya faberov ✭✭✭✭✭
    Answer ✓

    Hello @shyamala.k0 ,

    Let us consider the first of the example - CMCU

    In your Postman, please run the pattern as I have suggested:

    {
    "Request":
    { "RicPattern": "^CMCU[F-Z]{1}[0-9]{2}$",
    "BondTypeCodes": null,
    "ContributorIds": null,
    "CountryCodes": null,
    "CurrencyCodes": null,
    "ExchangeCodes": null,
    "FutureMonthCodes": null,
    "InstrumentTypeCodes": null,
    "OptionMonthCodes": null,
    "OptionTypeCodes": null,
    "CouponRate": null,
    "StrikePrice": null,
    "ExpiryDate": null,
    "MaturityDate": null,
    "Range": {
    "Start": "2022-05-17T00:00:00.000Z",
    "End": "2022-05-18T23:59:59.999Z"
    }
    }
    }

    Please observe CMCU instruments in the results, do you see them?

    If they are getting CMCU back, but dates are nor working, please widen the Start and End dates, as we are out of April 22 now and out of May as well.

    However, with their pattern, I do not get these results on my side either, please verify on your side?


Answers

  • zoya faberov
    zoya faberov ✭✭✭✭✭

    Hello @shyamala.k0 ,

    Historical Criteria Search regex implements limited regex matching- it is not the complete regex spec.

    It would be helpful to share an example of one or two RICs, that the client and you are looking to match?

    Please also see this previous discussion thread for the approach ideas.

  • zoya faberov
    zoya faberov ✭✭✭✭✭

    Hello @shyamala.k0 ,

    If you just looking for "unary" instruments like "CMALF23" please try

    {{protocol}}{{host}}{{api}}Search/HistoricalCriteriaSearch 
    {
    "Request":
    { "RicPattern": "^CMAL[F-Z]{1}[0-9]{2}$",
    "BondTypeCodes": null,
    "ContributorIds": null,
    "CountryCodes": null,
    "CurrencyCodes": null,
    "ExchangeCodes": null,
    "FutureMonthCodes": null,
    "InstrumentTypeCodes": null,
    "OptionMonthCodes": null,
    "OptionTypeCodes": null,
    "CouponRate": null,
    "StrikePrice": null,
    "ExpiryDate": null,
    "MaturityDate": null,
    "Range": {
    "Start": "2022-05-17T00:00:00.000Z",
    "End": "2022-05-18T23:59:59.999Z"
    }
    }
    }

    , if for ranges as well, for example "CMALZ24-63PD", which I notice is also a valid instrument, try:

    {{protocol}}{{host}}{{api}}Search/HistoricalCriteriaSearch 
    {
    "Request":
    { "RicPattern": "^CMAL[F-Z]{1}[0-9]{2}",
    "BondTypeCodes": null,
    "ContributorIds": null,
    "CountryCodes": null,
    "CurrencyCodes": null,
    "ExchangeCodes": null,
    "FutureMonthCodes": null,
    "InstrumentTypeCodes": null,
    "OptionMonthCodes": null,
    "OptionTypeCodes": null,
    "CouponRate": null,
    "StrikePrice": null,
    "ExpiryDate": null,
    "MaturityDate": null,
    "Range": {
    "Start": "2022-05-17T00:00:00.000Z",
    "End": "2022-05-18T23:59:59.999Z"
    }
    }
    }

    Let me know if this is what you are looking for, otherwise please share a couple of example RICs that would match the desired criteria?

  • zoya faberov
    zoya faberov ✭✭✭✭✭

    Hello @shyamala.k0 ,

    Having confirmed that you are looking for singular instruments, for example "CMCUM22", and because HistoricalCriteriaSearch implements limited regex, I would run 4 separate RicPattern searches:

     "RicPattern": "^CMAL[F-Z]{1}[0-9]{2}$"
    "RicPattern": "^CMCU[F-Z]{1}[0-9]{2}$"
    "RicPattern": "^CMNI[F-Z]{1}[0-9]{2}$"
    "RicPattern": "^CMZN[F-Z]{1}[0-9]{2}$"

    Let us know how this works for you



  • Hello @zoya faberov ,

    Client has comeback stating that "We tried to use the below recommended codes, however we could not receive the required data set.

    ^CMAL[F-Z]{1}[0-9]{2}$[0-9]$|TYO|FUT

    ^CMCU[F-Z]{1}[0-9]{2}$[0-9]$|TYO|FUT

    ^CMNI[F-Z]{1}[0-9]{2}$[0-9]$|TYO|FUT

    ^CMZN[F-Z]{1}[0-9]{2}$[0-9]$|TYO|FUT

    ^CMPB[F-Z]{1}[0-9]{2}$[0-9]$|TYO|FUT

    Sample RIC prefix we find missing:

    Ticker

    RIC prefix

    ISIN

    Name

    Description

    Exchange

    Type

    Currency

    LPA Comdty

    CMCU

    GB00DV8VP374

    LME COPPER FUTURE Apr22

    Copper

    LME 3rd Wednesday Prices & Options:LMF

    Future

    USD

    LAA Comdty

    CMAL

    GB00DV8PQ366

    LME PRI ALUM FUTR Apr22

    Primary Aluminum

    LME 3rd Wednesday Prices & Options:LMF

    Future

    USD

    LNA Comdty

    CMNI

    GB00GRXTCH88

    LME NICKEL FUTURE Apr22

    Nickel

    LME 3rd Wednesday Prices & Options:LMF

    Future

    USD

    LXA Comdty

    CMZN

    GB00GRY1R126

    LME ZINC FUTURE Apr22

    Zinc

    LME 3rd Wednesday Prices & Options:LMF

    Future

    USD

    SCOA Comdty

    SZZF

    SGXDB0017614

    SGX Iron Ore 62% May22

    SGX TSI Iron Ore CFR China (62% Fe Fines) Index Futures

    Singapore Exchange (was SIMEX):SGX

    Future

    USD

    LLA Comdty

    CMPB

    GB00GRXTCG71

    LME LEAD FUTURE Apr22

    Lead

    LME 3rd Wednesday Prices & Options:LMF

    Future

    USD

    Kindly advice."


    Request you to look into the above issue and advise.


    Regards,

    Shyamala K

  • Hi @zoya faberov ,


    Please find the attached screenshot for details on "Sample RIC prefix we find missing"

    Regards,

    Shyamala KSample RIC prefix.p

  • Hi @zoya faberov , using the above API code, I was able to get the results. And I had suggested the same API code to client. Initially client cameback stating it is working for the RICs they are looking for.


    Now they cameback with the below follow up question:


    "Thanks for helping out.

    Just to confirm, should use the pattern [F-Z]{1}[0-9]{2}$ instead of [FGHJKMNQUVXZ][0-9]$

    For example : ^CMCU[F-Z]{1}[0-9]{2}$|TYO|FUT instead of ^CMCU[FGHJKMNQUVXZ][0-9]$|TYO|FUT

    If so, should we apply the new pattern for the attached list please?"

    Please advise further.


    Regards,

    Shyamala K

  • zoya faberov
    zoya faberov ✭✭✭✭✭

    Hello @shyamala.k0 ,

    I suggest, to cover the requirement as you have discussed completely, to use 4 requests, with patterns:

    "RicPattern": "^CMAL[F-Z]{1}[0-9]{2}$"
    "RicPattern": "^CMCU[F-Z]{1}[0-9]{2}$"
    "RicPattern": "^CMNI[F-Z]{1}[0-9]{2}$"
    "RicPattern": "^CMZN[F-Z]{1}[0-9]{2}$"

    And to combine the results from the 4 requests.

    Please note, that ors ("|TYO|FUT ") that client has introduced intro regex do not concatenate to the regex that is defined before, rather, they replace the previously defined regex, as in "either or". If this is ok with them- this is up to them, but from the requirement that you have shared with us- they are not looking for this to happen.

  • subscriptionListWithReceivedRICs_new.txt

    We were able to get only 148 symbols for the date 6th of Jun, with the suggested pattern (Example "RicPattern": "^LCO[F-Z]{1}[0-9]{2}$")

    We gave another round of try, now combining both the pattern for 7th of Jun 2022 and we are able to get bigger range of products(Example : ^LCO[FGHJKMNQUVXZ][0-9]$|TYO|FUT and ^LCO[F-Z]{1}[0-9]{2}$

  • @zoya faberov - Hi Zoya,


    Client has directly posted the follow up query on this issue above. Request you to look into the issue and advise. Thanks


    Client's query again for reference:

    "We were able to get only 148 symbols for the date 6th of Jun, with the suggested pattern – attached details.

    Are we missing something?

    Should we be combining both the patterns?"


    @ashok.kumar.murahari@citi.com - fyi


    Regards,

    Shyamala K


  • @zoya faberov , Hi Zoya,


    The above request is pending from a long time, request you to prioritize and provide an update at the earliest.

    Please note client is chasing for updates.


    Regards,

    Shyamala K

    @ashok.kumar.murahari@citi.com - FYI

  • zoya faberov
    zoya faberov ✭✭✭✭✭

    Hello @shyamala.k0 and @ashok.kumar.murahari.1 ,

    Apologies for the belated response, I was out of the office.

    You may have had a chance to observe, that if you apply criteria as:

    {
    "Request":
    { "RicPattern": "^LCO[FGHJKMNQUVXZ][0-9]$|TYO|FUT",
    "BondTypeCodes": null,
    "ContributorIds": null,
    "CountryCodes": null,
    "CurrencyCodes": null,
    "ExchangeCodes": null,
    "FutureMonthCodes": null,
    "InstrumentTypeCodes": null,
    "OptionMonthCodes": null,
    "OptionTypeCodes": null,
    "CouponRate": null,
    "StrikePrice": null,
    "ExpiryDate": null,
    "MaturityDate": null,
    "Range": {
    "Start": "2022-05-17T00:00:00.000Z",
    "End": "2022-05-18T23:59:59.999Z"
    }
    }
    }

    You get more hits, but please examine the results, are these the hits that you require:

    {
    "@odata.context": "https://selectapi.datascope.refinitiv.com/RestApi/v1/$metadata#Collection(DataScope.Select.Api.Search.HistoricalSearchResult)",
    "value": [
    {
    "Identifier": "FUT",
    "IdentifierType": "Ric",
    "Source": "",
    "Key": "VjF8MHgzMDAwMDAwMDAwMDAwMDAwfDB4MzAwMDAwMDAwMDAwMDAwMHx8fHx8fHxGVVR8",
    "Description": "Historical Instrument",
    "InstrumentType": "Unknown",
    "Status": "Valid",
    "DomainCode": "6",
    "FirstDate": "2004-03-30T00:00:00.000Z",
    "LastDate": "2022-05-03T00:00:00.000Z",
    "History": []
    },
    {
     "Identifier": "FUT.A",
    "IdentifierType": "Ric",

    I believe they are not.

    If you are looking to retrieve results as per requirement communicated by @shyamala.k0 , I suggest running 4 requests with patterns as above, for clarity, and next, combining the 4 results. This should produce the results per your requirements.

    The regex that is supported by service is not the full regex spec . In my observation, "|" inside stanzas is not fully supported, and when used, leads to random results. Which is why I would not combine the patterns in the request, via "|", but combine the results.

    I hope that this information helps.

    I would not be able to confirm why there were only 148 results on a specific date. Refintiiv content experts will be able to help with this type of question about content, and if posted to Refinitiv Helpdesk Online, the expert for type of content in question will be assigned to help.