TickHistory Market-depth extraction. Retrieving only first records separated by a time interval

ricardo.siguero

Hi,

We have a huge amount of data for business activity (~12Gbytes per operating day) as a result of the request made to the API for retrieving market depth data. Because of our operational requirements, only a fraction of this data is ingested from our solution...

For the reason above, we'd like to find a way to reduce, at time of requesting data to TRTH API, the amount of data to be downloaded to our side, by providing additional settings to the related report template.

Given a market-depth data-extraction excerpt like the shown below:

#RIC,Domain,Date-Time,GMT Offset,Type,L1-BidPrice,L1-BidSize,L1-AskPrice,L1-AskSize
0168.HK,Market Price,2021-11-01T01:00:00.196001016Z,+8,Normalized LL2,72.35,6000,,
0168.HK,Market Price,2021-11-01T01:00:01.096821494Z,+8,Normalized LL2,72.35,6000,72.5,4000
0168.HK,Market Price,2021-11-01T01:00:02.275997723Z,+8,Normalized LL2,70.5,6000,70.5,80000
0168.HK,Market Price,2021-11-01T01:00:03.212531962Z,+8,Normalized LL2,70.5,6000,70.5,82000
0168.HK,Market Price,2021-11-01T01:00:14.711980183Z,+8,Normalized LL2,70.5,6000,70.5,84000
0168.HK,Market Price,2021-11-01T01:00:21.855970063Z,+8,Normalized LL2,70,6000,70,10000
0168.HK,Market Price,2021-11-01T01:00:39.672243794Z,+8,Normalized LL2,70,6000,70,12000
0168.HK,Market Price,2021-11-01T01:00:50.875939973Z,+8,Normalized LL2,70,6000,70,14000
0168.HK,Market Price,2021-11-01T01:02:13.739948591Z,+8,Normalized LL2,69.9,6000,69.9,12000
0168.HK,Market Price,2021-11-01T01:03:14.711980183Z,+8,Normalized LL2,70.5,6000,70.5,84000 

Is there a way to select records by a given time interval (e.g., extract only first record separated from each other by 2 minutes)?

#RIC,Domain,Date-Time,GMT Offset,Type,L1-BidPrice,L1-BidSize,L1-AskPrice,L1-AskSize
0168.HK,Market Price,2021-11-01T01:00:00.196001016Z,+8,Normalized LL2,72.35,6000,,
0168.HK,Market Price,2021-11-01T01:02:13.739948591Z,+8,Normalized LL2,69.9,6000,69.9,12000
0168.HK,Market Price,2021-11-01T01:03:14.711980183Z,+8,Normalized LL2,70.5,6000,70.5,84000 

The expected behaviour looks like quite similar to the one found in the Intraday data-extraction template, where the parameter intradayType (in this case, as far as I remember, only pre-defined values -e.g. "Intraday 5min"- are allowed...)

..."endDate":"2021-11-01", "endTime":"23:59:59", "fromDate":"2021-11-01", "fromTime":"00:00:00", "intradayType":"Intraday 5Min"...

Thanks, in advance, for your help

Regards

zoya faberov

Hi @ricardo.siguero ,

You are correct, TickHistoryIntradaySummaries only retrieves intraday pricing. For some realtime instruments, quotes include limited market depth information. As you were mentioning TickHistoryIntradaySummaries I have thought you may be seeing limited depth, but having tested on my side with the same example instrument, this is not the case.

If you are currently using TickHistoryMarketDepthExtraction request, this request type does not support filtering of ticks per time interval in a way that Intraday pricing does, as part of the request. In my understanding of this content set, these ticks are not equally relevant, for example, they can be of different size, different number of brokers.

You may still have to filter the results, rather then parametrize the request.

Some ideas:

You may request less levels, and this should result in a smaller result set, for example:

"NumberOfLevels": 10,

One can try with 5 levels instead.

Have you looked into requesting a different view of Market Depth::

thinking MarketByPrice or MarketByOrder, the aggregated L2 view, for example:

@{  "ExtractionRequest": {    "@odata.type": "#DataScope.Select.Api.Extractions.ExtractionRequests.TickHistoryMarketDepthExtractionRequest",    "IdentifierList": {      "@odata.type": "#DataScope.Select.Api.Extractions.ExtractionRequests.InstrumentIdentifierList",        "InstrumentIdentifiers": [{        "Identifier": "0168.HK",        "IdentifierType": "Ric"      }]    },    "Condition": {      "View": "RawMarketByPrice",      "MessageTimeStampIn": "GmtUtc",      "ReportDateRangeType": "Range",      "QueryStartDate": "2021-09-29T00:00:00.000Z",      "QueryEndDate": "2021-09-29T12:00:00.000Z",      "DisplaySourceRIC": true    }  } }

If the aggregated content satisfies your requirements.

I would suggest to additionally explore the best approach to request a filtered market depth with Refinitiv content experts, via DSS GUI. This forum can be of most help to you with API usage, with content questions, it is often best to engage Refinitiv content experts directly, by submitting your question to Refinitiv Helpdesk Online -> Content -> RTH and be connected to Refintiv expert for this specific content set. If there is a better approach, that can be identified via DSS GUI, the same request can next be submitted via API.

Hope this information helps

zoya faberov

Hello @ricardo.siguero,

I would suggest to review the fields that are available from TickHistoryIntradaySummaries extraction request, do they satisfy your requirement, please see spec:

REST API Reference Tree -> Extractions -> ExtractRaw -> Tick HistoryInstradaySummariesExtarctionRequest

Or, for the list of all available fields, run request:

{{protocol}}{{host}}{{api}}Extractions/GetValidContentFieldTypes(ReportTemplateType=DataScope.Select.Api.Extractions.ReportTemplates.ReportTemplateTypes'TickHistoryIntradaySummaries')

The next consideration, the intervals available are:

so there is no two minute interval, the closest available are FiveMinutes and FiveSeconds.

If this does not satisfy your requirement, another approach you may wish to look into is away from custom reporting and into Venue By Date standard files, please note this is a separate permission. VBDs are offered for a limited amount of time ( currently, it's last 30 days):

REST API Tutorial 2: Retrieving VBD files resulting in all data, per exchange, for one day, in a single request, and next the requirements need to be parsed out from the large file.

If VBDs are what you are looking for but, you require deeper history, the next to look at can be RDP CFS Tick History - VBD. Refinitiv Data Platform (RDP) is a separate product from Refinitiv Tick History, with HTTP REST programmatic interface, enabling access to the broad range of content sets.

ricardo.siguero

Thank you, @zoya faberov for your reply.

We'll work on the approach you suggest. It completely makes sense and seems it fits the operational requirements from our solution (the two-minute time interval was an arbitrary one, maybe not the best one for being taken as an example...)

One more time, thank you for the help provided

Regards

ricardo.siguero

Hi, again, @zoya faberov

Taking a deeper look at your reply (posted yesterday), I've realized I'm not really sure about understanding the suggested approach.

This one uses the TickHistoryIntradaySummaries report template that, correct me if I'm wrong please, should retrieve intraday princing data instead of market-depth data which is the required one from our solution (on the basis we described in our first post, i.e., only processing records with a timestamp difference, between them, never lower than 1 minute -as a difference from my post, this is the real value we want to use-)

Could you please clarify this point?

Thanks, one more time, for your help

Regards