question

Upvotes
Accepted
5 0 1 2

Optimize data pull

Related to a previous question, I am trying to pull historic data for a series of tickers.

Closing auction imbalance data via API

Would it more efficient to query one ticker at a time for n years of history, or a series of tickers for a given day/week/month? Or is there some other way that would be most efficient to query a large amount of historic data?

datascope-selectdatastream-api
icon clock
10 |1500

Up to 2 attachments (including images) can be used with a maximum of 512.0 KiB each and 1.0 MiB total.

1 Answer

· Write an Answer
Upvotes
Accepted
81.1k 264 53 76

@XINGTIAN.XU

You can refer to the Best Practices & Fair Usage Policy for DataScope Select and Tick History guide.

It mentions "Submitting a single job that reports on multiple instruments is faster than submitting several jobs that each report on one instrument.".

Moreover, there are several limits when you are using RTH and DSS, as mentioned in the guide. For example:

The number of concurrent extractions that can be processed on a per-report and per-user basis is two for all report templates except Intraday Pricing, Historical Reference, Elektron Timeseries report templates.

I have done a quick test by using the TickHistoryRawExtractionRequest to extract historical data from 2015 to 2022. For one RIC, it took 525.744 secs. For three RICs, it took 1335.32 secs. However, it depends on the server load at that time.

You can check the processing time from the Extraction Notes when the extraction was complete.

icon clock
10 |1500

Up to 2 attachments (including images) can be used with a maximum of 512.0 KiB each and 1.0 MiB total.

Write an Answer

Hint: Notify or tag a user in this post by typing @username.

Up to 2 attachments (including images) can be used with a maximum of 512.0 KiB each and 1.0 MiB total.