Optimize data pull

Related to a previous question, I am trying to pull historic data for a series of tickers.

Would it more efficient to query one ticker at a time for n years of history, or a series of tickers for a given day/week/month? Or is there some other way that would be most efficient to query a large amount of historic data?

Find more posts tagged with

datascope-select

datastream-api

Accepted answers

Jirapongse

@XINGTIAN.XU

You can refer to the Best Practices & Fair Usage Policy for DataScope Select and Tick History guide.

It mentions "Submitting a single job that reports on multiple instruments is faster than submitting several jobs that each report on one instrument.".

Moreover, there are several limits when you are using RTH and DSS, as mentioned in the guide. For example:

The number of concurrent extractions that can be processed on a per-report and per-user basis is two for all report templates except Intraday Pricing, Historical Reference, Elektron Timeseries report templates.

I have done a quick test by using the TickHistoryRawExtractionRequest to extract historical data from 2015 to 2022. For one RIC, it took 525.744 secs. For three RICs, it took 1335.32 secs. However, it depends on the server load at that time.

You can check the processing time from the Extraction Notes when the extraction was complete.

All comments