Historical TickHistory Data Download
We have 4200 RICs for which we want to download the tickhistory(all 364 fields) going back to 12 years. Our requirement is that the downloaded compressed file size can be of maximum 1.5GB/file but again we do not want too many (100K+) small files for our requirement. We cannot do this manually using Datascope web UI hence we are looking at REST API option.
I did some calculation 1. For 1RIC/1-day tickhistory, we need to make 12.6 million request. This will be 12.6 M files and too many requests which is not desirable. 2. For 1RIC/1-year tickhistory, we need to make 50,400 requests.How do we schedule - for each RIC and for each year, schedule a tickhistory extraction request. I also have come to know that at maximum of 50 REST calls can be made at a time, so please suggest the best strategy given that we do not want compressed file size of >1.5GB.
Best Answer
Hi @vboddula
About your requirement, what if you use Byte serving, would that be ok?
Basically, you just make big requests, i.e. 1RIC/12-years. The total file size would be bigger than 1.5GB. But instead of download the whole file in one go, you just download a portion of the file at a time and then combine them to make the whole file.
Would that be a viable solution?