Retrieving bulk data

@davet1,

those are quite big requests (>1200 RICs, 20 years of data). For such large requests (many RICs and/or long time range and/or many data fields), I recommend segmenting them into more manageable ones.

In the DSS best practices we recommend making fewer queries for more RICs, rather than many queries for fewer RICs, to minimize the overhead. This works well, but there is a point where very large requests become a disadvantage. This is not documented, but thinking of it, it appears logical: every system has its limits. And in your case it seems your queries have hit a limit.

The difficulty is evaluating when a request becomes too big. This is more of an art than a science, and it will take some experimenting with your queries to determine what works best for you. As a very rough rule of thumb, if a query takes more than an hour or two to complete, it is probably too big.

For your PriceHistory and Corporate Actions requests I suggest you try one of the following approaches:

1. Request 1 instrument over 20 years. If it takes less than 5 minutes, try 10 instruments. If that takes less than 5 minutes increase the instrument count more. At the point where the wait time increases too much, stop increasing the number of instruments.
2. An alternative could be to request more instruments per request, but for a shorter time period. I'd try ranges of 1 year as a start, and apply a similar methodology as above.

I agree this is not ideal, but there is no silver bullet I'm afraid.

I hope this helps.