Efficient retrieval of broker estimates
I'm trying to retrieve historical EPS broker estimates for a number of US equities for the last several years. Conceptually, I am trying to do something like:
ek.get_data(rics, ['TR.EPSEstValue.value', 'TR.EPSEstValue.origdate', 'TR.EPSEstValue.broker_id'], parameters={'SDate':'2014-01-01', 'EDate':'2019-06-30', 'Period':'FY-6';FY2'})
Now the above query appears to be prohibitively expensive to run in one shot, and times out. But conversely, if I requested each ric/date/fiscal period individually it would be a prohibitively high number of requests. What is the best way to batch this request? For example, should I request this as a time series (perhaps one per ric), or should I request all the RICs and periods for a particular date and iterate over dates? And would it be more efficient to request multiple fiscal periods or to request those one at a time?
For future users, having now done the benchmarking I was hoping to avoid:
1) Requesting additional fiscal periods inside an IBES request does not take appreciably more time (requesting 1 fiscal year and 12 fiscal years is the same amount of time when benchmarked).
2) Adding additional dates to your request increases time by what appears to be a sub-linear factor. However, too many dates at once can cause a request to time out.
3) Adding more securities to a request appears to be roughly linear time increase.
So ultimately I settled on each request requesting 3 months of forecasts for 1 RIC for all available fiscal periods.
Hi @davidk
Please refer to this document for API limitation when retrieving a large volume of data.
The effective way is to break down the period you request for to be a smaller period.
For example rather than retrieving data for 5.5 year, you can break it down to 2 years for 3 request on a sample set of RIC codes.
Or reduce the number of RIC codes per request but keep the same period.
Or reduce both number of RIC codes and period.
There is no definitive way to tell you which one is the most effective one.
You have to trial and error on these request and make sure that it is not exceed the API limitation.