The LSEG historical pricing API (via either lseg.data.content.historical_pricing or lseg.data.get_history) appears to request all securities at once. This means that for large universes we are prone to generate 429 errors, since the backend is rate limited. It does not appear that the API has any handling for this nor any internal rate limiting?
When I run a simple request for e.g. the Russell 2000 and call get_history, numerous of the stocks are simply silently empty. However, if I call lseg.data.content.historical_pricing, I can see that the resulting response.errors object has hundreds of 429s. My current workaround for this involves parsing the 429 errors: see below.
Questions:
1) Is this the best workaround here? I suppose another option would be me individually making requests with the async API and writing my own rate limiter class, and then merging the DataFrames that result?
2) I'm sort of surprised that one function call of mine to the API leads the API to in turn do enough requests to generate unhandled 429s. I'd have thought the API would handle any internal rate limiting. Am I potentially doing something wrong in my definition?
3) When I request data using definition.get_data() and receive hundreds of 429s: some of those stocks that gave back 429s ALSO have non-null data associated with them. What does that mean and how am I to interpret that? Should I throw away that data or keep it?
Current Workaround:
def get_history_workaround(universe: list, **definition_kwargs):
definition = hp.Definition(universe=universe, **definition_kwargs)
data = pd.DataFrame()
return_errors = []
ric_pattern = re.compile(r'Requested ric: (\S+)\. Requested fields:')
while universe:
response = definition.get_data()
if data.empty:
data = response.data.df
else:
for ric in universe:
# Since we are assigning into a DataFrame, everything should have the same index
# and we throw away additional dates from the smaller requests
data[ric] = response.data.df[ric]
universe = []
for error in response.data.errors:
if error.code == 429 and (match := ric_pattern.search(error.message)):
ric = match.group(1)
universe.append(ric)
del data[ric]
else:
return_errors.append(error)
if universe:
print(f'Received 429 for {len(universe)} RICs. Retrying...')
definition = hp.Definition(universe=universe, **definition_kwargs)
return data, return_errors
