429 error handling

davidk

The LSEG historical pricing API (via either lseg.data.content.historical_pricing or lseg.data.get_history) appears to request all securities at once. This means that for large universes we are prone to generate 429 errors, since the backend is rate limited. It does not appear that the API has any handling for this nor any internal rate limiting?

When I run a simple request for e.g. the Russell 2000 and call get_history, numerous of the stocks are simply silently empty. However, if I call lseg.data.content.historical_pricing, I can see that the resulting response.errors object has hundreds of 429s. My current workaround for this involves parsing the 429 errors: see below.

Questions:

1) Is this the best workaround here? I suppose another option would be me individually making requests with the async API and writing my own rate limiter class, and then merging the DataFrames that result?

2) I'm sort of surprised that one function call of mine to the API leads the API to in turn do enough requests to generate unhandled 429s. I'd have thought the API would handle any internal rate limiting. Am I potentially doing something wrong in my definition?

3) When I request data using definition.get_data() and receive hundreds of 429s: some of those stocks that gave back 429s ALSO have non-null data associated with them. What does that mean and how am I to interpret that? Should I throw away that data or keep it?

Current Workaround:

def get_history_workaround(universe: list, **definition_kwargs):
    definition = hp.Definition(universe=universe, **definition_kwargs)
    data = pd.DataFrame()
    return_errors = []
    ric_pattern = re.compile(r'Requested ric: (\S+)\. Requested fields:')
    while universe:
        response = definition.get_data()
        if data.empty:
            data = response.data.df
        else:
            for ric in universe:
                # Since we are assigning into a DataFrame, everything should have the same index
                # and we throw away additional dates from the smaller requests
                data[ric] = response.data.df[ric]
                
        universe = []
        for error in response.data.errors:
            if error.code == 429 and (match := ric_pattern.search(error.message)):
                ric = match.group(1)
                universe.append(ric)
                del data[ric]
            else:
                return_errors.append(error)
                
        if universe:
            print(f'Received 429 for {len(universe)} RICs. Retrying...')
            definition = hp.Definition(universe=universe, **definition_kwargs)
            
    return data, return_errors

Jirapongse

@davidk

Thank you for reaching out to us.

Under the hood, the Historical Pricing APIs support one RIC at a time. Therefore, if the universe contains 1000 items, it will send 1000 requests to get historical data.

I checked the code and found that the library uses the ThreadPoolExecutor to manage those requests. I think this can trigger the "429 Too Many Requests" error when requesting a lot of items. The rate limit of the interday historical pricing is 50 requests per second.

According to this limit, you may need to request (calling the get_data method) at most 50 items at a time.

Because the endpoint supports one RIC at a time, you can keep the data if the request is successful.