rate limit when requesting history via the LSEG Data Library
I gather that when requesting historical pricing, every stock is being requested individually, even when requesting one date, is that correct? E.g. if I just ask for Russell 3000 stocks since the start of June, I'm making 3,000 separate requests? That at least seems to be what is happening from debug logs.
I ask partly because I believe the rate limit may be far too low if that is the case. I believe I hit the daily rate limit today, on the debug log I am getting the following error:
2025-06-11 18:18:45 httpcore.http11 DEBUG> receive_response_headers.complete return_value=(b'HTTP/1.1', 429, b'Too Many Requests', [(b'Server', b'nginx/1.17.2'), (b'Date', b'Wed, 11 Jun 2025 22:18:45 GMT'), (b'Content-Type', b'text/html; charset=utf-8'), (b'Content-Length', b'42'), (b'Connection', b'keep-alive'), (b'Access-Control-Allow-Origin', b'*'), (b'X-Request-Id', b'4417aee8-4046-43f0-892b-f38f5f1d2355'), (b'RateLimit-Remaining', b'0'), (b'VolumeLimit-Remaining', b'4967298'), (b'QueueLimit-Remaining', b'100'), (b'RateLimit-Policy', b'5;w=1, 10000;w=86400'), (b'VolumeLimit-Policy', b'50000;w=60, 5000000;w=86400'), (b'RateLimit-Resource', b'*'), (b'VolumeLimit-Resource', b'*'), (b'ETag', b'W/"2a-UpTsLJ74nYuiLgNgEwlQMxGqwrE"')])
As I understand it the "Rate Limit" in this case is the number of requests per day. From
it says we can do 10,000 requests per day or 5 per second, which is in keeping with the Rate Limit policy. However, if I am burning 3,000 requests just getting 1 day of the Russell 3K for 1 data type, that could cause me problems (for example, if I also want to get TOPIX, and to get minutely bars as well as daily data for several days). How do we get around this? Other data fields seem to allow multiple securities per request which make this more manageable.
Answers
-
Thank you for reaching out to us.
If you encounter a 429 error, it likely indicates that your current usage has exceeded the allowed limits.
I understand that you're requesting both interday and intraday historical data. Please note that the historical endpoints support only one RIC per request. Given this requirement, you may need to consult your LSEG account representative or sales team directly for further assistance.
For historical data, we have another API (Tick History) which may be more suitable for your requirements. Please discuss this with your LSEG account team or Sales team.
0 -
Hi Jirapongse— understood on the bar data.For the daily data, if I'm just getting pricing data like open, close, volume, etc., it appears I can use a single request to the get_data API, e.g.
ld.get_data(['MSFT.O', 'AAPL.O'], ['TR.OpenPrice.date', 'TR.Volume.date', 'TR.OpenPrice', 'TR.Volume'], parameters={'SDate':'2025-06-01', 'EDate':'2025-06-06'}, header_type=lseg.ld.content._header_type.HeaderType.NAME)
then it seems to use just one request. Is there any reason not to do this?0 -
The library uses different endpoints to get historical data.
For TR.xxx fields, it gets data from the fundamental and reference endpoint which supports multiple RICs in a request message.
For real-time fields (such as BID, and ASK), it uses the historical pricing API to retrieve historical data. This historical pricing API supports only one RIC in a request message.
Sorry for the confusion. You didn't share the code in the question so I understand that you may use the historical pricing API to get historical data.
0 -
Thanks Jirapongse. One other question. when I request bar data, it seems like every request counts double against our limits. To make it concrete, here is an example:
import pandas as pd
import lseg.data.content.historical_pricing as hp import lseg.data as ldld.open_session()
BAR_FIELDS = ['BID', 'ASK', 'OPEN_PRC', 'HIGH_1', 'LOW_1', 'TRDPRC_1', 'ACVOL_UNS', 'NUM_MOVES', 'OPEN_BID', 'OPEN_ASK', 'BID_NUMMOV', 'ASK_NUMMOV', 'VWAP'] start_time = pd.to_datetime('2025-06-17 03:00').tz_localize('America/Toronto').tz_convert('UTC') end_time = pd.to_datetime('2025-06-17 21:00').tz_localize('America/Toronto').tz_convert('UTC') definition = hp.summaries.Definition(universe=['TD.CCP'],interval=hp.Intervals.MINUTE,start=start_time,end=end_time,adjustments=hp.Adjustments.EXCHANGE_CORRECTION,sessions=(hp.MarketSession.PRE, hp.MarketSession.NORMAL, hp.MarketSession.POST),fields=BAR_FIELDS,) z = definition.get_data()If I turn on debug logging locally, I see two HTTP requests made:
2025-06-17 16:53:33 httpx INFO> HTTP Request: GET http://localhost:9000/api/rdp/data/historical-pricing/v1/views/intraday-summaries/TD.CCP?interval=PT1M&start=2025-06-17T07%3A00%3A00.000000000Z&end=2025-06-18T01%3A00%3A00.000000000Z&adjustments=exchangeCorrection&sessions=pre%2Cnormal%2Cpost&fields=BID%2CASK%2COPEN_PRC%2CHIGH_1%2CLOW_1%2CTRDPRC_1%2CACVOL_UNS%2CNUM_MOVES%2COPEN_BID%2COPEN_ASK%2CBID_NUMMOV%2CASK_NUMMOV%2CVWAP%2CDATE_TIME "HTTP/1.1 200 OK" 2025-06-17 16:53:34 httpx INFO> HTTP Request: GET http://localhost:9000/api/rdp/data/historical-pricing/v1/views/intraday-summaries/TD.CCP?interval=PT1M&start=2025-06-17T07%3A00%3A00.000000000Z&end=2025-06-17T12%3A00%3A00.000000000Z&adjustments=exchangeCorrection&sessions=pre%2Cnormal%2Cpost&fields=BID%2CASK%2COPEN_PRC%2CHIGH_1%2CLOW_1%2CTRDPRC_1%2CACVOL_UNS%2CNUM_MOVES%2COPEN_BID%2COPEN_ASK%2CBID_NUMMOV%2CASK_NUMMOV%2CVWAP%2CDATE_TIME "HTTP/1.1 200 OK"
I also see the same on the LSEG log:
[2025-06-17T16:52:39.088167-04:00] - [sessions.desktop.workspace.0] - [DEBUG] - [432 - MainThread] - [_session] - [open] - Opened session[2025-06-17T16:53:30.572487-04:00] - [sessions.desktop.workspace.0] - [DEBUG] - [9376 - HistoricalRequestThread_0] - [http_service] - [build_request] - HTTP Request id 2url = http://localhost:9000/api/rdp/data/historical-pricing/v1/views/intraday-summaries/TD.CCP?interval=PT1M&start=2025-06-17T07%3A00%3A00.000000000Z&end=2025-06-18T01%3A00%3A00.000000000Z&adjustments=exchangeCorrection&sessions=pre%2Cnormal%2Cpost&fields=BID%2CASK%2COPEN_PRC%2CHIGH_1%2CLOW_1%2CTRDPRC_1%2CACVOL_UNS%2CNUM_MOVES%2COPEN_BID%2COPEN_ASK%2CBID_NUMMOV%2CASK_NUMMOV%2CVWAP%2CDATE_TIMEmethod = GETheaders = {'request-timeout': '20', 'x-tr-applicationid': 'LD-PYTHON-LIB', 'Authorization': 'Bearer eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJBcGlWZXJzaW9uIjoiMSIsIkFwcEtleSI6IjQ3NDIxYzg3YTNmZTQ4OWNhMzAyY2NiMDMyZDQ1OWYyMDY5ZWM1MzAiLCJBcHBTY29wZSI6InRyYXBpIiwiTGlicmFyeU5hbWUiOiJSRFAgUHl0aG9uIExpYnJhcnkiLCJMaWJyYXJ5VmVyc2lvbiI6IjIuMC4xIiwiaWF0IjoxNzUwMTkzNTU5LCJleHAiOjE3NTE0MDMxNTl9.EHZW_SUVI_dhK22yDZzzU30fhMHfwqAhL47TSzRIpik'} LONG RESPONSE [2025-06-17T16:53:33.510374-04:00] - [sessions.desktop.workspace.0] - [DEBUG] - [9376 - HistoricalRequestThread_0] - [http_service] - [request] - HTTP Response id 2status_code = 200[LONG RESPONSE] text = [{"universe":{"ric":"TD.CCP"},"interval":"PT1M","summaryTimestampLabel":"startPeriod","adjustments":["exchangeCorrection","manualCorrection"],"defaultPricingField":"TRDPRC_1","headers":[{"name":"DATE_TIME","type":"string"},{"name":"HIGH_1","type":"number","decimalChar":"."},{"name":"LOW_1","type":"number","decimalChar":"."},{"name":"OPEN_PRC","type":"number","decimalChar":"."},{"name":"TRDPRC_1","type":"number","decimalChar":"."},{"name":"NUM_MOVES","type":"number","decimalChar":"."},{"name":"ACVOL_UNS","type":"number","decimalChar":"."},{"name":"VWAP","type":"number","decimalChar":"."},{"name":"OPEN_BID","type":"number","decimalChar":"."},{"name":"BID","type":"number","decimalChar":"."},{"name":"BID_NUMMOV","type":"number","decimalChar":"."},{"name":"OPEN_ASK","type":"number","decimalChar":"."},{"name":"ASK","type":"number","decimalChar":"."},{"name":"ASK_NUMMOV","type":"number","decimalChar":"."}],"data":[["2025-06-17T20:53:00.000000000Z",null,null,null,null,null,null,null,95.74,96.66,10,97.39,97.4,10], ["2025-06-17T20:52:00.000000000Z",null,null,null,null,null,null,null,96.68,96.67,25,97.4,97.39,25…..] [2025-06-17T16:53:33.525998-04:00] - [sessions.desktop.workspace.0] - [DEBUG] - [9376 - HistoricalRequestThread_0] - [http_service] - [build_request] - HTTP Request id 3url = http://localhost:9000/api/rdp/data/historical-pricing/v1/views/intraday-summaries/TD.CCP?interval=PT1M&start=2025-06-17T07%3A00%3A00.000000000Z&end=2025-06-17T12%3A00%3A00.000000000Z&adjustments=exchangeCorrection&sessions=pre%2Cnormal%2Cpost&fields=BID%2CASK%2COPEN_PRC%2CHIGH_1%2CLOW_1%2CTRDPRC_1%2CACVOL_UNS%2CNUM_MOVES%2COPEN_BID%2COPEN_ASK%2CBID_NUMMOV%2CASK_NUMMOV%2CVWAP%2CDATE_TIMEmethod = GETheaders = {'request-timeout': '20', 'x-tr-applicationid': 'LD-PYTHON-LIB', 'Authorization': 'Bearer eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJBcGlWZXJzaW9uIjoiMSIsIkFwcEtleSI6IjQ3NDIxYzg3YTNmZTQ4OWNhMzAyY2NiMDMyZDQ1OWYyMDY5ZWM1MzAiLCJBcHBTY29wZSI6InRyYXBpIiwiTGlicmFyeU5hbWUiOiJSRFAgUHl0aG9uIExpYnJhcnkiLCJMaWJyYXJ5VmVyc2lvbiI6IjIuMC4xIiwiaWF0IjoxNzUwMTkzNTU5LCJleHAiOjE3NTE0MDMxNTl9.EHZW_SUVI_dhK22yDZzzU30fhMHfwqAhL47TSzRIpik'} EMPTY RESPONSE [2025-06-17T16:53:34.119382-04:00] - [sessions.desktop.workspace.0] - [DEBUG] - [9376 - HistoricalRequestThread_0] - [http_service] - [request] - HTTP Response id 3status_code = 200[EMPTY RESPONSE]text = [{"universe":{"ric":"TD.CCP"},"interval":"PT1M","summaryTimestampLabel":"startPeriod","adjustments":["exchangeCorrection","manualCorrection"],"defaultPricingField":"TRDPRC_1","headers":[{"name":"DATE_TIME","type":"string"},{"name":"HIGH_1","type":"number","decimalChar":"."},{"name":"LOW_1","type":"number","decimalChar":"."},{"name":"OPEN_PRC","type":"number","decimalChar":"."},{"name":"TRDPRC_1","type":"number","decimalChar":"."},{"name":"NUM_MOVES","type":"number","decimalChar":"."},{"name":"ACVOL_UNS","type":"number","decimalChar":"."},{"name":"VWAP","type":"number","decimalChar":"."},{"name":"OPEN_BID","type":"number","decimalChar":"."},{"name":"BID","type":"number","decimalChar":"."},{"name":"BID_NUMMOV","type":"number","decimalChar":"."},{"name":"OPEN_ASK","type":"number","decimalChar":"."},{"name":"ASK","type":"number","decimalChar":"."},{"name":"ASK_NUMMOV","type":"number","decimalChar":"."}],"data":[]}]Note that the response on the LSEG log that I have marked "LONG RESPONSE" contains all the data but is too long to print here. The response that contained no data I labelled EMPTY RESPONSE and is in full.Is there a reason that every request has to count at least double? When a single intraday request counts twice that makes the 10,000 request daily limit even more binding. This amount of data was never the least problematic under the legacy Eikon C# .NET API. For reference, the use case here is calculating volume curves each day for a number of US and Japanese equities.
0 -
Thank you for the concrete example. It explains the issue.
The code requested 1min historical data from 2025-06-17 07:00:00+00:00 to 2025-06-18 01:00:00+00:00.
The first request requested 1min historical data from 2025-06-17T07:00:00.000000000Z to 2025-06-18T01:00:00.000000000Z which is similar to the code.
http://localhost:9001/api/rdp/data/historical-pricing/v1/views/intraday-summaries/TD.CCP?interval=PT1M&start=2025-06-17T07%3A00%3A00.000000000Z&end=2025-06-18T01%3A00%3A00.000000000Z&adjustments=exchangeCorrection&sessions=pre%2Cnormal
%2Cpost&fields=BID%2CASK%2COPEN_PRC%2CHIGH_1%2CLOW_1%2CTRDPRC_1%2CACVOL_UNS%2CNUM_MOVES%2COPEN_BID%2COPEN_ASK%2CBID_NUMMOV%2CASK_NUMMOV%2CVWAP%2CDATE_TIMEHowever, the response contained 505 entries from "2025-06-17T20:59:00.000000000Z" to "2025-06-17T12:00:00.000000000Z".
I think this is a reason why the library sent the second request to request 1min historical data from 2025-06-17T07:00:00.000000000Z to 2025-06-17T12:00:00.000000000Z in order to cover all missing data.
http://localhost:9001/api/rdp/data/historical-pricing/v1/views/intraday-summaries/TD.CCP?interval=PT1M&start=2025-06-17T07%3A00%3A00.000000000Z&end=2025-06-17T12%3A00%3A00.000000000Z&adjustments=exchangeCorrection&sessions=pre%2Cnormal%2Cpost&fields=BID%2CASK%2COPEN_PRC%2CHIGH_1%2CLOW_1%2CTRDPRC_1%2CACVOL_UNS%2CNUM_MOVES%2COPEN_BID%2COPEN_ASK%2CBID_NUMMOV%2CASK_NUMMOV%2CVWAP%2CDATE_TIME
It turns out that the second response contains empty data and the library wastes one request for nothing. This will affect your usage limitations.
I will contact the product team to verify this behavior.
0 -
The product team is aware of this issue and is actively working on a fix.
A resolution is planned for the next release of the LSEG Data Library for Python, expected within the next four weeks.
0 -
Thanks very much for your answers, Jirapongse, and the update on the upcoming fix; my apologies for not responding yesterday but we had a public holiday in the US. I have another limit behavior question. When we see a 429 limit come back, it appears that the API is counting a separate request every time we do re-request the 429 data. Consider the following output. Note the very straightforward request, but it seems to actually take 3 limit hits (the first request takes us down to 2712 according to the log, and we end at 2710), because we received 429s, even though we're obviously not receiving any data when we get a 429. Presumably a 429 that does not result in any data sent shouldn't count against quota? I'm not sure if switching from pyeikon to lseg.data fundamental_and_reference API would affect this, as it appears to be occurring at the headend which I gather is shared between the two, though I could switch APIs for legacy code if that would help.
`In [69]: eikon.get_data(['FFIU5', 'FFIZ5'], ['TR.OpenInterest', 'TR.OpenInterest.Date', 'TR.SettlementPrice', 'TR.SettlementPrice.Date'], parameters={'SDate':'2025-05-31', 'EDate':'2025-
...: 06-18'})
2025-06-18 20:57:15 root DEBUG> Sum of times: 0.0, messages limit: 3.99
2025-06-18 20:57:15 root INFO> sending request for 2 rics
2025-06-18 20:57:15 pyeikon TRACE> entity: DataGrid_StandardAsync
2025-06-18 20:57:15 pyeikon TRACE> payload: {'requests': [{'instruments': ['FFIU5', 'FFIZ5'], 'fields': [{'name': 'TR.OpenInterest'}, {'name': 'TR.OpenInterest.Date'}, {'name': 'TR.SettlementPrice'}, {'name': 'TR.SettlementPrice.Date'}], 'parameters': {'SDate': '2025-05-31', 'EDate': '2025-06-18'}}]}
2025-06-18 20:57:15 pyeikon DEBUG> Request:{'Entity': {'E': 'DataGrid_StandardAsync', 'W': {'requests': [{'instruments': ['FFIU5', 'FFIZ5'], 'fields': [{'name': 'TR.OpenInterest'}, {'name': 'TR.OpenInterest.Date'}, {'name': 'TR.SettlementPrice'}, {'name': 'TR.SettlementPrice.Date'}], 'parameters': {'SDate': '2025-05-31', 'EDate': '2025-06-18'}}]}}}
2025-06-18 20:57:15 pyeikon DEBUG> Request to http://cc-ny-rt01.cipher.local/api/v1/data
headers = {'Content-Type': 'application/json', 'x-tr-applicationid': 'f8596118c17e4f0280ae75545b25a826891172ee'}
params = None
2025-06-18 20:57:15 httpcore.connection DEBUG> close.started
2025-06-18 20:57:15 httpcore.connection DEBUG> close.complete
2025-06-18 20:57:15 httpcore.connection DEBUG> connect_tcp.started host='cc-ny-rt01.cipher.local' port=80 local_address=None timeout=None socket_options=None
2025-06-18 20:57:15 httpcore.connection DEBUG> connect_tcp.complete return_value=<httpcore._backends.anyio.AnyIOStream object at 0x000002620ED2BED0>
2025-06-18 20:57:15 httpcore.http11 DEBUG> send_request_headers.started request=<Request [b'POST']>
2025-06-18 20:57:15 httpcore.http11 DEBUG> send_request_headers.complete
2025-06-18 20:57:15 httpcore.http11 DEBUG> send_request_body.started request=<Request [b'POST']>
2025-06-18 20:57:15 httpcore.http11 DEBUG> send_request_body.complete
2025-06-18 20:57:15 httpcore.http11 DEBUG> receive_response_headers.started request=<Request [b'POST']>
2025-06-18 20:57:15 httpcore.http11 DEBUG> receive_response_headers.complete return_value=(b'HTTP/1.1', 307, b'Temporary Redirect', [(b'Server', b'nginx/1.17.2'), (b'Date', b'Thu, 19 Jun 2025 00:57:15 GMT'), (b'Content-Length', b'0'), (b'Connection', b'keep-alive'), (b'Access-Control-Allow-Origin', b'*'), (b'X-ID', b'13044.341'), (b'Location', b'/api/udf'), (b'Vary', b'Accept'), (b'Expires', b'Fri, 20 Jun 2025 00:57:15 GMT'), (b'Cache-Control', b'max-age=86400')])
2025-06-18 20:57:15 httpx INFO> HTTP Request: POST http://cc-ny-rt01.cipher.local/api/v1/data "HTTP/1.1 307 Temporary Redirect"
2025-06-18 20:57:15 httpcore.http11 DEBUG> receive_response_body.started request=<Request [b'POST']>
2025-06-18 20:57:15 httpcore.http11 DEBUG> receive_response_body.complete
2025-06-18 20:57:15 httpcore.http11 DEBUG> response_closed.started
2025-06-18 20:57:15 httpcore.http11 DEBUG> response_closed.complete
2025-06-18 20:57:15 httpcore.http11 DEBUG> send_request_headers.started request=<Request [b'POST']>
2025-06-18 20:57:15 httpcore.http11 DEBUG> send_request_headers.complete
2025-06-18 20:57:15 httpcore.http11 DEBUG> send_request_body.started request=<Request [b'POST']>
2025-06-18 20:57:15 httpcore.http11 DEBUG> send_request_body.complete
2025-06-18 20:57:15 httpcore.http11 DEBUG> receive_response_headers.started request=<Request [b'POST']>
2025-06-18 20:57:24 httpcore.http11 DEBUG> receive_response_headers.complete return_value=(b'HTTP/1.1', 200, b'OK', [(b'Server', b'nginx/1.17.2'), (b'Date', b'Thu, 19 Jun 2025 00:57:24 GMT'), (b'Content-Type', b'application/json; charset=utf-8'), (b'Content-Length', b'717'), (b'Connection', b'keep-alive'), (b'Access-Control-Allow-Origin', b'*'), (b'X-ID', b'13044.342'), (b'X-Request-Id', b'6be9af25-4faf-4af8-bc92-ab3a40fd8f6b'), (b'RateLimit-Remaining', b'2712'), (b'VolumeLimit-Remaining', b'4867753'), (b'QueueLimit-Remaining', b'100'), (b'RateLimit-Policy', b'5;w=1, 10000;w=86400'), (b'VolumeLimit-Policy', b'50000;w=60, 5000000;w=86400'), (b'RateLimit-Resource', b'*'), (b'VolumeLimit-Resource', b'*'), (b'ETag', b'W/"2cd-nEziERZYYfT0PMilRSSLezz899I"'), (b'Expires', b'Fri, 20 Jun 2025 00:57:24 GMT'), (b'Cache-Control', b'max-age=86400')])
2025-06-18 20:57:24 httpx INFO> HTTP Request: POST http://cc-ny-rt01.cipher.local/api/udf "HTTP/1.1 200 OK"
2025-06-18 20:57:24 httpcore.http11 DEBUG> receive_response_body.started request=<Request [b'POST']>
2025-06-18 20:57:24 httpcore.http11 DEBUG> receive_response_body.complete
2025-06-18 20:57:24 httpcore.http11 DEBUG> response_closed.started
2025-06-18 20:57:24 httpcore.http11 DEBUG> response_closed.complete
2025-06-18 20:57:24 pyeikon Level 1> Send request with headers [(b'Host', b'cc-ny-rt01.cipher.local'), (b'Accept-Encoding', b'gzip, deflate, br'), (b'Connection', b'keep-alive'), (b'User-Agent', b'python-httpx/0.25.0'), (b'Accept', b'application/json'), (b'Content-Type', b'application/json'), (b'x-tr-applicationid', b'f8596118c17e4f0280ae75545b25a826891172ee'), (b'Content-Length', b'307')] and cookies None
2025-06-18 20:57:24 pyeikon Level 1> HTTP request response 200: {"responses":[{"estimatedDuration":10000,"ticket":"93071CBB80F445D0BB7EE7F01AD1EC63_AAAAAA==_eyJpbnN0cnVtZW50cyI6WyJGRklVNSIsIkZGSVo1Il0sImZpZWxkcyI6W3sibGFiZWwiOiIiLCJuYW1lIjoiVFIuT3BlbkludGVyZXN0IiwicGFyYW1ldGVycyI6bnVsbCwic29ydCI6Im5vbmUiLCJzb3J0UHJpb3JpdHkiOjB9LHsibGFiZWwiOiIiLCJuYW1lIjoiVFIuT3BlbkludGVyZXN0LkRhdGUiLCJwYXJhbWV0ZXJzIjpudWxsLCJzb3J0Ijoibm9uZSIsInNvcnRQcmlvcml0eSI6MH0seyJsYWJlbCI6IiIsIm5hbWUiOiJUUi5TZXR0bGVtZW50UHJpY2UiLCJwYXJhbWV0ZXJzIjpudWxsLCJzb3J0Ijoibm9uZSIsInNvcnRQcmlvcml0eSI6MH0seyJsYWJlbCI6IiIsIm5hbWUiOiJUUi5TZXR0bGVtZW50UHJpY2UuRGF0ZSIsInBhcmFtZXRlcnMiOm51bGwsInNvcnQiOiJub25lIiwic29ydFByaW9yaXR5IjowfV0sInBhcmFtZXRlcnMiOnsiRURhdGUiOiIyMDI1LTA2LTE4IiwiU0RhdGUiOiIyMDI1LTA1LTMxIn19"}]}
2025-06-18 20:57:24 pyeikon DEBUG> HTTP Response code: 200
2025-06-18 20:57:24 pyeikon DEBUG> HTTP Response: {"responses":[{"estimatedDuration":10000,"ticket":"93071CBB80F445D0BB7EE7F01AD1EC63_AAAAAA==_eyJpbnN0cnVtZW50cyI6WyJGRklVNSIsIkZGSVo1Il0sImZpZWxkcyI6W3sibGFiZWwiOiIiLCJuYW1lIjoiVFIuT3BlbkludGVyZXN0IiwicGFyYW1ldGVycyI6bnVsbCwic29ydCI6Im5vbmUiLCJzb3J0UHJpb3JpdHkiOjB9LHsibGFiZWwiOiIiLCJuYW1lIjoiVFIuT3BlbkludGVyZXN0LkRhdGUiLCJwYXJhbWV0ZXJzIjpudWxsLCJzb3J0Ijoibm9uZSIsInNvcnRQcmlvcml0eSI6MH0seyJsYWJlbCI6IiIsIm5hbWUiOiJUUi5TZXR0bGVtZW50UHJpY2UiLCJwYXJhbWV0ZXJzIjpudWxsLCJzb3J0Ijoibm9uZSIsInNvcnRQcmlvcml0eSI6MH0seyJsYWJlbCI6IiIsIm5hbWUiOiJUUi5TZXR0bGVtZW50UHJpY2UuRGF0ZSIsInBhcmFtZXRlcnMiOm51bGwsInNvcnQiOiJub25lIiwic29ydFByaW9yaXR5IjowfV0sInBhcmFtZXRlcnMiOnsiRURhdGUiOiIyMDI1LTA2LTE4IiwiU0RhdGUiOiIyMDI1LTA1LTMxIn19"}]}
2025-06-18 20:57:24 pyeikon TRACE> Response size: 770
2025-06-18 20:57:24 pyeikon INFO> Receive ticket from responses, wait for 10.0 seconds
2025-06-18 20:57:34 pyeikon DEBUG> Send ticket request:{'Entity': {'E': 'DataGrid_StandardAsync', 'W': {'requests': [{'ticket': '93071CBB80F445D0BB7EE7F01AD1EC63_AAAAAA==_eyJpbnN0cnVtZW50cyI6WyJGRklVNSIsIkZGSVo1Il0sImZpZWxkcyI6W3sibGFiZWwiOiIiLCJuYW1lIjoiVFIuT3BlbkludGVyZXN0IiwicGFyYW1ldGVycyI6bnVsbCwic29ydCI6Im5vbmUiLCJzb3J0UHJpb3JpdHkiOjB9LHsibGFiZWwiOiIiLCJuYW1lIjoiVFIuT3BlbkludGVyZXN0LkRhdGUiLCJwYXJhbWV0ZXJzIjpudWxsLCJzb3J0Ijoibm9uZSIsInNvcnRQcmlvcml0eSI6MH0seyJsYWJlbCI6IiIsIm5hbWUiOiJUUi5TZXR0bGVtZW50UHJpY2UiLCJwYXJhbWV0ZXJzIjpudWxsLCJzb3J0Ijoibm9uZSIsInNvcnRQcmlvcml0eSI6MH0seyJsYWJlbCI6IiIsIm5hbWUiOiJUUi5TZXR0bGVtZW50UHJpY2UuRGF0ZSIsInBhcmFtZXRlcnMiOm51bGwsInNvcnQiOiJub25lIiwic29ydFByaW9yaXR5IjowfV0sInBhcmFtZXRlcnMiOnsiRURhdGUiOiIyMDI1LTA2LTE4IiwiU0RhdGUiOiIyMDI1LTA1LTMxIn19'}]}}}
2025-06-18 20:57:34 pyeikon DEBUG> Request to http://cc-ny-rt01.cipher.local/api/v1/data
headers = {'Content-Type': 'application/json', 'x-tr-applicationid': 'f8596118c17e4f0280ae75545b25a826891172ee'}
params = None
2025-06-18 20:57:34 httpcore.connection DEBUG> close.started
2025-06-18 20:57:34 httpcore.connection DEBUG> close.complete
2025-06-18 20:57:34 httpcore.connection DEBUG> connect_tcp.started host='cc-ny-rt01.cipher.local' port=80 local_address=None timeout=15.0 socket_options=None
2025-06-18 20:57:34 httpcore.connection DEBUG> connect_tcp.complete return_value=<httpcore._backends.anyio.AnyIOStream object at 0x0000026212A62450>
2025-06-18 20:57:34 httpcore.http11 DEBUG> send_request_headers.started request=<Request [b'POST']>
2025-06-18 20:57:34 httpcore.http11 DEBUG> send_request_headers.complete
2025-06-18 20:57:34 httpcore.http11 DEBUG> send_request_body.started request=<Request [b'POST']>
2025-06-18 20:57:34 httpcore.http11 DEBUG> send_request_body.complete
2025-06-18 20:57:34 httpcore.http11 DEBUG> receive_response_headers.started request=<Request [b'POST']>
2025-06-18 20:57:34 httpcore.http11 DEBUG> receive_response_headers.complete return_value=(b'HTTP/1.1', 307, b'Temporary Redirect', [(b'Server', b'nginx/1.17.2'), (b'Date', b'Thu, 19 Jun 2025 00:57:34 GMT'), (b'Content-Length', b'0'), (b'Connection', b'keep-alive'), (b'Access-Control-Allow-Origin', b'*'), (b'X-ID', b'13044.343'), (b'Location', b'/api/udf'), (b'Vary', b'Accept'), (b'Expires', b'Fri, 20 Jun 2025 00:57:34 GMT'), (b'Cache-Control', b'max-age=86400')])
2025-06-18 20:57:34 httpx INFO> HTTP Request: POST http://cc-ny-rt01.cipher.local/api/v1/data "HTTP/1.1 307 Temporary Redirect"
2025-06-18 20:57:34 httpcore.http11 DEBUG> receive_response_body.started request=<Request [b'POST']>
2025-06-18 20:57:34 httpcore.http11 DEBUG> receive_response_body.complete
2025-06-18 20:57:34 httpcore.http11 DEBUG> response_closed.started
2025-06-18 20:57:34 httpcore.http11 DEBUG> response_closed.complete
2025-06-18 20:57:34 httpcore.http11 DEBUG> send_request_headers.started request=<Request [b'POST']>
2025-06-18 20:57:34 httpcore.http11 DEBUG> send_request_headers.complete
2025-06-18 20:57:34 httpcore.http11 DEBUG> send_request_body.started request=<Request [b'POST']>
2025-06-18 20:57:34 httpcore.http11 DEBUG> send_request_body.complete
2025-06-18 20:57:34 httpcore.http11 DEBUG> receive_response_headers.started request=<Request [b'POST']>
2025-06-18 20:57:35 httpcore.http11 DEBUG> receive_response_headers.complete return_value=(b'HTTP/1.1', 200, b'OK', [(b'Server', b'nginx/1.17.2'), (b'Date', b'Thu, 19 Jun 2025 00:57:35 GMT'), (b'Content-Type', b'application/json; charset=utf-8'), (b'Content-Length', b'718'), (b'Connection', b'keep-alive'), (b'Access-Control-Allow-Origin', b'*'), (b'X-ID', b'13044.344'), (b'X-Request-Id', b'84dc6a32-447b-453a-bd93-765970a15e93'), (b'RateLimit-Remaining', b'2711'), (b'VolumeLimit-Remaining', b'4867752'), (b'QueueLimit-Remaining', b'100'), (b'RateLimit-Policy', b'5;w=1, 10000;w=86400'), (b'VolumeLimit-Policy', b'50000;w=60, 5000000;w=86400'), (b'RateLimit-Resource', b'*'), (b'VolumeLimit-Resource', b'*'), (b'ETag', b'W/"2ce-x+42cXDkHD1wxfMZ+S/RaKCZzmE"'), (b'Expires', b'Fri, 20 Jun 2025 00:57:35 GMT'), (b'Cache-Control', b'max-age=86400')])
2025-06-18 20:57:35 httpx INFO> HTTP Request: POST http://cc-ny-rt01.cipher.local/api/udf "HTTP/1.1 200 OK"
2025-06-18 20:57:35 httpcore.http11 DEBUG> receive_response_body.started request=<Request [b'POST']>
2025-06-18 20:57:35 httpcore.http11 DEBUG> receive_response_body.complete
2025-06-18 20:57:35 httpcore.http11 DEBUG> response_closed.started
2025-06-18 20:57:35 httpcore.http11 DEBUG> response_closed.complete
2025-06-18 20:57:35 pyeikon Level 1> Send request with headers [(b'Host', b'cc-ny-rt01.cipher.local'), (b'Accept-Encoding', b'gzip, deflate, br'), (b'Connection', b'keep-alive'), (b'User-Agent', b'python-httpx/0.25.0'), (b'Accept', b'application/json'), (b'Content-Type', b'application/json'), (b'x-tr-applicationid', b'f8596118c17e4f0280ae75545b25a826891172ee'), (b'Content-Length', b'742')] and cookies None
2025-06-18 20:57:35 pyeikon Level 1> HTTP request response 200: {"responses":[{"estimatedDuration":921000,"ticket":"93071CBB80F445D0BB7EE7F01AD1EC63_AAAAAA==_eyJpbnN0cnVtZW50cyI6WyJGRklVNSIsIkZGSVo1Il0sImZpZWxkcyI6W3sibGFiZWwiOiIiLCJuYW1lIjoiVFIuT3BlbkludGVyZXN0IiwicGFyYW1ldGVycyI6bnVsbCwic29ydCI6Im5vbmUiLCJzb3J0UHJpb3JpdHkiOjB9LHsibGFiZWwiOiIiLCJuYW1lIjoiVFIuT3BlbkludGVyZXN0LkRhdGUiLCJwYXJhbWV0ZXJzIjpudWxsLCJzb3J0Ijoibm9uZSIsInNvcnRQcmlvcml0eSI6MH0seyJsYWJlbCI6IiIsIm5hbWUiOiJUUi5TZXR0bGVtZW50UHJpY2UiLCJwYXJhbWV0ZXJzIjpudWxsLCJzb3J0Ijoibm9uZSIsInNvcnRQcmlvcml0eSI6MH0seyJsYWJlbCI6IiIsIm5hbWUiOiJUUi5TZXR0bGVtZW50UHJpY2UuRGF0ZSIsInBhcmFtZXRlcnMiOm51bGwsInNvcnQiOiJub25lIiwic29ydFByaW9yaXR5IjowfV0sInBhcmFtZXRlcnMiOnsiRURhdGUiOiIyMDI1LTA2LTE4IiwiU0RhdGUiOiIyMDI1LTA1LTMxIn19"}]}
2025-06-18 20:57:35 pyeikon DEBUG> HTTP Response: 200 - {"responses":[{"estimatedDuration":921000,"ticket":"93071CBB80F445D0BB7EE7F01AD1EC63_AAAAAA==_eyJpbnN0cnVtZW50cyI6WyJGRklVNSIsIkZGSVo1Il0sImZpZWxkcyI6W3sibGFiZWwiOiIiLCJuYW1lIjoiVFIuT3BlbkludGVyZXN0IiwicGFyYW1ldGVycyI6bnVsbCwic29ydCI6Im5vbmUiLCJzb3J0UHJpb3JpdHkiOjB9LHsibGFiZWwiOiIiLCJuYW1lIjoiVFIuT3BlbkludGVyZXN0LkRhdGUiLCJwYXJhbWV0ZXJzIjpudWxsLCJzb3J0Ijoibm9uZSIsInNvcnRQcmlvcml0eSI6MH0seyJsYWJlbCI6IiIsIm5hbWUiOiJUUi5TZXR0bGVtZW50UHJpY2UiLCJwYXJhbWV0ZXJzIjpudWxsLCJzb3J0Ijoibm9uZSIsInNvcnRQcmlvcml0eSI6MH0seyJsYWJlbCI6IiIsIm5hbWUiOiJUUi5TZXR0bGVtZW50UHJpY2UuRGF0ZSIsInBhcmFtZXRlcnMiOm51bGwsInNvcnQiOiJub25lIiwic29ydFByaW9yaXR5IjowfV0sInBhcmFtZXRlcnMiOnsiRURhdGUiOiIyMDI1LTA2LTE4IiwiU0RhdGUiOiIyMDI1LTA1LTMxIn19"}]}
2025-06-18 20:57:35 pyeikon ERROR> HTTP request failed: EikonError-921.0 seconds exceeds max ticket duration
G:\Users\krychd\miniconda3\envs\prod_20250515\Lib\site-packages\eikon\data_grid.py:190:UserWarning: Error code 408 | 921.0 seconds exceeds max ticket duration
Error code 408 | 921.0 seconds exceeds max ticket duration
2025-06-18 20:57:45 pyeikon TRACE> entity: DataGrid_StandardAsync
2025-06-18 20:57:45 pyeikon TRACE> payload: {'requests': [{'instruments': ['FFIU5', 'FFIZ5'], 'fields': [{'name': 'TR.OpenInterest'}, {'name': 'TR.OpenInterest.Date'}, {'name': 'TR.SettlementPrice'}, {'name': 'TR.SettlementPrice.Date'}], 'parameters': {'SDate': '2025-05-31', 'EDate': '2025-06-18'}}]}
2025-06-18 20:57:45 pyeikon DEBUG> Request:{'Entity': {'E': 'DataGrid_StandardAsync', 'W': {'requests': [{'instruments': ['FFIU5', 'FFIZ5'], 'fields': [{'name': 'TR.OpenInterest'}, {'name': 'TR.OpenInterest.Date'}, {'name': 'TR.SettlementPrice'}, {'name': 'TR.SettlementPrice.Date'}], 'parameters': {'SDate': '2025-05-31', 'EDate': '2025-06-18'}}]}}}
2025-06-18 20:57:45 pyeikon DEBUG> Request to http://cc-ny-rt01.cipher.local/api/v1/data
headers = {'Content-Type': 'application/json', 'x-tr-applicationid': 'f8596118c17e4f0280ae75545b25a826891172ee'}
params = None
2025-06-18 20:57:45 httpcore.connection DEBUG> close.started
2025-06-18 20:57:45 httpcore.connection DEBUG> close.complete
2025-06-18 20:57:45 httpcore.connection DEBUG> connect_tcp.started host='cc-ny-rt01.cipher.local' port=80 local_address=None timeout=None socket_options=None
2025-06-18 20:57:45 httpcore.connection DEBUG> connect_tcp.complete return_value=<httpcore._backends.anyio.AnyIOStream object at 0x0000026212A94B10>
2025-06-18 20:57:45 httpcore.http11 DEBUG> send_request_headers.started request=<Request [b'POST']>
2025-06-18 20:57:45 httpcore.http11 DEBUG> send_request_headers.complete
2025-06-18 20:57:45 httpcore.http11 DEBUG> send_request_body.started request=<Request [b'POST']>
2025-06-18 20:57:45 httpcore.http11 DEBUG> send_request_body.complete
2025-06-18 20:57:45 httpcore.http11 DEBUG> receive_response_headers.started request=<Request [b'POST']>
2025-06-18 20:57:45 httpcore.http11 DEBUG> receive_response_headers.complete return_value=(b'HTTP/1.1', 307, b'Temporary Redirect', [(b'Server', b'nginx/1.17.2'), (b'Date', b'Thu, 19 Jun 2025 00:57:45 GMT'), (b'Content-Length', b'0'), (b'Connection', b'keep-alive'), (b'Access-Control-Allow-Origin', b'*'), (b'X-ID', b'13044.345'), (b'Location', b'/api/udf'), (b'Vary', b'Accept'), (b'Expires', b'Fri, 20 Jun 2025 00:57:45 GMT'), (b'Cache-Control', b'max-age=86400')])
2025-06-18 20:57:45 httpx INFO> HTTP Request: POST http://cc-ny-rt01.cipher.local/api/v1/data "HTTP/1.1 307 Temporary Redirect"
2025-06-18 20:57:45 httpcore.http11 DEBUG> receive_response_body.started request=<Request [b'POST']>
2025-06-18 20:57:45 httpcore.http11 DEBUG> receive_response_body.complete
2025-06-18 20:57:45 httpcore.http11 DEBUG> response_closed.started
2025-06-18 20:57:45 httpcore.http11 DEBUG> response_closed.complete
2025-06-18 20:57:45 httpcore.http11 DEBUG> send_request_headers.started request=<Request [b'POST']>
2025-06-18 20:57:45 httpcore.http11 DEBUG> send_request_headers.complete
2025-06-18 20:57:45 httpcore.http11 DEBUG> send_request_body.started request=<Request [b'POST']>
2025-06-18 20:57:45 httpcore.http11 DEBUG> send_request_body.complete
2025-06-18 20:57:45 httpcore.http11 DEBUG> receive_response_headers.started request=<Request [b'POST']>
2025-06-18 20:57:45 httpcore.http11 DEBUG> receive_response_headers.complete return_value=(b'HTTP/1.1', 200, b'OK', [(b'Server', b'nginx/1.17.2'), (b'Date', b'Thu, 19 Jun 2025 00:57:45 GMT'), (b'Content-Type', b'application/json; charset=utf-8'), (b'Content-Length', b'2132'), (b'Connection', b'keep-alive'), (b'Access-Control-Allow-Origin', b'*'), (b'X-ID', b'13044.346'), (b'X-Request-Id', b'7c5c9b5c-8f17-48da-8bc0-69381abe8a53'), (b'RateLimit-Remaining', b'2710'), (b'VolumeLimit-Remaining', b'4867750'), (b'QueueLimit-Remaining', b'100'), (b'RateLimit-Policy', b'5;w=1, 10000;w=86400'), (b'VolumeLimit-Policy', b'50000;w=60, 5000000;w=86400'), (b'RateLimit-Resource', b'*'), (b'VolumeLimit-Resource', b'*'), (b'ETag', b'W/"854-8gtzCfSSDG/2s9R58z/HAu5g3vE"'), (b'Expires', b'Fri, 20 Jun 2025 00:57:45 GMT'), (b'Cache-Control', b'max-age=86400')])
2025-06-18 20:57:45 httpx INFO> HTTP Request: POST http://cc-ny-rt01.cipher.local/api/udf "HTTP/1.1 200 OK"
2025-06-18 20:57:45 httpcore.http11 DEBUG> receive_response_body.started request=<Request [b'POST']>
2025-06-18 20:57:45 httpcore.http11 DEBUG> receive_response_body.complete
2025-06-18 20:57:45 httpcore.http11 DEBUG> response_closed.started
2025-06-18 20:57:45 httpcore.http11 DEBUG> response_closed.complete
2025-06-18 20:57:45 pyeikon Level 1> Send request with headers [(b'Host', b'cc-ny-rt01.cipher.local'), (b'Accept-Encoding', b'gzip, deflate, br'), (b'Connection', b'keep-alive'), (b'User-Agent', b'python-httpx/0.25.0'), (b'Accept', b'application/json'), (b'Content-Type', b'application/json'), (b'x-tr-applicationid', b'f8596118c17e4f0280ae75545b25a826891172ee'), (b'Content-Length', b'307')] and cookies None0 -
Is it about the RateLimit-Remaining in the response's headers?
According to the log, the library sent a request to the DataGrid_StandardAsync service. If the service cannot provide a response in time, it returns a ticket. The library then uses this ticket to poll for the data.
Currently, it appears that we also count poll requests toward the rate limits.However, I have tested this with the LSEG Data Library for Python and configure the library to use "rdp" for the underlying datagrid service.
config = ld.get_config() config.set_param("apis.data.datagrid.underlying-platform", "rdp")The RDP datagrid service doesn't have the polling concept. It uses the configuration (http.request-timeout) to control the HTTP timeout.
0 -
Yes, I'm looking at ratelimit_remaining in the response headers to determine how quickly the rate limit is being depleted.
We can certainly switch from the pyeikon.get_data API to the lseg.data fundamental_and_reference API (i was sort of under the impression they had simliar headends as they were both supported and take similar parameters, but that seems to not be the case based on what you're saying below, so that may be a reason to make the switch to the newer API).
Do we still have cases under the fundamental_and_reference API where the ticket is for longer than 15 minutes, as it is above? Perhaps the new endpoint no longer exhibits that behavior?0 -
I understand that both DataGrid_StandardAsync and fundamental_and_reference (RDP Datagrid) have the limitation regarding server timeout. If the request takes a lot of times to extract data, it can be timed out by the server.
0 -
Thanks Jirapongse, I'm a bit confused by your prior two answers perhaps because I am not super-familiar with RDP vs the DataGrid. How do the following options differ from each other in terms of reliability of data retrieval and limit usage in a 429 scenario:
- pyeikon .get_data
- lseg.data.content.fundamental_and_reference, with no config explicitly set
- lseg.data.content.fundamental_and_reference, withconfig.set_param("apis.data.datagrid.underlying-platform", "rdp")It sounds like any or all of them could have a server timeout from your latest request but I guess what I'm really wondering is, if I have generally working code using pyeikon.get_data, but that seems to use extra limits when the server gets busy and 429s me, should I nonetheless port the code to lseg.data.content.fundamental_and_reference to make it more reliable?
0 -
There are two endpoints to get fundamental and reference data (TR.xxx fields). The old one is DataGrid_StandardAsync on http://cc-ny-rt01.cipher.local/api/v1/data and the new one is http://localhost:9001/api/rdp/data/datagrid/beta1/.
The product team recommended using the new one supported by the LSEG Data Library for Python. However, this also has limitations as mentioned on this page.
0
Categories
- All Categories
- 3 Polls
- 6 AHS
- 36 Alpha
- 166 App Studio
- 6 Block Chain
- 4 Bot Platform
- 18 Connected Risk APIs
- 47 Data Fusion
- 34 Data Model Discovery
- 690 Datastream
- 1.4K DSS
- 629 Eikon COM
- 5.2K Eikon Data APIs
- 11 Electronic Trading
- 1 Generic FIX
- 7 Local Bank Node API
- 3 Trading API
- 2.9K Elektron
- 1.4K EMA
- 255 ETA
- 559 WebSocket API
- 39 FX Venues
- 15 FX Market Data
- 1 FX Post Trade
- 1 FX Trading - Matching
- 12 FX Trading – RFQ Maker
- 5 Intelligent Tagging
- 2 Legal One
- 25 Messenger Bot
- 3 Messenger Side by Side
- 9 ONESOURCE
- 7 Indirect Tax
- 60 Open Calais
- 280 Open PermID
- 45 Entity Search
- 2 Org ID
- 1 PAM
- PAM - Logging
- 6 Product Insight
- Project Tracking
- ProView
- ProView Internal
- 23 RDMS
- 2K Refinitiv Data Platform
- 721 Refinitiv Data Platform Libraries
- 4 LSEG Due Diligence
- LSEG Due Diligence Portal API
- 4 Refinitiv Due Dilligence Centre
- Rose's Space
- 1.2K Screening
- 18 Qual-ID API
- 13 Screening Deployed
- 23 Screening Online
- 12 World-Check Customer Risk Screener
- 1K World-Check One
- 46 World-Check One Zero Footprint
- 45 Side by Side Integration API
- 2 Test Space
- 3 Thomson One Smart
- 10 TR Knowledge Graph
- 151 Transactions
- 143 REDI API
- 1.8K TREP APIs
- 4 CAT
- 27 DACS Station
- 121 Open DACS
- 1.1K RFA
- 106 UPA
- 194 TREP Infrastructure
- 229 TRKD
- 918 TRTH
- 5 Velocity Analytics
- 9 Wealth Management Web Services
- 95 Workspace SDK
- 11 Element Framework
- 5 Grid
- 19 World-Check Data File
- 1 Yield Book Analytics
- 48 中文论坛
