getting KeyError: 'headers' from data request
Hi, I am using the ek.get_data() function to request data, which often works but sometimes gives the following error. What is causing this and how can I resolve it? Thanks.
ERROR:
Traceback (most recent call last):
File "/usr/local/lib/python3.7/site-packages/eikon/data_grid.py", line 169, in get_data
return get_data_frame(result, field_name)
File "/usr/local/lib/python3.7/site-packages/eikon/data_grid.py", line 218, in get_data_frame
headers = [header['displayName'] for header in data_dict['headers'][0]]
KeyError: 'headers'
Best Answer
-
The issue with the Web service returning empty response or "Backend error. 400 Bad Request" has been addressed with the release of Eikon Data APIs library for Python v1.0.1 available on PyPI. This does not mean that with Eikon Data APIs library for Python v1.0.1 one can never receive "Bad request" error or that the service can now provide unlimited data downloads. But one should no longer experience random and frequent empty responses or "Backend error. 400 Bad Request" forcing one to resubmit the same requests in a loop until the data is returned.
0
Answers
-
@jr123 looks like it returns an empty dict without returning an error, difficult to say without more data. Try re-executing it with either debug set to True, or as a raw_output set to True.
0 -
I'm getting the same thing but seems to happen randomly (i.e. running the same code and sometimes it works sometimes it fails). Would be great to understand what it is and why it is happening!
0 -
@jr123
Since we're unable to reproduce the issue we have to rely on you to troubleshoot it. I would suggest using Fiddler or another HTTP analyzer to capture the HTTP traffic and see what response you get from the Web service providing the data that results in the exception you experienced. We'd be very grateful if you could share your findings on this thread.0 -
@Zhenya Kovalyov Hello! Same problem with get_data. With debug=True raise Error 408:
0 -
@oalexandere thanks, I have raised this with the product owner.
So, it seems that the call times out, I would suggest repeating the call.
0 -
There's a 60 seconds time out by default.
You can extend to a longer duration with the function ek.set_timeout(timeout) function.
0 -
This exception means that returned data_dict is empty ('headers' key error is raised)
It reproduced the case :
>>> ek.get_data("PEUP.PA", "ASK", raw_output=True)
{}
>>>It isn't expected at all and the root cause is on the back-end side.
0 -
if it is a timeout, then why doesn't it error accordingly in the console? the KeyError headers is quite confusing...
0 -
This case (timeout) is different than the first issue (empty result).
0 -
@574c708d-6b25-445f-8303-2b495473ec99_deprecated_6f93d9f8-9d25-487c-91bf-b3a796f48f40
Are you still experiencing either the timeout or the empty results issue?
0 -
Hi - I'm not sure bc I creater a wrapper to automatically handle timeouts by breaking down the query. will let you know if I run into problems though.
0 -
@Alex Putkov.1 I can confirm this behavior is still occurring. On our side the incidence coincided roughly with the auto-installation of Eikon 4.0.46, though I don't know enough about the architecture to know if it's the Eikon desktop application itself that changed or a change at the headend that was rolled out at the same time.
0 -
@davidk
Are you referring to the empty results? Do you have an example of the call that still returns empty results? I can no longer reproduce the issue using ek.get_data("PEUP.PA", "ASK", raw_output=True), which @pierre.faurel previously successfully used to reproduce the problem.0 -
Try asking for an large number of RICs (below, rics is a list of 3000 US rics and I slice out the top 500), so you can induce a timeout:
In [14]: ek.get_data(rics[:500], ['TR.InvestorFullName', 'TR.InvestorType', 'TR.InvAddrCountry', 'TR.InvInvestmentStyleCode', 'TR.In
vInvmtOrientation', 'TR.SharesHeldValue', 'TR.SharesHeldValue.date', 'TR.FilingType'], {'sdate': '2018-12-31', 'edate': '2018-12-
31'}, raw_output=True)Out[14]: {}
0 -
Thanks @davidk. I reproduced the issue using the example you provided. I expect the service to either return the data requested or an error. In this case it returns empty JSON with no indication of any errors. I opened the service case #07236642 on your behalf for this issue.
0 -
@Alex Putkov.1 Where can I follow the service case #07236642?
I think I'm getting similar errors when I run get_data(screen(contains(name))) for a large list of names.0 -
Just to update you, our development team is looking to implement a fix that would return an error in response to requests similar to the example @davidk provided instead of an empty JSON the Web service returns now. This will make user experience better, as you will see a more meaningful error message compared to what the current KeyError exception provides. We're not however looking to fulfill such enormous requests. If we were to do this, the fulfilment time would be in minutes and the payload would be in Gigabytes. We don't think the bulk download capability is appropriate for an end user terminal product, nor do we think Eikon infrastructure could reliably support this at scale. So, in order to retrieve this data you'll still need to break down the list of RICs into much smaller chunks and use only a small handful of RICs (or perhaps even just one RIC) in each request.
0 -
Ok, Thanks @Alex Putkov.1
0 -
Understood Alex. The better error message is really what we're looking for here. Note my sample request was very large, but I made it such to reliably and repeatably induce a timeout-- I think the original behavior reported indicates that timeouts are an occasional fact of life even on smaller requests (such is the Internet, esp. when connecting via HTTPS with corporate middleboxes in the way, etc. etc.)
0 -
The problem has apparently gotten worse after the recent DataGrid deployment. Reconstruction below using a simple EPS data request that should generate a payload of under 60 KB:
import eikon as ek
fields = ['TR.PriceTargetMean', 'TR.PriceTargetStdDev', 'TR.PriceTargetNumOfEstimates', 'TR.PriceTargetNumIncEstimates', 'TR.RecMean', 'TR.NumOfRecommendations', 'TR.EPSMean', 'TR.EPSStdDev', 'TR.EPSNumberofEstimates', 'TR.EPSNumIncEstimates', 'TR.EPSPctNumEstRevisingDown(WP=1d)', 'TR.EPSPctNumEstRevisingUp(WP=1d)', 'TR.EPSMean.periodenddate']
rics = pd.Index(ek.get_data('0#.SPX', 'TR.RIC')[0]['RIC']).tolist()# Returns empty dict
ek.get_data(rics, fields, parameters={'Period':'FY1'}, raw_output=True)# returns correctly on the first 100 securities, payload is 11,120 bytes
output = ek.get_data(rics[:100], fields, parameters={'Period':'FY1'}, raw_output=True)I have followed up with support on case [ ref:_00D30602X._5001W1Py1dr:ref ] as well, but thought you and the Data API team should know. These increasing errors are making it unnecessarily difficult to use the Eikon Data API for any but the smallest requests.
0 -
@davidk
The new version of Data Grid, which we believe will alleviate the issue with empty responses, has not been deployed into production yet. Any variations in the quality of service you've seen are due to random factors. We expect the new version of Data Grid to be deployed into production next month.0 -
Customer support emailed me on March 6 to say that the DataGrid folks told them this was fixed in the v1.4 deploy on Feb 28, but clearly it wasn't fixed; perhaps v1.4 wasn't actually deployed?
0 -
Correct. Customer support must have made a mistake. Data Grid 1.4 has not been deployed in production. It is expected to be deployed next month.
0 -
@Alex Putkov.1 Has DataGrid 1.4 been deployed? If not, do you know what the ETA is? We're still seeing KeyError 'headers' but have also seen some new errors crop up in the last few days.
0 -
Data Grid 1.4 should be targeted to external users in production later today or tomorrow. What you should expect to see is eikon Python library raising eikon.EikonError exception "Backend error. 400 Bad Request" where it previously raised KeyError exception due to empty response from Data Grid. In other words instead of an empty response Data Grid will return an error message.
0 -
However our testing showed that contrary to expectations Data Grid 1.4 release did not improve the success rate of regular requests like a dozen fields for a few hundred stocks. This is very disappointing and we're working on addressing this issue as a top priority item.
0 -
Thanks Alex. I see this behavior fixed on Eikon 1.0.1. Note the new async ticket system introduced in 1.0.1 does appear to have a corner-case bug on Python 2.7 (https://community.developers.refinitiv.com/questions/44621/view.html), that occurs only when the request generates a delay ticket from DataGrid and the eventual response contains non-ASCII Unicode characters. Python 3+ is unaffected.
0
Categories
- All Categories
- 6 AHS
- 36 Alpha
- 166 App Studio
- 6 Block Chain
- 4 Bot Platform
- 18 Connected Risk APIs
- 47 Data Fusion
- 33 Data Model Discovery
- 682 Datastream
- 1.4K DSS
- 613 Eikon COM
- 5.2K Eikon Data APIs
- 10 Electronic Trading
- Generic FIX
- 7 Local Bank Node API
- 3 Trading API
- 2.9K Elektron
- 1.4K EMA
- 248 ETA
- 552 WebSocket API
- 37 FX Venues
- 14 FX Market Data
- 1 FX Post Trade
- 1 FX Trading - Matching
- 12 FX Trading – RFQ Maker
- 5 Intelligent Tagging
- 2 Legal One
- 23 Messenger Bot
- 3 Messenger Side by Side
- 9 ONESOURCE
- 7 Indirect Tax
- 60 Open Calais
- 275 Open PermID
- 44 Entity Search
- 2 Org ID
- 1 PAM
- PAM - Logging
- 6 Product Insight
- Project Tracking
- ProView
- ProView Internal
- 22 RDMS
- 1.8K Refinitiv Data Platform
- 625 Refinitiv Data Platform Libraries
- 4 LSEG Due Diligence
- LSEG Due Diligence Portal API
- 4 Refinitiv Due Dilligence Centre
- Rose's Space
- 1.2K Screening
- 18 Qual-ID API
- 13 Screening Deployed
- 23 Screening Online
- 12 World-Check Customer Risk Screener
- 1K World-Check One
- 46 World-Check One Zero Footprint
- 45 Side by Side Integration API
- 2 Test Space
- 3 Thomson One Smart
- 10 TR Knowledge Graph
- 151 Transactions
- 143 REDI API
- 1.8K TREP APIs
- 4 CAT
- 26 DACS Station
- 121 Open DACS
- 1.1K RFA
- 104 UPA
- 191 TREP Infrastructure
- 228 TRKD
- 915 TRTH
- 5 Velocity Analytics
- 9 Wealth Management Web Services
- 83 Workspace SDK
- 11 Element Framework
- 5 Grid
- 18 World-Check Data File
- 1 Yield Book Analytics
- 46 中文论坛
