question

Upvotes
Accepted
31 6 3 6

getting KeyError: 'headers' from data request

Hi, I am using the ek.get_data() function to request data, which often works but sometimes gives the following error. What is causing this and how can I resolve it? Thanks.

ERROR:

Traceback (most recent call last):

File "/usr/local/lib/python3.7/site-packages/eikon/data_grid.py", line 169, in get_data

return get_data_frame(result, field_name)

File "/usr/local/lib/python3.7/site-packages/eikon/data_grid.py", line 218, in get_data_frame

headers = [header['displayName'] for header in data_dict['headers'][0]]

KeyError: 'headers'

eikoneikon-data-apipythonworkspacerefinitiv-dataplatform-eikonworkspace-data-apikey-error
icon clock
10 |1500

Up to 2 attachments (including images) can be used with a maximum of 512.0 KiB each and 1.0 MiB total.

Upvotes
Accepted
39.4k 78 11 27

The issue with the Web service returning empty response or "Backend error. 400 Bad Request" has been addressed with the release of Eikon Data APIs library for Python v1.0.1 available on PyPI. This does not mean that with Eikon Data APIs library for Python v1.0.1 one can never receive "Bad request" error or that the service can now provide unlimited data downloads. But one should no longer experience random and frequent empty responses or "Backend error. 400 Bad Request" forcing one to resubmit the same requests in a loop until the data is returned.

icon clock
10 |1500

Up to 2 attachments (including images) can be used with a maximum of 512.0 KiB each and 1.0 MiB total.

Thanks Alex. I see this behavior fixed on Eikon 1.0.1. Note the new async ticket system introduced in 1.0.1 does appear to have a corner-case bug on Python 2.7 (https://community.developers.refinitiv.com/questions/44621/view.html), that occurs only when the request generates a delay ticket from DataGrid and the eventual response contains non-ASCII Unicode characters. Python 3+ is unaffected.

Upvotes
4.6k 27 7 22

@jr123 looks like it returns an empty dict without returning an error, difficult to say without more data. Try re-executing it with either debug set to True, or as a raw_output set to True.

icon clock
10 |1500

Up to 2 attachments (including images) can be used with a maximum of 512.0 KiB each and 1.0 MiB total.

Upvotes
6 2 4 3

I'm getting the same thing but seems to happen randomly (i.e. running the same code and sometimes it works sometimes it fails). Would be great to understand what it is and why it is happening!

icon clock
10 |1500

Up to 2 attachments (including images) can be used with a maximum of 512.0 KiB each and 1.0 MiB total.

@jr123
Since we're unable to reproduce the issue we have to rely on you to troubleshoot it. I would suggest using Fiddler or another HTTP analyzer to capture the HTTP traffic and see what response you get from the Web service providing the data that results in the exception you experienced. We'd be very grateful if you could share your findings on this thread.

Upvotes
79 5 9 8

@Zhenya Kovalyov Hello! Same problem with get_data. With debug=True raise Error 408:


icon clock
10 |1500

Up to 2 attachments (including images) can be used with a maximum of 512.0 KiB each and 1.0 MiB total.

@oalexandere thanks, I have raised this with the product owner.

So, it seems that the call times out, I would suggest repeating the call.

There's a 60 seconds time out by default.

You can extend to a longer duration with the function ek.set_timeout(timeout) function.

if it is a timeout, then why doesn't it error accordingly in the console? the KeyError headers is quite confusing...

This case (timeout) is different than the first issue (empty result).

Upvotes
4.3k 2 4 5

This exception means that returned data_dict is empty ('headers' key error is raised)

It reproduced the case :

>>> ek.get_data("PEUP.PA", "ASK", raw_output=True)
{}
>>>

It isn't expected at all and the root cause is on the back-end side.

icon clock
10 |1500

Up to 2 attachments (including images) can be used with a maximum of 512.0 KiB each and 1.0 MiB total.

Upvote
39.4k 78 11 27
@574c708d-6b25-445f-8303-2b495473ec99

, @jr123, @oalexandere

Are you still experiencing either the timeout or the empty results issue?

icon clock
10 |1500

Up to 2 attachments (including images) can be used with a maximum of 512.0 KiB each and 1.0 MiB total.

Hi - I'm not sure bc I creater a wrapper to automatically handle timeouts by breaking down the query. will let you know if I run into problems though.

@Alex Putkov. I can confirm this behavior is still occurring. On our side the incidence coincided roughly with the auto-installation of Eikon 4.0.46, though I don't know enough about the architecture to know if it's the Eikon desktop application itself that changed or a change at the headend that was rolled out at the same time.

@davidk
Are you referring to the empty results? Do you have an example of the call that still returns empty results? I can no longer reproduce the issue using ek.get_data("PEUP.PA", "ASK", raw_output=True), which @pierre.faurel previously successfully used to reproduce the problem.

Try asking for an large number of RICs (below, rics is a list of 3000 US rics and I slice out the top 500), so you can induce a timeout:

In [14]: ek.get_data(rics[:500], ['TR.InvestorFullName', 'TR.InvestorType', 'TR.InvAddrCountry', 'TR.InvInvestmentStyleCode', 'TR.In vInvmtOrientation', 'TR.SharesHeldValue', 'TR.SharesHeldValue.date', 'TR.FilingType'], {'sdate': '2018-12-31', 'edate': '2018-12- 31'}, raw_output=True)

Out[14]: {}

Show more comments

@Alex Putkov. Where can I follow the service case #07236642?
I think I'm getting similar errors when I run get_data(screen(contains(name))) for a large list of names.

Upvote
39.4k 78 11 27

@davidk and @dshi

Just to update you, our development team is looking to implement a fix that would return an error in response to requests similar to the example @davidk provided instead of an empty JSON the Web service returns now. This will make user experience better, as you will see a more meaningful error message compared to what the current KeyError exception provides. We're not however looking to fulfill such enormous requests. If we were to do this, the fulfilment time would be in minutes and the payload would be in Gigabytes. We don't think the bulk download capability is appropriate for an end user terminal product, nor do we think Eikon infrastructure could reliably support this at scale. So, in order to retrieve this data you'll still need to break down the list of RICs into much smaller chunks and use only a small handful of RICs (or perhaps even just one RIC) in each request.

icon clock
10 |1500

Up to 2 attachments (including images) can be used with a maximum of 512.0 KiB each and 1.0 MiB total.

Understood Alex. The better error message is really what we're looking for here. Note my sample request was very large, but I made it such to reliably and repeatably induce a timeout-- I think the original behavior reported indicates that timeouts are an occasional fact of life even on smaller requests (such is the Internet, esp. when connecting via HTTPS with corporate middleboxes in the way, etc. etc.)

Upvotes
58 5 4 8

Ok, Thanks @Alex Putkov.

icon clock
10 |1500

Up to 2 attachments (including images) can be used with a maximum of 512.0 KiB each and 1.0 MiB total.

Upvotes
244 15 14 17

@Alex Putkov.

The problem has apparently gotten worse after the recent DataGrid deployment. Reconstruction below using a simple EPS data request that should generate a payload of under 60 KB:

import eikon as ek
fields = ['TR.PriceTargetMean', 'TR.PriceTargetStdDev', 'TR.PriceTargetNumOfEstimates', 'TR.PriceTargetNumIncEstimates', 'TR.RecMean', 'TR.NumOfRecommendations', 'TR.EPSMean', 'TR.EPSStdDev', 'TR.EPSNumberofEstimates', 'TR.EPSNumIncEstimates', 'TR.EPSPctNumEstRevisingDown(WP=1d)', 'TR.EPSPctNumEstRevisingUp(WP=1d)', 'TR.EPSMean.periodenddate'] 
rics = pd.Index(ek.get_data('0#.SPX', 'TR.RIC')[0]['RIC']).tolist() 
# Returns empty dict
ek.get_data(rics, fields, parameters={'Period':'FY1'}, raw_output=True)
# returns correctly on the first 100 securities, payload is 11,120 bytes

output = ek.get_data(rics[:100], fields, parameters={'Period':'FY1'}, raw_output=True)

I have followed up with support on case [ ref:_00D30602X._5001W1Py1dr:ref ] as well, but thought you and the Data API team should know. These increasing errors are making it unnecessarily difficult to use the Eikon Data API for any but the smallest requests.

icon clock
10 |1500

Up to 2 attachments (including images) can be used with a maximum of 512.0 KiB each and 1.0 MiB total.

@davidk
The new version of Data Grid, which we believe will alleviate the issue with empty responses, has not been deployed into production yet. Any variations in the quality of service you've seen are due to random factors. We expect the new version of Data Grid to be deployed into production next month.

Customer support emailed me on March 6 to say that the DataGrid folks told them this was fixed in the v1.4 deploy on Feb 28, but clearly it wasn't fixed; perhaps v1.4 wasn't actually deployed?

Correct. Customer support must have made a mistake. Data Grid 1.4 has not been deployed in production. It is expected to be deployed next month.

@Alex Putkov. Has DataGrid 1.4 been deployed? If not, do you know what the ETA is? We're still seeing KeyError 'headers' but have also seen some new errors crop up in the last few days.

Data Grid 1.4 should be targeted to external users in production later today or tomorrow. What you should expect to see is eikon Python library raising eikon.EikonError exception "Backend error. 400 Bad Request" where it previously raised KeyError exception due to empty response from Data Grid. In other words instead of an empty response Data Grid will return an error message.

Show more comments

Write an Answer

Hint: Notify or tag a user in this post by typing @username.

Up to 2 attachments (including images) can be used with a maximum of 512.0 KiB each and 1.0 MiB total.