getting KeyError: 'headers' from data request

Hi, I am using the ek.get_data() function to request data, which often works but sometimes gives the following error. What is causing this and how can I resolve it? Thanks.

ERROR:

Traceback (most recent call last):

File "/usr/local/lib/python3.7/site-packages/eikon/data_grid.py", line 169, in get_data

return get_data_frame(result, field_name)

File "/usr/local/lib/python3.7/site-packages/eikon/data_grid.py", line 218, in get_data_frame

headers = [header['displayName'] for header in data_dict['headers'][0]]

KeyError: 'headers'

Find more posts tagged with

python

eikon

workspace

eikon-data-api

refinitiv-dataplatform-eikon

workspace-data-api

key-error

Accepted answers

Alex Putkov.1

The issue with the Web service returning empty response or "Backend error. 400 Bad Request" has been addressed with the release of Eikon Data APIs library for Python v1.0.1 available on PyPI. This does not mean that with Eikon Data APIs library for Python v1.0.1 one can never receive "Bad request" error or that the service can now provide unlimited data downloads. But one should no longer experience random and frequent empty responses or "Backend error. 400 Bad Request" forcing one to resubmit the same requests in a loop until the data is returned.

All comments

Zhenya Kovalyov

@jr123 looks like it returns an empty dict without returning an error, difficult to say without more data. Try re-executing it with either debug set to True, or as a raw_output set to True.

reid

I'm getting the same thing but seems to happen randomly (i.e. running the same code and sometimes it works sometimes it fails). Would be great to understand what it is and why it is happening!

Alex Putkov.1

@jr123
Since we're unable to reproduce the issue we have to rely on you to troubleshoot it. I would suggest using Fiddler or another HTTP analyzer to capture the HTTP traffic and see what response you get from the Web service providing the data that results in the exception you experienced. We'd be very grateful if you could share your findings on this thread.

oalexandere

@Zhenya Kovalyov Hello! Same problem with get_data. With debug=True raise Error 408:

screenshot-2018-12-21-145127.png

Zhenya Kovalyov

@oalexandere thanks, I have raised this with the product owner.

So, it seems that the call times out, I would suggest repeating the call.

There's a 60 seconds time out by default.

You can extend to a longer duration with the function ek.set_timeout(timeout) function.

This exception means that returned data_dict is empty ('headers' key error is raised)

It reproduced the case :

>>> ek.get_data("PEUP.PA", "ASK", raw_output=True)
{}
>>>

It isn't expected at all and the root cause is on the back-end side.

reid

if it is a timeout, then why doesn't it error accordingly in the console? the KeyError headers is quite confusing...

Alex Putkov.1

@574c708d-6b25-445f-8303-2b495473ec99_deprecated_6f93d9f8-9d25-487c-91bf-b3a796f48f40

, @jr123, @oalexandere

Are you still experiencing either the timeout or the empty results issue?

reid

Hi - I'm not sure bc I creater a wrapper to automatically handle timeouts by breaking down the query. will let you know if I run into problems though.

davidk

@Alex Putkov.1 I can confirm this behavior is still occurring. On our side the incidence coincided roughly with the auto-installation of Eikon 4.0.46, though I don't know enough about the architecture to know if it's the Eikon desktop application itself that changed or a change at the headend that was rolled out at the same time.

Alex Putkov.1

@davidk
Are you referring to the empty results? Do you have an example of the call that still returns empty results? I can no longer reproduce the issue using ek.get_data("PEUP.PA", "ASK", raw_output=True), which @pierre.faurel previously successfully used to reproduce the problem.

davidk

Try asking for an large number of RICs (below, rics is a list of 3000 US rics and I slice out the top 500), so you can induce a timeout:

In [14]: ek.get_data(rics[:500], ['TR.InvestorFullName', 'TR.InvestorType', 'TR.InvAddrCountry', 'TR.InvInvestmentStyleCode', 'TR.In
vInvmtOrientation', 'TR.SharesHeldValue', 'TR.SharesHeldValue.date', 'TR.FilingType'], {'sdate': '2018-12-31', 'edate': '2018-12-
31'}, raw_output=True)

Out[14]: {}

dshi

@Alex Putkov.1 Where can I follow the service case #07236642?
I think I'm getting similar errors when I run get_data(screen(contains(name))) for a large list of names.

Alex Putkov.1

@davidk and @dshi

Just to update you, our development team is looking to implement a fix that would return an error in response to requests similar to the example @davidk provided instead of an empty JSON the Web service returns now. This will make user experience better, as you will see a more meaningful error message compared to what the current KeyError exception provides. We're not however looking to fulfill such enormous requests. If we were to do this, the fulfilment time would be in minutes and the payload would be in Gigabytes. We don't think the bulk download capability is appropriate for an end user terminal product, nor do we think Eikon infrastructure could reliably support this at scale. So, in order to retrieve this data you'll still need to break down the list of RICs into much smaller chunks and use only a small handful of RICs (or perhaps even just one RIC) in each request.

dshi

Ok, Thanks @Alex Putkov.1

davidk

Understood Alex. The better error message is really what we're looking for here. Note my sample request was very large, but I made it such to reliably and repeatably induce a timeout-- I think the original behavior reported indicates that timeouts are an occasional fact of life even on smaller requests (such is the Internet, esp. when connecting via HTTPS with corporate middleboxes in the way, etc. etc.)

davidk

@Alex Putkov.1

The problem has apparently gotten worse after the recent DataGrid deployment. Reconstruction below using a simple EPS data request that should generate a payload of under 60 KB:

import eikon as ek

fields = ['TR.PriceTargetMean', 'TR.PriceTargetStdDev', 'TR.PriceTargetNumOfEstimates', 'TR.PriceTargetNumIncEstimates', 'TR.RecMean', 'TR.NumOfRecommendations', 'TR.EPSMean', 'TR.EPSStdDev', 'TR.EPSNumberofEstimates', 'TR.EPSNumIncEstimates', 'TR.EPSPctNumEstRevisingDown(WP=1d)', 'TR.EPSPctNumEstRevisingUp(WP=1d)', 'TR.EPSMean.periodenddate']

rics = pd.Index(ek.get_data('0#.SPX', 'TR.RIC')[0]['RIC']).tolist()

# Returns empty dict
ek.get_data(rics, fields, parameters={'Period':'FY1'}, raw_output=True)

# returns correctly on the first 100 securities, payload is 11,120 bytes

output = ek.get_data(rics[:100], fields, parameters={'Period':'FY1'}, raw_output=True)

I have followed up with support on case [ ref:_00D30602X._5001W1Py1dr:ref ] as well, but thought you and the Data API team should know. These increasing errors are making it unnecessarily difficult to use the Eikon Data API for any but the smallest requests.

Alex Putkov.1

@davidk
The new version of Data Grid, which we believe will alleviate the issue with empty responses, has not been deployed into production yet. Any variations in the quality of service you've seen are due to random factors. We expect the new version of Data Grid to be deployed into production next month.

davidk

Customer support emailed me on March 6 to say that the DataGrid folks told them this was fixed in the v1.4 deploy on Feb 28, but clearly it wasn't fixed; perhaps v1.4 wasn't actually deployed?

Alex Putkov.1

Correct. Customer support must have made a mistake. Data Grid 1.4 has not been deployed in production. It is expected to be deployed next month.

davidk

@Alex Putkov.1 Has DataGrid 1.4 been deployed? If not, do you know what the ETA is? We're still seeing KeyError 'headers' but have also seen some new errors crop up in the last few days.

Alex Putkov.1

Data Grid 1.4 should be targeted to external users in production later today or tomorrow. What you should expect to see is eikon Python library raising eikon.EikonError exception "Backend error. 400 Bad Request" where it previously raised KeyError exception due to empty response from Data Grid. In other words instead of an empty response Data Grid will return an error message.

Alex Putkov.1

However our testing showed that contrary to expectations Data Grid 1.4 release did not improve the success rate of regular requests like a dozen fields for a few hundred stocks. This is very disappointing and we're working on addressing this issue as a top priority item.

Alex Putkov.1

davidk

Thanks Alex. I see this behavior fixed on Eikon 1.0.1. Note the new async ticket system introduced in 1.0.1 does appear to have a corner-case bug on Python 2.7 (https://community.developers.refinitiv.com/questions/44621/view.html), that occurs only when the request generates a delay ticket from DataGrid and the eventual response contains non-ASCII Unicode characters. Python 3+ is unaffected.

EXPLORE OUR SITES