KeyError: 'headlines'

noel.randewich01

Hi,

I'm trying to iterate through several RICs and dates collecting news stories using this, and variations of this.

In the code below, x is a list of anywhere from 3 to 500 RICs.

for y in dates[1::60]:
        sdate = y
        edate = y
        edate += timedelta(days=60)
        str_sdate = sdate.strftime("%d-%b-%Y %H:%M:%S.%f")
        str_edate = edate.strftime("%d-%b-%Y %H:%M:%S.%f")
        q="R:" + x + ' Language:LEN AND NS:RTRS US AND -BRIEF -FACTBOX -IMBALANCE -AA -MOODY -MEDIA-'
        stories = ek.get_news_headlines(q, date_from = str_sdate,  date_to = str_edate, count = 100)

When I run it, it works for anywhere between 1 minute and 20 minutes, giving me a few dozen or a few hundred results, but it always eventually fails with this error code:

---------------------------------------------------------------------------
KeyError                                  Traceback (most recent call last)
<ipython-input-4-772ef7887bd0> in <module>
     14         str_edate = edate.strftime("%d-%b-%Y %H:%M:%S.%f")
     15         q="R:" + x + ' Language:LEN AND NS:RTRS US AND -BRIEF -FACTBOX -IMBALANCE -AA -MOODY -MEDIA-'
---> 16         stories = ek.get_news_headlines(q, date_from = str_sdate,  date_to = str_edate, count = 100)
     17         storycount = stories['versionCreated'].count()
     18 
~\Anaconda3\lib\site-packages\eikon\news_request.py in get_news_headlines(query, count, date_from, date_to, raw_output, debug)
    124         return result
    125     else:
--> 126         return get_data_frame(result)
    127     128 
~\Anaconda3\lib\site-packages\eikon\news_request.py in get_data_frame(json_data)
    133                              'sourceName', 'versionCreated']
    134 --> 135     json_headlines_array = json_data['headlines']
    136     first_created = [tz_replacer(headline['firstCreated']) for headline in json_headlines_array]
    137     headlines = [[headline[field] for field in Headline_Selected_Fields] 
KeyError: 'headlines'

I have tried shortening my query to make it simpler, taking out everything but language, RIC and news source. I have tried inserting a 3 second delay between each iteration in case there is some limit on how often I can make a request, but no matter what I change, I always eventually get the "KeyError: 'headlines'" error.

I see other questions have been asked about "KeyError: Header" but I can't see anything related to "KeyError: Headline".

Thank you

Alex Putkov.1

Every time get_news_headlines method is executed Eikon Data APIs library uses the values provided in the arguments to construct an HTTP request, which is sent to the Web service providing news headlines. The expected response contains JSON, which includes 'headlines' key. You can see it by including raw_output=True kwarg in get_news_headlines method. When 'headlines' keyword is missing in the response, this indicates a failed request. Printing out full response, which you can obtain by including raw_output=True kwarg in get_news_headlines method, may provide a clue why the request failed. Capturing failed request with Fiddler would be even better. This said, an HTTP request may fail for any number of reasons. If what you experience is one request failing out of hundreds, it may not be worth investigating. You could just add defensive code to your script to resubmit the request when it fails. But if you have a high number of requests failing, then it's definitely worth looking into. A Fiddler capture of a failed request would be most helpful. Here's an example of defensive code that resubmits the request when KeyError is encountered.

for y in dates[1::60]:
        sdate = y
        ...
        try:
            stories = ek.get_news_headlines(q, date_from = str_sdate, 
                                            date_to = str_edate, count = 100)
        except KeyError:
            stories = ek.get_news_headlines(q, date_from = str_sdate, 
                                            date_to = str_edate, count = 100)

noel.randewich01

Alex Putkov. I will experiment with the exception code you showed me above.

I really appreciate your patience with all my questions recently.

Tks

Perle

https://community.developers.refinitiv.com/discussion/comment/69390#Comment_69390

Hi @Alex Putkov.1

I ran into the same error, my code worked until yesterday but not anymore. I have also included the defensive code. Do you have any ideas on what the reasons might be? I have the most up to date version of eikon.

Basically I wrote functions with loops to request news just like @noel.randewich01

Thanks in advance!

Alex Putkov.1

https://community.developers.refinitiv.com/discussion/comment/76110#Comment_76110

@Perle

Old threads with accepted answers are not monitored by moderators on this forum. When asking a new question, always create a new thread. If you need to refer to an existing thread, include the link in your post. As for troubleshooting the issue you experienced, see my comments about using Fiddler to capture failed HTTP request or at least printing the full response, which hopefully will provide some info about why the request failed.