Uncaught UnicodeEncodeError on Python 2 in debug statement

davidk · June 2019

In Eikon Python 1.0.1, at eikon.json_requests line 114, there is a call to logger.debug('HTTP Response:{}'.format(response.text)) that is not wrapped in a try: except UnicodeEncodeError call (as is the case with most other calls, see e.g. lines 90-94). If the HTTP request returns a value with a non-ASCII Unicode value this then causes an uncaught exception to raise in Python 2. In Python 3.6+ this is not an issue as the non-ASCII unicode value is handled by the Python 3 str.format method.

davidk · December 2020

As I noted in the original question, this was not an issue in Python 3.6+. It was only an issue in Python 2 where Unicode had to be explicitly decoded. Since Python 2 went end of life exactly a year ago there is no longer a need to fix this.

chavalit-jintamalit · June 2019

Hi @davidk

I think I can reproduce the issue.

I will consult with my senior colleague and will give you an update.

davidk · June 2019

https://community.developers.refinitiv.com/discussion/comment/44631#Comment_44631

This is not quite a reproduction-- I think you're hitting a different logger line that is wrapped in a try: except: (likely line 92), rather than line 114 (otherwise, you would have received a raised exception). In order to verify this, in a new python window run:

import logging; logging.basicConfig(level=logging.DEBUG)

Before importing the eikon API. You should then see the log messages generated and the line they came from.

In order to hit the error on line 114 you need to have received a ticket from DataGrid. Based on its behavior I believe the ticket is sent if the server warns you that there will be a delay in the response. For a true reproduction therefore you might need to send a particularly large request (say, for a few thousand RICs for several fields).

chavalit-jintamalit · June 2019

https://community.developers.refinitiv.com/discussion/comment/44631#Comment_44631

Hi @davidk

Can you provide the code which you use to reproduce the issue ?

Thanks a lot.

davidk · June 2019

https://community.developers.refinitiv.com/discussion/comment/44693#Comment_44693

I suggest you simply review the source code, it's quite clear what the problem is. All the other logger.debug('HTTP Response: {}'.format(response.text)) calls in json_requests.py are wrapped in try: except UnicodeEncodeError: other than the one on line 114.

Basically if you hit one of the other calls, you get the pyeikon message you received above (if you haven't initialized the root logger) or an error message saying that Unicode parsing failed (if you have initialized the root logger). But if you get a ticket from DataGrid first and THEN hit the Unicode error on line 114, you get an exception and the program blows its stack.

davidk · June 2019

https://community.developers.refinitiv.com/discussion/comment/44719#Comment_44719

Sorry to be cagey about reproduction but reproducing once requires running enough queries to generate a DataGrid ticket and that depends on the server load. We were hitting this error once every 1-2 hours while downloading analyst recommendations for a quant analysis we were running, since many of the analyst's names contain accented Unicode characters. So basically if you really want to reproduce, you can try running this code on a loop for several hours:

import logging

logging.basicConfig(level=logging.DEBUG)

import eikon

rics = eikon.get_data('0#.SPX', 'TR.IndexConstituentRIC')[0].iloc[:, 0].tolist()

fields = ['TR.RecEstValue.value', 'TR.RecEstValue.brokername',
          'TR.RecEstValue.brokerid', 'TR.RecEstValue.analystname',
          'TR.RecEstValue.analystcode', 'TR.RecEstDate']

while True:

   eikon.get_data(rics, fields)

davidk · September 2019

This bug has not been fixed, even though I explained above how to patch the problem (by simply wrapping json_requests.py line 114 in a try: except UnicodeEncodeError: block).This should have been an easy patch to create a 1.0.2. In our firm we have distributed the fix but others should receive it as well. If Eikon wishes not to support bugfixes on python 2.7 given python 2.7's end-of-life on Jan 1, 2020, that is fine, but they should say so if so.