screening with a loop and throwing fatal errors

Hello! I am querying Refinitiv Workspace for equities based on a screen. I used to run this screen, or ones just like it, with no problem, even while using the same Python 3.11.4 that I'm using now. But now it's throwing fatal errors and not always at the same place in the loop. When I take the specific screen text that threw the error in the loop and run it on its own, it returns results with multiple companies. So the underlying screener query wouldn't seem to be a problem. So what is wrong with my below code?

import refinitiv.data as rd
rd.open_session()
import pandas as pd

indcode = [50,51,52,53,54,55,56,57,59,60,62,63]
WEurope = ["AT,BE,DK,FO,FI,FR,DE,GI,GL,GG,IS,IE,IM,IT,JE,LI,LU,MC,NL,NO,PT,SM,ES,SE,CH,GB"]
NAmerica = ["US,CA"]
CNJP = ["CN,JP"]
region = WEurope + NAmerica + CNJP

CATfields = ['TR.CommonName','TR.HeadquartersCountry','TR.HQCountryCode']
syntax = 'SCREEN(U(IN(Equity(primary,public))),\
   IN(TR.TRBCEconSectorCode,{j}), IN(TR.HQCountryCode,{i}),\
   TR.TRESGScore(Period=FY0)>0)'

dataCAT2 = []
df_CAT_list = []
for j in range(len(indcode)):
   for i in range(len(region)):
      dataCAT1 = rd.get_data(syntax.format(i=region[i],j=indcode[j]),CATfields)
      dataCAT2.append(dataCAT1)
      df1 = pd.concat(dataCAT2)
print(df1)

Very oddly, I can get the screen to run successfully in my global environment rather than virual (i.e., the fatal errors are only thrown when I'm in the virtual environment). This makes me wonder whether I have some package inconsistency error -- but having just installed pandas and refinitiv-data today, that wouldn't seem to be the problem.

What do you think is wrong with my code?

Find more posts tagged with

python

screener

workspace

#technology

Accepted answers

andrii.sidachenko01

Hi, @LRE42 ,

For the use case with region = "GB" server returns empty data with code 200.

{"responses":[{"columnHeadersCount":0,"data":[],"rowHeadersCount":0,"totalColumnsCount":0,"totalRowsCount":0}]}

There is difference between RD versions you run in global and local env.

In RD 1.0 we have the logic to log errors but not raised the exceptions at all.

In RD 1.4.0 we have approach we decided to let users know about server-side issues.

- raise no exception in case when any of requests under the hood of get_history() at least have partial data -> build Dataframe.

- raise an exception if all requests failed;

- raise an exc if it was only one request (for example to Datagrid endpoint) and it failed.

In current case we raised an exception because datagrid platform returned response without header, no possible to build Dataframe with data.

In the next release we will make a fix to build empty dataframe that avoid exceptions in case of headers and data absence.

I will let you know about the new version.

Thank you.

All comments

jonathan.legrand

Hi @LRE42 ,

May I ask: (i) what IDE are you using to run all this (e.g.: CodeBook, VSCode, ...), (ii) what session are you using (info on this here) (guessing from the code above, in your question, I think you're using the desktop session and that's causing the issue), (iii) how does your virtual environment (e.g.: with VSCode) limit proxy usage (I believe that's where the issue is and your virtual env. is limiting access to the desktop application proxies needed for authentication to LSEG API services in desktop session mode), and (iv) what error do you get exactly?

LRE42

Hi @jonathan.legrand -- thanks for responding. Here I will answer your questions:

i. VSCode is the IDE

ii I am using desktop session

iii I am using a virtual environment with VSCode. I don't know what limits it has. Would this be an issue with my local network? Something I noticed after asking my initial question is that while my virtual environment has RD 1.4, my global environment has RD 1.0, which may be relevant.

iv Here is the full text of the error message:

RDError                                   Traceback (most recent call last)
Cell In[1], line 20
     18 for j in range(len(indcode)):
     19    for i in range(len(region)):
---> 20       dataCAT1 = rd.get_data(syntax.format(i=region[i],j=indcode[j]),CATfields)
     21       dataCAT2.append(dataCAT1)
     22       df1 = pd.concat(dataCAT2)

File [local address redacted]\.venv\Lib\site-packages\refinitiv\data\_access_layer\get_data_func.py:98, in get_data(universe, fields, parameters, use_field_names_in_headers)
     96 if exceptions and all(exceptions):
     97     except_msg = "\n\n".join(exceptions)
---> 98     raise RDError(-1, except_msg)
    100 hp_and_cust_inst_data = HPAndCustInstDataContainer(stream_columns, stream_data, stream_df)
    101 adc_data = ADCDataContainer(adc_raw, adc_df, fields)

RDError: Error code -1 | Backend error. 400 Bad Request Requested universes: ['SCREEN(U(IN(Equity(primary,public))),   IN(TR.TRBCEconSectorCode,54), IN(TR.HQCountryCode,CN,JP),   TR.TRESGScore(Period=FY0)>0)']. Requested fields: ['TR.COMMONNAME', 'TR.HEADQUARTERSCOUNTRY', 'TR.HQCOUNTRYCODE']

LRE42

Hi @jonathan.legrand , I accidentally posted as an answer instead of a comment - I hope you'll still see my response.

jonathan.legrand

Hi @LRE42, Are you able to use any rd.get_data call at all in your virtual environment at all? I have the impression that your VSCode environment blocks incoming data in packages larger than a specific amount. This could be a setting in VSCode maybe?

LRE42

Hi @jonathan.legrand, thanks again. To answer your question, yes, I am able to download data for several loop iterations. It's not always the same loop iteration that throws the fatal error.

Below I show some code. First I ran it with the long list of countries, and then I ran it with just GB. Both of these failed. I would think that the combination of one country and industry code wouldn't produce "too" much data, but that one also threw the same error.

import refinitiv.data as rd
rd.open_session()
import pandas as pd

CATfields = ['TR.CommonName','TR.HeadquartersCountry','TR.HQCountryCode']
syntax = 'SCREEN(U(IN(Equity(primary,public))),\
   IN(TR.TRBCEconSectorCode,{j}), IN(TR.HQCountryCode,{i}),\
   TR.TRESGScore(Period=FY0)>0)'

region = "AT,BE,DK,FO,FI,FR,DE,GI,GL,GG,IS,IE,IM,IT,JE,LI,LU,MC,NL,NO,PT,SM,ES,SE,CH,GB" # ran this one first
region = "GB" # ran this one second
indcode = "62"
df2 = rd.get_data(syntax.format(i=region,j=indcode),CATfields)
print(df2)

(I wasn't able to post this as a comment so trying now as an answer)

andrii.sidachenko01

Hi, @LRE42 ,

For the use case with region = "GB" server returns empty data with code 200.

{"responses":[{"columnHeadersCount":0,"data":[],"rowHeadersCount":0,"totalColumnsCount":0,"totalRowsCount":0}]}

There is difference between RD versions you run in global and local env.

In RD 1.0 we have the logic to log errors but not raised the exceptions at all.

In RD 1.4.0 we have approach we decided to let users know about server-side issues.

- raise no exception in case when any of requests under the hood of get_history() at least have partial data -> build Dataframe.

- raise an exception if all requests failed;

- raise an exc if it was only one request (for example to Datagrid endpoint) and it failed.

In current case we raised an exception because datagrid platform returned response without header, no possible to build Dataframe with data.

In the next release we will make a fix to build empty dataframe that avoid exceptions in case of headers and data absence.

I will let you know about the new version.

Thank you.

LRE42

Thank you. I have installed v1.0 in my virtual environment and was able to run screening queries without any fatal errors. So that has solved my problem for now. It will be good to have the updated version when this is fixed in the next RD release. Thank you

EXPLORE OUR SITES