Continuously getting backend error and internal server error.

vishnu01

Hi, I have around 6000 RICS and i want to fetch the earnings data for those RICS on daily basis. I'm using eikon. But continuously I'm getting backend error and internal server error. and even if I ran it on the for all 6000 RICS I'm getting the data for only in between 3 to 3.5K. I'm tried giving the time sleep of 2 sec or 1 sec but that also not works. How can I solve this issue. Is Eikon is capable of fetching the data for all the RICS or is there any another solution for this. Because I need to fetch the data for all on daily basis (each business day)

aramyan.h

Hi @vishnu01 ,

This caused because of the service overload. You may try to make the request in chunks (e.g 1000 RICs in chunk) wrapped by try except statement so your code will retry in case of the error. There is an example in this thread which you may find useful.

Best regards,

Haykaz

vishnu01

https://community.developers.refinitiv.com/discussion/comment/111137#Comment_111137

Hi @aramyan.h

Thank you for the response , I'm attempting to get the market cap and I'm using the code below to do so. I get an HTTP timeout exception if I increase the chunk size , and backend error if I provide one RIC at a time.

I frequently receive the backend error even when I train on small list of RIC's.

Our goal is to setup the regular tasks to collect the data like market cap, earnings, merge and acquisitions, estimates and numerous others factors.

So how can I get the data without the errors ?

chunk_size = 50

merged_df = pd.DataFrame()

try:

for i in range(0, len(Final_df), chunk_size):

rics_chunk = Final_df['ric'].iloc[i:i + chunk_size].tolist()

df, err = ek.get_data(rics_chunk, fields=['TR.MarketCapLocalCurn', 'TR.MarketCapLocalCurn.date'],

parameters={'SDate': '2010-01-01', 'EDate': '2023-11-07', 'Frq': 'D'})

merged_chunk = pd.merge(df, Final_df.iloc[i:i + chunk_size], left_on='Instrument', right_on='ric', how='left')

merged_df = pd.concat([merged_df, merged_chunk])

break

except Exception as e:

print(f"Error occurred: {e}")

aramyan.h

https://community.developers.refinitiv.com/discussion/comment/111137#Comment_111137

Hi @vishnu01 ,

have you looked at the thread I have shared? there is an example how to do that. I have tried to midify your code, you may try it and see if works:

chunk_size = 50

merged_df = pd.DataFrame()

for i in range(0, len(Final_df), chunk_size):
    retries = 0
    rics_chunk = Final_df['ric'].iloc[i:i + chunk_size].tolist()
    while retries < 3:
        try:
            df, err = ek.get_data(rics_chunk, fields=['TR.MarketCapLocalCurn', 'TR.MarketCapLocalCurn.date'],
            parameters={'SDate': '2010-01-01', 'EDate': '2023-11-07', 'Frq': 'D'})
            merged_chunk = pd.merge(df, Final_df.iloc[i:i + chunk_size], left_on='Instrument', right_on='ric', how='left')
            merged_df = pd.concat([merged_df, merged_chunk])
        except Exception as e:
            print(f"Error occurred: {e}")
            retries +=1
            continue
        break

Best regards,

Haykaz

vishnu01

https://community.developers.refinitiv.com/discussion/comment/111262#Comment_111262

Hi @aramyan.h,

When I attempted to use the code you provided above, Eikon gave me two errors. I tried reducing the chunk size all the way to 10, but the problem persisted.

Error code 408 | HTTP TimeoutException and "code":504,"message":"A remote server did not reply in a timely fashion.

aramyan.h

https://community.developers.refinitiv.com/discussion/comment/111262#Comment_111262

Hi @vishnu01 ,

I think your issue is coming from the line merged_chunk = pd.merge(df, Final_df.iloc[i:i + chunk_size], left_on='Instrument', right_on='ric', how='left'), not the API.

I have commented out that line and have tested with chunk 100 and it works fine:

Also I have indented the break that should have been inside while (updated in the code above)

Best regards,

Haykaz

vishnu01

https://community.developers.refinitiv.com/discussion/comment/111414#Comment_111414

Hi, @aramyan.h I tried your suggestions and even when I reduce the size to 50 and apply the same logic, I still receive the same error. Are these errors connected to server or my eikon desktop? I'm running eikon version 4.0.64 and Python version 3.11.3. When I use the chunk_size=100 it gives me the HTTP timeout exception.

aramyan.h

https://community.developers.refinitiv.com/discussion/comment/111414#Comment_111414

Hi @vishnu01 ,

Please move break inside the while loop as in my example above