Upgrade from Eikon -> Workspace. Learn about programming differences.

For a deeper look into our Eikon Data API, look into:

Overview |  Quickstart |  Documentation |  Downloads |  Tutorials |  Articles

question

Upvotes
Accepted
3 0 2 7

Continuously getting backend error and internal server error.

Hi, I have around 6000 RICS and i want to fetch the earnings data for those RICS on daily basis. I'm using eikon. But continuously I'm getting backend error and internal server error. and even if I ran it on the for all 6000 RICS I'm getting the data for only in between 3 to 3.5K. I'm tried giving the time sleep of 2 sec or 1 sec but that also not works. How can I solve this issue. Is Eikon is capable of fetching the data for all the RICS or is there any another solution for this. Because I need to fetch the data for all on daily basis (each business day)

#productdeveloper-communitysupporthelpdesk
icon clock
10 |1500

Up to 2 attachments (including images) can be used with a maximum of 512.0 KiB each and 1.0 MiB total.

1 Answer

· Write an Answer
Upvotes
Accepted
4.3k 11 2 7

Hi @vishnu01 ,


This caused because of the service overload. You may try to make the request in chunks (e.g 1000 RICs in chunk) wrapped by try except statement so your code will retry in case of the error. There is an example in this thread which you may find useful.


Best regards,

Haykaz

icon clock
10 |1500

Up to 2 attachments (including images) can be used with a maximum of 512.0 KiB each and 1.0 MiB total.

Hi @aramyan.h


Thank you for the response , I'm attempting to get the market cap and I'm using the code below to do so. I get an HTTP timeout exception if I increase the chunk size , and backend error if I provide one RIC at a time.

I frequently receive the backend error even when I train on small list of RIC's.

Our goal is to setup the regular tasks to collect the data like market cap, earnings, merge and acquisitions, estimates and numerous others factors.

So how can I get the data without the errors ?


chunk_size = 50

merged_df = pd.DataFrame()


try:

for i in range(0, len(Final_df), chunk_size):

rics_chunk = Final_df['ric'].iloc[i:i + chunk_size].tolist()

df, err = ek.get_data(rics_chunk, fields=['TR.MarketCapLocalCurn', 'TR.MarketCapLocalCurn.date'],

parameters={'SDate': '2010-01-01', 'EDate': '2023-11-07', 'Frq': 'D'})

merged_chunk = pd.merge(df, Final_df.iloc[i:i + chunk_size], left_on='Instrument', right_on='ric', how='left')

merged_df = pd.concat([merged_df, merged_chunk])

break


except Exception as e:

print(f"Error occurred: {e}")

Hi @vishnu01 ,


have you looked at the thread I have shared? there is an example how to do that. I have tried to midify your code, you may try it and see if works:

chunk_size = 50

merged_df = pd.DataFrame()

for i in range(0, len(Final_df), chunk_size):
    retries = 0
    rics_chunk = Final_df['ric'].iloc[i:i + chunk_size].tolist()
    while retries < 3:
        try:
            df, err = ek.get_data(rics_chunk, fields=['TR.MarketCapLocalCurn', 'TR.MarketCapLocalCurn.date'],
            parameters={'SDate': '2010-01-01', 'EDate': '2023-11-07', 'Frq': 'D'})
            merged_chunk = pd.merge(df, Final_df.iloc[i:i + chunk_size], left_on='Instrument', right_on='ric', how='left')
            merged_df = pd.concat([merged_df, merged_chunk])
        except Exception as e:
            print(f"Error occurred: {e}")
            retries +=1
            continue
        break


Best regards,

Haykaz

Hi @aramyan.h,

When I attempted to use the code you provided above, Eikon gave me two errors. I tried reducing the chunk size all the way to 10, but the problem persisted.

Error code 408 | HTTP TimeoutException and "code":504,"message":"A remote server did not reply in a timely fashion.

Hi @vishnu01 ,


I think your issue is coming from the line merged_chunk = pd.merge(df, Final_df.iloc[i:i + chunk_size], left_on='Instrument', right_on='ric', how='left'), not the API.

I have commented out that line and have tested with chunk 100 and it works fine:

screenshot-2023-11-21-at-102304.png

Also I have indented the break that should have been inside while (updated in the code above)


Best regards,

Haykaz

Hi, @aramyan.h I tried your suggestions and even when I reduce the size to 50 and apply the same logic, I still receive the same error. Are these errors connected to server or my eikon desktop? I'm running eikon version 4.0.64 and Python version 3.11.3. When I use the chunk_size=100 it gives me the HTTP timeout exception.

screenshot-2023-11-22-141414.png

Show more comments

Write an Answer

Hint: Notify or tag a user in this post by typing @username.

Up to 2 attachments (including images) can be used with a maximum of 512.0 KiB each and 1.0 MiB total.