A CSV download via the API scheduled every work day fails at least once per week with no apparent...
... cause. How do I fix it?
#WARNING: the Eikon terminal must be running in order to connect to the API
#import packages
import eikon as ek # the Eikon Python wrapper package
import pandas as pd
import numpy as np
import datetime
from datetime import timedelta, date, datetime
from pandas.tseries.offsets import BDay
#connects to Bill's Eikon terminal
ek.set_app_key('XXXX')
#retreive the RICs from Eikon
df_rics,e = ek.get_data("lists('Inv Trust List')","TR.RIC")
#convert that into a list and set as an object
ric_list = df_rics['Instrument'].tolist()
#Slice, loop and concatenate the retrieval request - must be done in order to avoid a backend request timeout which
#happens when we use too many RICs. n can be toggled below.
n = 25
df = pd.DataFrame()
for ric_chunk in [ric_list[i:i + n]
for i in range(0, len(ric_list), n)]:
tmp_df, e = ek.get_data(ric_chunk,
['TR.RNSFilerName',
'TR.RNSAnnouncedDate',
'TR.RNSTransactionType',
'TR.RNSARNumShrsTransacted',
'TR.RNSARPctOSTransacted',
'TR.RNSARTransactionPrice',
'TR.RNSARMktValTransaction',
'TR.RNSARTotShrsPostTrans',
'TR.RNSARPctOSPostTrans'])
df = tmp_df.append(df)
#set boundary dates of the download
end_date = date.today()
start_date = end_date - BDay(2)
#set those dates in the necessary string format
end_date_str = datetime.strftime(end_date, "%Y-%m-%d")
start_date_str = datetime.strftime(start_date, "%Y-%m-%d")
#denote dates in datetime format
df['RNS Announced Date'] = pd.to_datetime(df['RNS Announced Date'])
#set the date constraints as an object
mask = (df['RNS Announced Date'] >= start_date_str) & (df['RNS Announced Date'] < end_date_str)
#set the dataframe as only those with values inside the boundary dates
df = df.loc[mask]
#rename some of columns headings, some for compatibility purposes - some programs don't like the '£' symbol
df.rename(columns={'RNS Announced Date': 'Transaction Date','RNS AR Price (at Transaction) - £': 'RNS AR Price (at Transaction) GBP',
'RNS AR Market Value of Transaction - £': 'RNS AR Market Value of Transaction - GBP'},
inplace=True)
#create file name and export as CSV
todays_date = date.today()
todays_date_str = datetime.strftime(todays_date, "%Y%m%d")
df.to_csv('Daily API Download_' + todays_date_str + '.csv')
Best Answer
-
Hi @bill39
The two fixes I proposed are workaround to detect errors and manage it when it's possible (that's what I named "secure your code").
1) Check that session is started and available with this code:
cnx_state = ek.get_desktop_session().get_open_state()
if cnx_state != ek.Session.State.Open:
exit(-1)If the cnx_state is equal to ek.Session.State.Closed, that means Eikon API Proxy is not avalaible (need to restart Eikon ?).
2) Catch 2504 error code to request again
while retry:
try:
tmp_df, e = ek.get_data(ric_chunk, ['TR.RNSFilerName',
...
'TR.RNSARPctOSPostTrans'])
retry = False
except EikonError as e:
if e.code not in [2504]:
retry = False
else:
time.sleep(0.5) # delay retry in 500 msecNote that you can add other error code than 2504 in the list to catch (ex: 400).
0
Answers
-
This is the error
Error code 503 | Server Error: API Proxy is not available
Traceback (most recent call last):
File "daily_api_download.py", line 15, in <module>
df_rics,e = ek.get_data("lists('Inv Trust List')","TR.RIC")
File "C:\Users\william\AppData\Local\Continuum\anaconda3\lib\site-packages\eikon\data_grid.py", line 186, in get_data
result = eikon.json_requests.send_json_request(_endpoint, payload, debug=debug)
File "C:\Users\william\AppData\Local\Continuum\anaconda3\lib\site-packages\eikon\json_requests.py", line 121, in send_json_request
_raise_for_status(response)
File "C:\Users\william\AppData\Local\Continuum\anaconda3\lib\site-packages\eikon\json_requests.py", line 244, in _raise_for_status
raise EikonError(response.status_code, error_msg)
eikon.eikonError.EikonError: Error code 503 | Server Error: API Proxy is not available
0 -
Hello @bill39,
I understand that the cause of the issue is always error 503? And you seem to be seeing the error consistently, as opposite to once or twice? If this is correct, this is not quite the expected behavior.
Please see this relevant previous discussion, the Eikon troubleshooting guide is very often a key tool to determine the specific cause for the issue, hopefully, helpful to you.
Some other relevant factors are: are you on the recent version of Eikon? Are you on the latest version of the Eikon library, as some important fixes have been released in version of 1.1.6.post2. The issues fixed also appear to be tracked to python 3.8+, if you are on python 3.8+, the use of 1.1.6.post2 is recommended.
Hope that this info is of help.
0 -
Yes 503 is always the error. I have python version 3.8.+. For some reason the troubleshooting guide link you have provided directs me to a page that 'cannot be found'. Please would you email me the troubleshooting instructions for error code 503? Thanks0
-
Hello @bill39,
Please try again- the article link should work now.
The dev portal went through a major update today and some links were migrated.
Thanks for letting us know
0 -
The API request ran today when I executed it manually but it didn't run when run by my Task Scheduler early this morning. What could be the cause of the requests sometimes working and sometimes not? And how do I fix it? Thank you
0 -
Hello @bill39,
Does it run via scheduler in general for you? If you schedule it in 1 min, does it run?
Eikon/Refinitiv Workspace desktop needs to be up, running and logged in, Eikon Proxy within it up and running as part of it, for Eikon Data API python to interact with it. If that is not the case, the task will not run successfully. By default, Eikon desktop is logged out at night, do you find yourself logging back in, in the morning?
0 -
If I schedule it in 1 minute it does sometimes work but not always. I.e. some days it will run every time I try and other days it does not. I have also set a task to open the Eikon terminal every morning before the API download so it is always on when the API runs, yes. I also run a macro in an excel file every morning that uses Eikon data and sometimes this fails also. When this macro fails the buyback also fails but there are times when the macro runs and the API download fails.
0 -
Hello @bill39,
As you mention that, for the most part, the scheduled task succeeds, and sometimes it does not, I think it would be helpful to collect any error or info at that time that it fails. Try introducing
import eikon as ek
...
import logging.config
ek.set_log_level(logging.DEBUG)
ek.set_app_key(XXXXXXplease collect and share any info that is logged at the time of the failed call?
Let's see if we can find out more to help us track down the issue.
---
Would like to also note, that the resulting CSV content is subject to Eikon, Eikon Excel and any data obtained via Eikon Data API single user license, to be used by logged in Eikon user only. While sharing of the content would require Enterprise level product and license.
0 -
Noted, thanks. Thanks, I have added the logging lines to my code. Where will the error log be stored? When running the script manually today the error I receive is:
Traceback (most recent call last):
File "daily_api_download.py", line 39, in <module>
'TR.RNSARPctOSPostTrans'])
File "C:\Users\william\AppData\Local\Continuum\anaconda3\lib\site-packages\eikon\data_grid.py", line 186, in get_data
result = eikon.json_requests.send_json_request(_endpoint, payload, debug=debug)
File "C:\Users\william\AppData\Local\Continuum\anaconda3\lib\site-packages\eikon\json_requests.py", line 118, in send_json_request
_check_server_error(result)
File "C:\Users\william\AppData\Local\Continuum\anaconda3\lib\site-packages\eikon\json_requests.py", line 194, in _check_server_error
raise EikonError(int(server_response['ErrorCode']), error_message)
eikon.eikonError.EikonError: Error code 2504 | UDF Core request failed. Gateway Time-out
_______________________________________
Thanks
0 -
Hi,
error code 2504 is something you can catch to retry to send your request.
You can update your code as below:
import time
...
for ric_chunk in [ric_list[i:i + n]
retry = True
while retry:
try:
tmp_df, e = ek.get_data(ric_chunk,
['TR.RNSFilerName',
'TR.RNSAnnouncedDate',
'TR.RNSTransactionType',
'TR.RNSARNumShrsTransacted',
'TR.RNSARPctOSTransacted',
'TR.RNSARTransactionPrice',
'TR.RNSARMktValTransaction',
'TR.RNSARTotShrsPostTrans',
'TR.RNSARPctOSPostTrans'])
retry = False
except EikonError as e:
if e.code not in [2504]:
retry = False
else:
time.sleep(0.5) # delay retry in 500 msec
df = tmp_df.append(df)For the Error code 503 | Server Error: API Proxy is not available, if you continue to reproduce (a fix with eikon 1.1.6.post3 was published on pypi), as you trigger your script to run daily, that's possible that Eikon Proxy turned down.
You can secure your script with :
ek.set_app_key('XXXX')
cnx_state = ek.get_desktop_session().get_open_state()
if cnx_state != ek.Session.State.Open:
exit(-1)0 -
How would you suggest I integrate that into my current code? It already splits my retreival into chunks like this:
n = 10
df = pd.DataFrame()
for ric_chunk in [ric_list[i:i + n]
for i in range(0, len(ric_list), n)]:
tmp_df, e = ek.get_data(ric_chunk,
['TR.RNSFilerName',
'TR.RNSAnnouncedDate',
'TR.RNSTransactionType',
'TR.RNSARNumShrsTransacted',
'TR.RNSARPctOSTransacted',
'TR.RNSARTransactionPrice',
'TR.RNSARMktValTransaction',
'TR.RNSARTotShrsPostTrans',
'TR.RNSARPctOSPostTrans'])
df = tmp_df.append(df)Additionally, what do you mean by "securing my script"? What does that code do?
Thank you
0 -
UPDATE
Hi @zoya.farberov, as explained on another post, I have been logging my errors and there are three scenarios that occur:
1) download works (happens c.25% of the time),
2) download fails because Eikon Terminal is not open at the time of running (happens c.25% of the time) -this is a Windows Task Scheduling error and not one I need help fixing,
3) download fails and I get the error code attached (happens 50% of the time)
Please would you help me resolve scenario 3. The error code that is generated is attached. ThanksError log.zip
0 -
Hi @pierre.faurel,
Are you able to take a look at the follow-up information on the issue and the logs from @bill39?
Thanks
0 -
Hi @bill39,
{"ErrorCode":400,"ErrorMessage":"Backend error. 400 Bad Request"} is coming from the platform. It can't be fixed in python lib but you can adapt your code to catch and manage it with a retry.
Example of retry implementation:
for ric_chunk in [ric_list[i:i + n]
for i in range(0, len(ric_list), n)]:
while True:
try:
tmp_df, e = ek.get_data(ric_chunk,
['TR.RNSFilerName',
'TR.RNSAnnouncedDate',
'TR.RNSTransactionType',
'TR.RNSARNumShrsTransacted',
'TR.RNSARPctOSTransacted',
'TR.RNSARTransactionPrice',
'TR.RNSARMktValTransaction', 'TR.RNSARTotShrsPostTrans', 'TR.RNSARPctOSPostTrans'])
break
except ek.EikonError as err:
if err.code == 400:
time.sleep(0.5)
else:
break
df = tmp_df.append(df)It seems you're using a eikon version older than latest (1.1.7).
If it's confirmed, you can upgrade to 1.1.7 (pip install --upgrade eikon).0 -
Hi @pierre.faurel,
Could you please take a look at the follow-up question?
The client has attached their log to this duplicate question.
Thanks
0 -
Hi @pierre.faurel, please would you help me integrate this into my code? Firstly for part 1), I am getting
AttributeError: module 'eikon' has no attribute 'get_desktop_session'
and secondly I have already incorporated an error catching code to stop error code 400 (which I don't understand either) following guidance from one of your colleagues. I don't know how I should be incorporating the two together, unfortunately now this has gotten beyond my skill level - I'm a stock analyst by trade, not a programmer! Please help. Many thanks. Ideally we would set up a remote session on my computer where we can go through it and amend the code as necessary.
0
Categories
- All Categories
- 3 Polls
- 6 AHS
- 36 Alpha
- 166 App Studio
- 6 Block Chain
- 4 Bot Platform
- 18 Connected Risk APIs
- 47 Data Fusion
- 34 Data Model Discovery
- 690 Datastream
- 1.4K DSS
- 629 Eikon COM
- 5.2K Eikon Data APIs
- 11 Electronic Trading
- 1 Generic FIX
- 7 Local Bank Node API
- 3 Trading API
- 2.9K Elektron
- 1.4K EMA
- 255 ETA
- 559 WebSocket API
- 39 FX Venues
- 15 FX Market Data
- 1 FX Post Trade
- 1 FX Trading - Matching
- 12 FX Trading – RFQ Maker
- 5 Intelligent Tagging
- 2 Legal One
- 25 Messenger Bot
- 3 Messenger Side by Side
- 9 ONESOURCE
- 7 Indirect Tax
- 60 Open Calais
- 280 Open PermID
- 45 Entity Search
- 2 Org ID
- 1 PAM
- PAM - Logging
- 6 Product Insight
- Project Tracking
- ProView
- ProView Internal
- 23 RDMS
- 2K Refinitiv Data Platform
- 717 Refinitiv Data Platform Libraries
- 4 LSEG Due Diligence
- LSEG Due Diligence Portal API
- 4 Refinitiv Due Dilligence Centre
- Rose's Space
- 1.2K Screening
- 18 Qual-ID API
- 13 Screening Deployed
- 23 Screening Online
- 12 World-Check Customer Risk Screener
- 1K World-Check One
- 46 World-Check One Zero Footprint
- 45 Side by Side Integration API
- 2 Test Space
- 3 Thomson One Smart
- 10 TR Knowledge Graph
- 151 Transactions
- 143 REDI API
- 1.8K TREP APIs
- 4 CAT
- 27 DACS Station
- 121 Open DACS
- 1.1K RFA
- 106 UPA
- 194 TREP Infrastructure
- 229 TRKD
- 918 TRTH
- 5 Velocity Analytics
- 9 Wealth Management Web Services
- 95 Workspace SDK
- 11 Element Framework
- 5 Grid
- 19 World-Check Data File
- 1 Yield Book Analytics
- 48 中文论坛
