Optimizing Eikon API Calls for Retrieving Fundamental Data

Hey everyone,
I'm working on a project on a codebook workspace where I need to pull a lot of fundamental data(322 fields) for various companies using their RICs via the Refinitiv Eikon API. The problem I'm facing is that the process is taking way too long, and I'm frequently running into gateway timeouts (Error code 2504).
- Long Processing Time: The entire process is very slow.
- Gateway Timeouts: Getting frequent gateway timeouts (Error code 2504), especially with larger batches.
EXPORT_FIELDS = [
"TR.F.TotRevPerShr(Curn=USD)",
"TR.F.SalesOfGoodsSrvcNet(Curn=USD)",
"TR.Revenue(Curn=USD)",
"TR.F.COGS(Curn=USD)",
"TR.F.COGSUNCLASSIF(Curn=USD)",
"TR.F.OpExpnTot(Curn=USD)",
"TR.F.SGATot(Curn=USD)",
"TR.F.EBITDANorm(Curn=USD)",
"TR.F.DeprDeplAmortTot(Curn=USD)",
"TR.F.RnD(Curn=USD)",
# 311 more fields...
]
ek.set_app_key('DEFAULT_CODE_BOOK_APP_KEY')
def process_single_company(company):
max_attempts = 4
attempts = 0
while attempts < max_attempts:
try:
print(f"Fetching data for {company} (Attempt {attempts + 1})")
fundamentals_data, e = ek.get_data(
instruments=[company],
fields=["TR.RevenueActReportDate", "TR.F.StatementDate", *EXPORT_FIELDS],
parameters={
'SDate': '1940-01-01',
'EDate': datetime.now().strftime("%Y-%m-%d"),
'Frq': 'Y'
}
)
if e:
print(f"Error fetching data for {company}: {e}")
time.sleep(2) # Pause before retry
attempts += 1
continue
print(f"Saving data for {company}")
fundamentals_data.to_csv(file_pat
print(f"Finished processing for {company}. Moving to the next RIC.\n")
return
except Exception as e:
print(f"Failed to get market cap for {company} due to: {tb.format_exc()}")
attempts += 1
time.sleep(2)
for company in COMPANY:
process_single_company(company)
I tried:
- Batching the requests to process multiple companies or multiple fields at once.
- Processing fields one by one to handle local errors.
- Implementing rate limiting.
How can I speed up the process or maybe I am missing something and there is a better way to handle this amount of data? Or maybe any tips on how to better handle large volumes of data without hitting timeouts?
Thank you!
Best Answer
-
Thank you for reaching out to us.
The request can be timed out by the server. Typically the timed out can happen when the application requests a lot of data in one request. To avoid this issue, the application can reduce the number of items, fields, or date range in each request.
Moreover, you can contact your LSEG account team or sales team directly to discuss this kind of usage. The sales team may be able to suggest other methods that are more suitable for your requirements.
0
Categories
- All Categories
- 3 Polls
- 6 AHS
- 36 Alpha
- 166 App Studio
- 6 Block Chain
- 4 Bot Platform
- 18 Connected Risk APIs
- 47 Data Fusion
- 34 Data Model Discovery
- 684 Datastream
- 1.4K DSS
- 615 Eikon COM
- 5.2K Eikon Data APIs
- 10 Electronic Trading
- Generic FIX
- 7 Local Bank Node API
- 3 Trading API
- 2.9K Elektron
- 1.4K EMA
- 249 ETA
- 554 WebSocket API
- 37 FX Venues
- 14 FX Market Data
- 1 FX Post Trade
- 1 FX Trading - Matching
- 12 FX Trading – RFQ Maker
- 5 Intelligent Tagging
- 2 Legal One
- 23 Messenger Bot
- 3 Messenger Side by Side
- 9 ONESOURCE
- 7 Indirect Tax
- 60 Open Calais
- 275 Open PermID
- 44 Entity Search
- 2 Org ID
- 1 PAM
- PAM - Logging
- 6 Product Insight
- Project Tracking
- ProView
- ProView Internal
- 22 RDMS
- 1.9K Refinitiv Data Platform
- 643 Refinitiv Data Platform Libraries
- 4 LSEG Due Diligence
- LSEG Due Diligence Portal API
- 4 Refinitiv Due Dilligence Centre
- Rose's Space
- 1.2K Screening
- 18 Qual-ID API
- 13 Screening Deployed
- 23 Screening Online
- 12 World-Check Customer Risk Screener
- 1K World-Check One
- 46 World-Check One Zero Footprint
- 45 Side by Side Integration API
- 2 Test Space
- 3 Thomson One Smart
- 10 TR Knowledge Graph
- 151 Transactions
- 143 REDI API
- 1.8K TREP APIs
- 4 CAT
- 26 DACS Station
- 121 Open DACS
- 1.1K RFA
- 104 UPA
- 192 TREP Infrastructure
- 228 TRKD
- 915 TRTH
- 5 Velocity Analytics
- 9 Wealth Management Web Services
- 90 Workspace SDK
- 11 Element Framework
- 5 Grid
- 18 World-Check Data File
- 1 Yield Book Analytics
- 46 中文论坛