Hi everyone,
I am trying to retrieve some Fundamental and Reference data for all of the SNP 500 stocks. These datapoints include values such as Income before Extraordinary Items, EBIT, Capital, etc. Now, I am trying to asynchronously retrieve all of this data. At first, I tried to loop ethrough the list of the RICs of the SNP, and from there, retrieve the data for each RIC one at a time (by looping with a for loop). However, this was rather inefficient, and it took about 9 minutes to run.
Now, I tried to separate each call into multiple tasks and I used asyncio.gather() to run all of these requests concurrently.
import asyncio
import pandas as pd
from datetime import datetime
async def fetch_ebit_data(ric, start_year, current_year):
try:
ebit_datapoints = await content.fundamental_and_reference.Definition(
universe=[f'{ric}'],
fields=["TR.F.EBIT.date", "TR.F.EBIT"],
parameters={"SDate": f"{start_year}-01-01", "EDate": f"{current_year}-12-31", "Frq": "Y"}
).get_data_async(closure=f'')
# Check if the response contains data
if ebit_datapoints and not ebit_datapoints.data.df.empty:
return ebit_datapoints.data.df
else:
print(f"No data returned for {ric}")
return None
except Exception as e:
print(f"Error retrieving data for {ric}: {str(e)}")
return None
async def retrieve_ebit_data(list_of_snp_rics, start_year, current_year):
tasks = [fetch_ebit_data(ric, start_year, current_year) for ric in list_of_snp_rics]
ebit_values_list = await asyncio.gather(*tasks)
# Filter out None values (i.e., failed or empty responses)
ebit_values_list = [df for df in ebit_values_list if df is not None]
combined_ebit_df = pd.concat(ebit_values_list, ignore_index=True)
display(combined_ebit_df)
return combined_ebit_df
# Setup
current_year = datetime.today().year
start_year = current_year - 8
list_of_snp_rics = await retrieve_ric_snp_stocks()
# Retrieve and combine EBIT data
combined_ebit_df = await retrieve_ebit_data(list_of_snp_rics, start_year, current_year)
The problem with this method is that, I couldn't get any values for MAJORITY of the tickers / RICs. So currently, I am faced with a problem as I am unsure of a workaround in this situation. One thing I can think of is to use the ThreadPoolExecutor for the number of RICs there are (deploying about 503 threads), but this might be a really CPU-intensive process. I need some guidance on how I can navigate through this, as I can then apply it to retrieving multiple datapoints (EBIT and Capital for example).
Thanks!
