Let's say I want to get around 500 fields for around 4000 instruments (stocks) for about 20 year (SDate = 2020-01-01). Given the multiple limits I cannot just use
get_date(instrument, fields, {'SDate': '2000-01-01', 'EDate': today_as_string})
as this will give all kinds of errors / limits breaching.
Currently I will chunk the instruments 50 at a time, select 1 field and query for the full date range.
dfs = []
for field in fields:
for instruments_chunk in split_list(instruments, max_items_in_list=50):
df = ek.get_data(instruments_chunk, [field + 'date', field], {'SDate': '2000-01-01', 'EDate': today_as_string})
df.rename(columns={df.columns[1]: 'date'}, inplace=True)
df['date'] = pd.to_datetime(df['date']).dt.date.astype('datetime64')
dfs.append(df)
final_df = pd.concat(dfs)
Although this works it is very slow and will take several days to complete.
Is there a better solution to this kind of problem?
