I am trying to download roughly 10 years of daily price data for 3000 firms with rdp.get_get_historical_price When trying it at once, the API times out. What are best practices for requesting data in batches and the combining the outputs to a single pandas dataframe?
Currently I am trying to do it in batches by appending lists, but I'm not satisfied with the output as i can't format it properly into a dataframe
dict_list = ric_lists #ric_lists is list of ALL RICs batch_list = [] return_list = [] for i in dict_list: batch_list.append(i) if len(batch_list) == 5: return_list.append(Pricegetter(batch_list)) batch_list.clear() if batch_list: return_list.append(Pricegetter(batch_list))
[ KAER.VI 2011-01-04 18.056605 2011-01-05 18.056605 2011-01-11 18.253407 2011-01-12 18.253407 2011-01-18 18.253407 ... ... 2021-12-23 14.900000 2021-12-27 15.100000 2021-12-28 15.100000 2021-12-29 15.200000 2021-12-30 15.300000 [1794 rows x 1 columns], BHAV.VI 2011-01-03 43.05 2011-01-07 43.01 2011-01-11 43.00 2011-01-19 43.05 2011-01-20 43.05 ... ... 2021-12-17 95.00 2021-12-20 98.00 2021-12-21 98.00 2021-12-23 98.50 2021-12-27 98.50 [918 rows x 1 columns], SMPV.VI 2011-01-03 39.60 2011-01-04 39.50 2011-01-05 39.45 2011-01-07 39.10 2011-01-10 39.30 ... ... 2021-12-23 27.85 2021-12-27 28.40 2021-12-28 28.65 2021-12-29 28.75 2021-12-30 29.00 [2740 rows x 1 columns]]
Should I change how I use the batches or is there way to tranfrom the output into a dataframe with datetime?