Hello,
I'm working on a Python project using the Refinitiv Data API to retrieve large sets of equity quote data. I need to fetch data in chunks of 10,000 records due to API limitations, but I'm struggling with implementing effective pagination to avoid overlapping data without missing any records. The API doesn't seem to support direct offset management.
Here's what I'm trying to do:
- Retrieve 10,000 records at a time.
- Ensure the next batch of 10,000 starts right after the last record of the previous batch.
- I'm using
rd.discovery.search method but can't find a way to paginate properly.
Here is a code with offset(I know that search() doesn't have such an argument, it's just an example):
import refinitiv.data as rd
import pandas as pd
def retrieve_data():
rd.open_session()
offset = 0
all_data = pd.DataFrame()
while True:
results = rd.discovery.search(
view=rd.discovery.Views.EQUITY_QUOTES,
top=10000,
filter="(RCSAssetCategoryLeaf eq 'Ordinary Share')",
select="RIC, DTSubjectName, RCSAssetCategoryLeaf, ExchangeName, ExchangeCountry",
offset=offset # Adjust the offset for each iteration
)
df = pd.DataFrame(results)
if df.empty:
break
all_data = pd.concat([all_data, df], ignore_index=True)
offset += 10000 # Increase the offset to get the next batch of records
df.to_csv(f'./Retrieved_RICs_{offset}.csv')
all_data.to_csv('./Retrieved_RICs_All.csv')
print("Data retrieval complete.")
rd.close_session()
retrieve_data()
Any advice on managing continuation tokens or other effective methods would be greatly appreciated!
Thank you in advance!
