API Pagination in Python Using Refinitiv Data API

vitali

Hello,

I'm working on a Python project using the Refinitiv Data API to retrieve large sets of equity quote data. I need to fetch data in chunks of 10,000 records due to API limitations, but I'm struggling with implementing effective pagination to avoid overlapping data without missing any records. The API doesn't seem to support direct offset management.

Here's what I'm trying to do:

1. Retrieve 10,000 records at a time.
2. Ensure the next batch of 10,000 starts right after the last record of the previous batch.
3. I'm using rd.discovery.search method but can't find a way to paginate properly.

Here is a code with offset(I know that search() doesn't have such an argument, it's just an example):

import refinitiv.data as rd
import pandas as pd

def retrieve_data():
    rd.open_session()
    offset = 0
    all_data = pd.DataFrame()

    while True:
        results = rd.discovery.search(
            view=rd.discovery.Views.EQUITY_QUOTES,
            top=10000,
            filter="(RCSAssetCategoryLeaf eq 'Ordinary Share')",
            select="RIC, DTSubjectName, RCSAssetCategoryLeaf, ExchangeName, ExchangeCountry",
            offset=offset  # Adjust the offset for each iteration
        )

        df = pd.DataFrame(results)
        if df.empty:
            break

        all_data = pd.concat([all_data, df], ignore_index=True)
        offset += 10000  # Increase the offset to get the next batch of records


        df.to_csv(f'./Retrieved_RICs_{offset}.csv')

    all_data.to_csv('./Retrieved_RICs_All.csv')
    print("Data retrieval complete.")

    rd.close_session()

retrieve_data()
 

Any advice on managing continuation tokens or other effective methods would be greatly appreciated!

Thank you in advance!

nick.zincone

Hi @vitali

I don't know the policies regarding rate limit and recovery. I did research the API Playground Reference document around this:

I would suggest you open a ticket and they will bring in the Search team managing policy around these issues.