Performance Concern When Fetching News Stories

souyaku1121

I am currently using the Refinitiv Data Platform (RDP) to fetch news data by executing Python code.

I’m trying to retrieve individual stories using their story IDs, with the sample code below:

def fetch_single_story(story_id, access_token):

    story_url = f"https://api.refinitiv.com/data/news/v1/stories/{story_id}"

    headers = {

        'Accept': 'application/json',

        'Authorization': f'Bearer {access_token}'

    }

    response = requests.get(story_url, headers=headers)

    if response.status_code != 200:

        log(f"⚠️ Failed to fetch story {story_id}: {response.status_code}")

        return None

However, this approach is quite slow since each execution fetches only one story at a time.

I’m considering using parallel processing to fetch multiple stories simultaneously. In that case, I’d like to know:

Is there any request rate limit per second for the /stories/{story_id} endpoint?

I couldn’t find a clear description of this in the News Service User Guide, so I would appreciate any clarification. Many thanks!

Jirapongse

It should be a ticket and the case must be escalated to the product support team.

Jirapongse

@souyaku1121

Thank you for reaching out to us.

If the documentation does not specify any limitations, please reach out to the product support team directly for confirmation.

souyaku1121

Hi, thank you for your reply.

Do you know the email address for the RDP support team?

Or should I raise a help ticket from my account?

https://community.developers.refinitiv.com/discussion/comment/127391#Comment_127391

souyaku1121

https://community.developers.refinitiv.com/discussion/comment/127398#Comment_127398

Thanks, let me try to raise a help ticket

souyaku1121

Hello.

Lseg helpdesk answered me that user rate limit for the /stories/{storyId} endpoint is 20 requests per second (rps)

I tested the attached sample code , which requests stories one by one using their story IDs, while respecting the limit of 20 requests per second by adding a 1-second delay between batches.

I ran it in a Snowflake notebook, which provides a Python environment, still I encounter an HTTP 429 error.

As far as I understand, this approach aligns with the technical requirements of the endpoint.

I would appreciate any advice on the limitations involved.

sample_code.txt

wasin.w V3

https://community.developers.refinitiv.com/discussion/comment/127418#Comment_127418

Hello @souyaku1121

I am noticed that your attachment contains the app-key information. Please be informed that the app-key is also a secret credential that should not be shared on the public forum.

I have removed it from your code file.

wasin.w V3

Hello @souyaku1121

I have tested your code on my local environment (removed Snowflake code). I can replicate the issue with the batch_size 10, 12, 15, and 20.

Some news story request got "[14:25:03] ⚠️ Failed to fetch story <story ID>: 429" error message.

I strongly suggest you contact the RDP News team to verify the issue on server side. The server side should have a number of incoming request from your end to confirm where it was exceed.

souyaku1121

Hello, thank you very much for checking this issue.

I’m reaching out to the RDP team to ask them for further investigation

https://community.developers.refinitiv.com/discussion/comment/127421#Comment_127421