I need to fetch all the news headlines and stories for a given day via `data/news/v1` API.
Since there's no such endpoint which can provide daily dump of news data, what I did was to first get all the headlines using `headlines?dateFrom=....&dateTo=...` endpoint (using `next` cursor) then iterate over the headlines to get the `storyId`, then for each `storyId` query the `stories/{storyId}` endpoint to fetch the story for that given `storyId`.
Now for a given day usually there are more than 15K news stories.
My single-threaded program takes forever to fetch these many stories (not counting the issues with token refresh/expiration).
If I use multiprocessing I get below exception
`Code 429, 'Too Many Requests' for data/news/v1/stories/{storyId}`
Could you please let me know what is the most efficient way to fetch all the news (headlines + stories) for a given day?
Would it be possible for you to provide this data via SFTP instead?
Note: We are already using the streaming news service of Refinitiv but we also want the data via `data/news/` API for reconciliation purposes.
