get_timeseries returns 50,000 rows max

Following on my earlier related question:

https://community.developers.refinitiv.com/questions/88816/returned-data-is-in-a-shortened-format.html

I am calling the get_timeseries command with the interval = tick. But I am only getting 50,000 rows of data in the returned dataframe. Setting the count parameter to a high value such as 1000000 still leads to a maximum of 50,000 rows. Is there a limit that I am not aware of?

Related question: Is there are way to pre-filter the data to only return tick transactions that are greater than 100 shares?

Find more posts tagged with

eikon-data-api

Accepted answers

jason.ramchandani01

@dgarrard So I was going to suggest this to you - we have a more comprehensive timeseries history API for both interday and intraday use cases Historical Pricing API from our Refinitiv Data Platform (RDP) API. I would strongly recommend that (particularly for non-economic data) for exchange pricing histories. The main reason for this is that it has an expanded data model - based on the asset-type requested and the main point of interest for you here is that for trades (it provides both trades and quotes or either) it contains RTL and sequence number to help you deal with such paging type issues - note the limit for Events API is 10k rows. Note - at the moment it is a single RIC multipoint API, but we will be releasing a multi-RIC, multipoint API at some point soon. You can see examples in Codebook App - I have included an example below for MSFT:

import refinitiv.dataplatform as rdp
import datetime
rdp.open_desktop_session('YOUR APP KEY HERE')

rdp.get_historical_price_events(
    universe = 'AAPL.O',
    eventTypes = ['trade'],
    start = datetime.timedelta(-1), 
    end = datetime.timedelta(0)    
)

In terms of exploring the data model further please use the API Playground which contains code samples, swagger and the reference guide for all RDP endpoints. You can search for Historical Pricing to find the correct endpoint - /data/historical-pricing/v1/views/events/{universe} and then look at the resources there. This is more complex but the only robust method I think. I hope this can help.

All comments

jason.ramchandani01

@dgarrard You are correct 50,000 rows is the limit for a single intraday timeseries API call in Eikon. Please see the API limits documents here. You can of course iterate using the datetime parameters (see the Reference guide in the previous link) - FYI Eikon carries 3 months of intraday tick data - which can be very substantial - though if you need more history than this please check our Tick History API offering which provide tick data gong back to 1996 as well as full order book/Level 2 data .

There is no way to pre-filter with get_timeseries call - though of course you can do this in Pandas after. I hope this can help.

dgarrard

jason.ramchandani01. My use case is to retrieve the ticks from the most recent trading session. I run this in the evening after market closes. For many stocks, 50000 rows is a sufficient limit to get all the ticks for the day. But for other stocks, such as MSFT and AAPL, there will need to be some paging of api calls and building of a summary dataframe. Tick data returns a row such as this: 2022-01-21 14:30:00.046,26.15,100 (datetimestamp,price,shares traded) where the time has hundreds of seconds.

Because there may be situations where more than one transaction occurs on a particular time stamp, how do I ensure I do not page data with gaps or redundant data if I must pass a datetime as a query parameter. In the list below, if my 50,000 limit occurs at the row I have underlined, I could make my next query start at 2022-01-21 14:30:02.405 or 2022-01-21 14:30:02.406, but in the .405 case I would have redundancies at the page boundaries, and in the .406 case I would be missing data at the page boundaries.

2022-01-21 14:30:01.375,26.15,100
2022-01-21 14:30:01.375,26.15,100
2022-01-21 14:30:01.375,26.15,100
2022-01-21 14:30:01.375,26.15,100
2022-01-21 14:30:01.375,26.15,200
2022-01-21 14:30:01.375,26.15,100
2022-01-21 14:30:01.375,26.15,300
2022-01-21 14:30:01.375,26.15,200
2022-01-21 14:30:01.375,26.15,200
2022-01-21 14:30:01.375,26.15,300
2022-01-21 14:30:01.375,26.16,100
2022-01-21 14:30:01.569,26.16,200
2022-01-21 14:30:01.569,26.16,100
2022-01-21 14:30:01.989,26.16,4000
2022-01-21 14:30:01.999,26.12,200
2022-01-21 14:30:02.405,26.12,200
2022-01-21 14:30:02.405,26.14,100
2022-01-21 14:30:02.405,26.12,100
2022-01-21 14:30:02.405,26.13,100
2022-01-21 14:30:02.405,26.12,300
2022-01-21 14:30:02.405,26.12,200
2022-01-21 14:30:02.405,26.12,100
2022-01-21 14:30:02.405,26.12,100
2022-01-21 14:30:02.405,26.19,200
2022-01-21 14:30:02.985,26.12,100
2022-01-21 14:30:02.985,26.12,200
2022-01-21 14:30:02.985,26.12,2200
2022-01-21 14:30:02.985,26.12,100
2022-01-21 14:30:02.999,26.13,100
2022-01-21 14:30:02.999,26.12,100
2022-01-21 14:30:02.999,26.13,100

Does someone have some sample code where they have successfully iteratively pulled a full and accurate trading session of ticks for a single instrument, from New York perhaps, and reconstructed a full session dataframe locally in python?

jason.ramchandani01

import refinitiv.dataplatform as rdp
import datetime
rdp.open_desktop_session('YOUR APP KEY HERE')

rdp.get_historical_price_events(
    universe = 'AAPL.O',
    eventTypes = ['trade'],
    start = datetime.timedelta(-1), 
    end = datetime.timedelta(0)    
)

dgarrard

jason.ramchandani01. I attempted to run the rdp code you provided and got the following:

This seems to be clear evidence that this clients account is not entitled to RDP.

I will forge ahead with the Eikon Data API 50,000 row limit and the edge effects that will perhaps be just a nuisance. My feel is that our use case can accept some of these effects without being a problem in the tick data analysis we do.

Thank you.

jason.ramchandani01

@dgarrard you need to install the refinitiv.dataplatform library first typing:

pip install refinitiv.dataplatform

from a terminal window or:

!pip install refinitiv.dataplatform

from a jupyter notebook. I think most variants that have access to EDAPI will have access to RDP.

dgarrard

I have other python scripts running on the same machine that call:

import refinitiv.dataplatform.eikon as ek

and they function correctly. Is that not sufficient to ensure that

import refinitiv.dataplatform as rdp

would not need a pip install?

jason.ramchandani01

@dgarrard Yes but I am pretty sure that if you have access to the Eikon Data API you should have access to RDP Historical Pricing. OK you may want to investigate that with the account manager. Just my 2 cents. I feel it would be a more robust workflow for the customer - assuming they do have access. How are you opening the desktop session? As follows:

import refinitiv.dataplatform as rdp
session = rdp.open_desktop_session('YOUR APPKEY HERE') #Have you created an appkey with EDP rights - just a thought.

EXPLORE OUR SITES