why is a simple retrieval taking so long?

i have a script that I use for other chains (like vix futures) and it takes a couple of minutes to run. I've now been waiting 2 hours for this... why does it take so long?

I just want minute bars for SPX futures for a few months...

{
    "ExtractionRequest": {
        "@odata.type": "#ThomsonReuters.Dss.Api.Extractions.ExtractionRequests.TickHistoryIntradaySummariesExtractionRequest",
        "ContentFieldNames": [
            "Open",# "Open Ask", "Open Bid",
            "High",# "High Ask", "High Bid",
            "Low",# "Low Ask", "Low Bid",
            "Last", "Close Ask", "Close Bid",
            "Volume"
        ],
        "IdentifierList": {
            "@odata.type": "#ThomsonReuters.Dss.Api.Extractions.ExtractionRequests.InstrumentIdentifierList",
            "InstrumentIdentifiers": [
                #{"Identifier": ".SPX", "IdentifierType": "Ric"},
                {"Identifier": "0#ES:", "IdentifierType": "ChainRIC"}
                #{"Identifier": "0#VX:", "IdentifierType": "ChainRIC"}
            ],
            "ValidationOptions": {"AllowHistoricalInstruments": "true"},
            "UseUserPreferencesForValidationOptions": "false"
        },
        "Condition": {
            "MessageTimeStampIn": "GmtUtc",
            "ReportDateRangeType": "Range",
            "QueryStartDate": "2021-02-01T00:00:00.000Z",
            "QueryEndDate": "2021-06-01T23:59:00.000Z",
            "SummaryInterval": "OneMinute",    # "FifteenMinutes" # OneSecond # OneMinute
            "DisplaySourceRIC": "true"
        }
    }
}

Find more posts tagged with

dss

dss-rest-api

tick-history-rest-api

datascope-select

Accepted answers

zoya faberov

Hello @opsp,

Absolutely, glad to share my thoughts. RTH HTTP REST on demand, custom report requesting is primarily intended as a light way of requesting RTH content.

As you are looking to request 100s of chains and a decade or more of data, I would consider design and product options:

The first approach I would consider, is Venue By Day or VBD. This will allow you to request a a complete exchange day's worth of data, compacted by zip and next select what you require from it, for details see:

Data Dictionary - Refinitiv Tick History Venue By Day Message Reporting and REST API Tutorial 2: Retrieving VBD files.

VBD is an additional product.

If the content is required to run large-scale calculations on it, I would consider taking Google Big Query approach, and not pulling it down, rather running query in cloud and only obtaining the results. This is also a different product.
If you are looking to stay with custom reporting, I would split interval into days, and use scheduled approach, while scheduling in the off-hours.
For Tick History Intraday Summary custom report, direct download from AWS is supported, please see REFINITIV TICK HISTORY REST API USER GUIDE -> Downloading Files Faster via Direct Download From Amazon Web Services for X-Direct-Download: true description.

Hope this information helps

All comments

opsp

it's still running....should I kill it? how long should it take? (approx)

zoya faberov

Hello @opsp,

From what I see, this is a very large request. So would expect it to take long.

Have you tried requesting OneMinute bars, on the same chain RIC, for 1 day interval of time?

In a general scenario, if a request is taking unreasonably long time all of a sudden, especially if the same request takes a short time otherwise, I would suggest verifying with My Refinitiv -> Service Alerts section, to know of any potential outage that could be impacting your request at the time.

Hope this information helps

opsp

it's 3MB of sensibly encoded data... how can this be classed as a big request? I can almost fit it on a floppy disk....

it's just minute bars for SPX futures for just a few months... this was the small one.

I'm planning to download a decade or more of it and for 100s more root stems. why does it take such a long time for such a small amount of data?

should I expect it to really take weeks to download all that data? can you help me find optimizations or performance improvements?

zoya faberov

Hello @opsp,

Absolutely, glad to share my thoughts. RTH HTTP REST on demand, custom report requesting is primarily intended as a light way of requesting RTH content.

As you are looking to request 100s of chains and a decade or more of data, I would consider design and product options:

The first approach I would consider, is Venue By Day or VBD. This will allow you to request a a complete exchange day's worth of data, compacted by zip and next select what you require from it, for details see:

Data Dictionary - Refinitiv Tick History Venue By Day Message Reporting and REST API Tutorial 2: Retrieving VBD files.

VBD is an additional product.

If the content is required to run large-scale calculations on it, I would consider taking Google Big Query approach, and not pulling it down, rather running query in cloud and only obtaining the results. This is also a different product.
If you are looking to stay with custom reporting, I would split interval into days, and use scheduled approach, while scheduling in the off-hours.
For Tick History Intraday Summary custom report, direct download from AWS is supported, please see REFINITIV TICK HISTORY REST API USER GUIDE -> Downloading Files Faster via Direct Download From Amazon Web Services for X-Direct-Download: true description.

Hope this information helps

EXPLORE OUR SITES