question

Upvotes
Accepted
5 0 0 3

Avoiding empty rows during RTH extraction

I'm trying to extract historical RTH data. I'm getting a LOT of blank rows which take up lots of memory and slow everything down. These are days that are weekends, holidays, or even days where the underlying RIC was not listed/trading at all. Is there a way to avoid returning those rows at all? Seems like a needless waste for everyone involved. 1657053335612.png

tick-history-rest-apitrth-rest-api
1657053335612.png (62.9 KiB)
icon clock
10 |1500

Up to 2 attachments (including images) can be used with a maximum of 512.0 KiB each and 1.0 MiB total.

Upvotes
Accepted
55.9k 145 45 65

I would like to share the answer from the Tick History support team regarding this question.

To avoid blank rows in RTH extraction, we can set TimebarPersistence to false in the request, this will remove the blank rows and weekend data. Time bar Persistence is applicable for all instruments irrespective of Content sets.

{
    "ExtractionRequest": {
        "@odata.type": "#DataScope.Select.Api.Extractions.ExtractionRequests.TickHistoryIntradaySummariesExtractionRequest",
        "ContentFieldNames": [
            "Close Ask",
            "Close Bid",
            "High",
            "High Ask",
            "High Bid",
            "Last",
            "Low",
            "Low Ask",
            "Low Bid",
            "No. Asks",
            "No. Bids",
            "No. Trades",
            "Open",
            "Open Ask",
            "Open Bid",
            "Volume"
        ],
        "IdentifierList": {
            "@odata.type": "#DataScope.Select.Api.Extractions.ExtractionRequests.InstrumentIdentifierList",
            "InstrumentIdentifiers": [
                {
                    "Identifier": "USH0",
                    "IdentifierType": "Ric"
                }
            ],
            "UseUserPreferencesForValidationOptions": "false",
            "ValidationOptions": {
                "AllowHistoricalInstruments": true,
                "AllowInactiveInstruments": true
            }
        },
        "Condition": {
            "MessageTimeStampIn": "GmtUtc",
            "ReportDateRangeType": "Range",
            "QueryStartDate": "1997-06-20T00:00:00.000Z",
            "QueryEndDate": "2020-06-23T00:00:00.000Z",
            "SummaryInterval": "OneHour",
            "TimebarPersistence": false,
            "DisplaySourceRIC": true
        }
    }
}


Tick History Intraday Summaries extraction provides transaction messages constructed on the fly from either Trade or Quote messages, summarized into time-series intervals representing aggregations at 1 second, 5 seconds, 1 minute, 5 minutes, 10 minutes, 15 minutes, and 60 minutes. Summaries are created based on rules (Refinitiv Intellectual Property) varying on a per-exchange basis and cannot be shared.

In Refinitiv Tick History v2 Intraday Summaries extraction, when the Time bar Persistence option is true the output result for quotes (Bid and Ask value) will have all the data included for the date and time range, irrespective of the same bid and ask quoted for different timings. However, when the Time bar Persistence option is false, the output result will capture only the changed values for the date and time range selected.

Time bar persistence is applicable only for quote messages and not for trade messages. The logic behind this is that the quotes are still alive in the market unless there’s a change and therefore the quotes summaries are copied down when the option is checked. An easy way of identifying if the quotes are a summary of an existing time interval or a copy from a previous time interval is with the help of fields “No. Bids” and “No. Asks” which is 0 when the values are getting copied down. For trades, copying down values from prior time bars amounts to manipulation of data as there are no trades.

The output when setting TimebarPersistence to true is:

1657182052120.png

The output when setting TimebarPersistence to false is:

1657182116829.png


1657182052120.png (119.9 KiB)
1657182116829.png (145.9 KiB)
icon clock
10 |1500

Up to 2 attachments (including images) can be used with a maximum of 512.0 KiB each and 1.0 MiB total.

Upvotes
55.9k 145 45 65

@pdebaz

I could not find an option in the TickHistoryIntradaySummariesExtractionRequest that can be used to remove empty rows. You can see all available options on the REST API Reference Tree.

1657106048422.png

You may need to contact the Tick History Support team via MyRefinitv to confirm it or request to add this feature to the TickHistoryIntradaySummariesExtractionRequest.


1657106048422.png (52.1 KiB)
icon clock
10 |1500

Up to 2 attachments (including images) can be used with a maximum of 512.0 KiB each and 1.0 MiB total.