Avoiding empty rows during RTH extraction

I'm trying to extract historical RTH data. I'm getting a LOT of blank rows which take up lots of memory and slow everything down. These are days that are weekends, holidays, or even days where the underlying RIC was not listed/trading at all. Is there a way to avoid returning those rows at all? Seems like a needless waste for everyone involved.

Find more posts tagged with

tick-history-rest-api

trth-rest-api

Accepted answers

Jirapongse

I would like to share the answer from the Tick History support team regarding this question.

To avoid blank rows in RTH extraction, we can set TimebarPersistence to false in the request, this will remove the blank rows and weekend data. Time bar Persistence is applicable for all instruments irrespective of Content sets.

{
    "ExtractionRequest": {
        "@odata.type": "#DataScope.Select.Api.Extractions.ExtractionRequests.TickHistoryIntradaySummariesExtractionRequest",
        "ContentFieldNames": [
            "Close Ask",
            "Close Bid",
            "High",
            "High Ask",
            "High Bid",
            "Last",
            "Low",
            "Low Ask",
            "Low Bid",
            "No. Asks",
            "No. Bids",
            "No. Trades",
            "Open",
            "Open Ask",
            "Open Bid",
            "Volume"
        ],
        "IdentifierList": {
            "@odata.type": "#DataScope.Select.Api.Extractions.ExtractionRequests.InstrumentIdentifierList",
            "InstrumentIdentifiers": [
                {
                    "Identifier": "USH0",
                    "IdentifierType": "Ric"
                }
            ],
            "UseUserPreferencesForValidationOptions": "false",
            "ValidationOptions": {
                "AllowHistoricalInstruments": true,
                "AllowInactiveInstruments": true
            }
        },
        "Condition": {
            "MessageTimeStampIn": "GmtUtc",
            "ReportDateRangeType": "Range",
            "QueryStartDate": "1997-06-20T00:00:00.000Z",
            "QueryEndDate": "2020-06-23T00:00:00.000Z",
            "SummaryInterval": "OneHour",
            "TimebarPersistence": false,
            "DisplaySourceRIC": true
        }
    }
}

Tick History Intraday Summaries extraction provides transaction messages constructed on the fly from either Trade or Quote messages, summarized into time-series intervals representing aggregations at 1 second, 5 seconds, 1 minute, 5 minutes, 10 minutes, 15 minutes, and 60 minutes. Summaries are created based on rules (Refinitiv Intellectual Property) varying on a per-exchange basis and cannot be shared.

In Refinitiv Tick History v2 Intraday Summaries extraction, when the Time bar Persistence option is true the output result for quotes (Bid and Ask value) will have all the data included for the date and time range, irrespective of the same bid and ask quoted for different timings. However, when the Time bar Persistence option is false, the output result will capture only the changed values for the date and time range selected.

Time bar persistence is applicable only for quote messages and not for trade messages. The logic behind this is that the quotes are still alive in the market unless there’s a change and therefore the quotes summaries are copied down when the option is checked. An easy way of identifying if the quotes are a summary of an existing time interval or a copy from a previous time interval is with the help of fields “No. Bids” and “No. Asks” which is 0 when the values are getting copied down. For trades, copying down values from prior time bars amounts to manipulation of data as there are no trades.

The output when setting TimebarPersistence to true is:

The output when setting TimebarPersistence to false is:

All comments

Jirapongse

@pdebaz

I could not find an option in the TickHistoryIntradaySummariesExtractionRequest that can be used to remove empty rows. You can see all available options on the REST API Reference Tree.

You may need to contact the Tick History Support team via MyRefinitv to confirm it or request to add this feature to the TickHistoryIntradaySummariesExtractionRequest.

Jirapongse

I would like to share the answer from the Tick History support team regarding this question.

{
    "ExtractionRequest": {
        "@odata.type": "#DataScope.Select.Api.Extractions.ExtractionRequests.TickHistoryIntradaySummariesExtractionRequest",
        "ContentFieldNames": [
            "Close Ask",
            "Close Bid",
            "High",
            "High Ask",
            "High Bid",
            "Last",
            "Low",
            "Low Ask",
            "Low Bid",
            "No. Asks",
            "No. Bids",
            "No. Trades",
            "Open",
            "Open Ask",
            "Open Bid",
            "Volume"
        ],
        "IdentifierList": {
            "@odata.type": "#DataScope.Select.Api.Extractions.ExtractionRequests.InstrumentIdentifierList",
            "InstrumentIdentifiers": [
                {
                    "Identifier": "USH0",
                    "IdentifierType": "Ric"
                }
            ],
            "UseUserPreferencesForValidationOptions": "false",
            "ValidationOptions": {
                "AllowHistoricalInstruments": true,
                "AllowInactiveInstruments": true
            }
        },
        "Condition": {
            "MessageTimeStampIn": "GmtUtc",
            "ReportDateRangeType": "Range",
            "QueryStartDate": "1997-06-20T00:00:00.000Z",
            "QueryEndDate": "2020-06-23T00:00:00.000Z",
            "SummaryInterval": "OneHour",
            "TimebarPersistence": false,
            "DisplaySourceRIC": true
        }
    }
}

The output when setting TimebarPersistence to true is:

The output when setting TimebarPersistence to false is:

EXPLORE OUR SITES