Retrieving TickHistoryRaw data for economic series via DSS REST API

nelson.lam

I'm trying to retrieve TickHistoryRaw data for economic series like USNFAR=ECI via DSS REST API in Python. I followed the tutorial example and posted a TickHistoryRawExtractionRequest successfully. I got a status_code 200 immediately but there's no data in the response body.

Here's the Condition arguments that I sent:

"MessageTimeStampIn": "GmtUtc",

"ReportDateRangeType": "Range",

"QueryStartDate": "2006-01-01T00:00:00.000Z",

"QueryEndDate": "2010-01-01T00:00:00.000Z",

"ExtractBy": "Ric",

"SortBy": "SingleByRic",

"DomainCode": "MarketPrice",

"DisplaySourceRIC": True

I expect this request to return all FIDs. I checked on DSS GUI and confirmed there was data for the series since May 2005.

Christiaan Meihsl

@nelson.lam,

I just tested your query, this is what it returns:

{
    "@odata.context": "https://hosted.datascopeapi.reuters.com/RestApi/v1/$metadata#RawExtractionResults/$entity",
    "JobId": "0x06983b877610786a",
    "Notes": [
        "All identifiers were invalid.  No extraction performed."
    ],
    "IdentifierValidationErrors": [
        {
            "Identifier": {
                "@odata.type": "#ThomsonReuters.Dss.Api.Content.InstrumentIdentifier",
                "Identifier": "USNFAR=ECI",
                "IdentifierType": "Ric",
                "Source": ""
            },
            "Message": "Not found"
        }
    ]
}

The extraction notes are key to understand issues; this is why I always recommend logging them. Experimenting in Postman is one of my favorite tools, as it automatically displays error messages and such, which helps a great deal.

In this particular case, the issue is that the instrument was Not found. This is because it is currently no longer quoted. For the instrument validation to succeed on an old instrument, you need to specify that it is a historical instrument, by allowing historical instruments in the ValidationOptions, and overriding the user preferences, using the following 2 lines after the InstrumentIdentifiers object:

{
  "ExtractionRequest": {
    "@odata.type": "#ThomsonReuters.Dss.Api.Extractions.ExtractionRequests.TickHistoryRawExtractionRequest",
    "IdentifierList": {
      "@odata.type": "#ThomsonReuters.Dss.Api.Extractions.ExtractionRequests.InstrumentIdentifierList", 
      "InstrumentIdentifiers": [ { "Identifier": "USNFAR=ECI", "IdentifierType": "Ric" } ],
      "ValidationOptions": {"AllowHistoricalInstruments": true},
      "UseUserPreferencesForValidationOptions": false
    },
    "Condition": {
      "MessageTimeStampIn": "GmtUtc",
      "ReportDateRangeType": "Range",
      "QueryStartDate": "2006-01-01T00:00:00.000Z",
      "QueryEndDate": "2010-01-01T00:00:00.000Z",
      "ExtractBy": "Ric",
      "SortBy": "SingleByRic",
      "DomainCode": "MarketPrice",
      "DisplaySourceRIC": true
    }
  }
}

This request will deliver an HTTP status 202, and a location URL in the response headers.

After polling the location URL you will get a JobId and extraction notes that clearly state when the instrument was active (2006-2009):

Manifest: #RIC,Domain,Start,End,Status,Count\nManifest: USNFAR=ECI,Market Price,2006-01-03T14:01:24.175993000Z,2009-12-27T00:54:07.885198000Z,Active,16993

Using the JobId you can retrieve the data (222896 lines).

Christiaan Meihsl

https://community.developers.refinitiv.com/discussion/comment/41293#Comment_41293

For more information on the extraction notes, see this article.

nelson.lam

https://community.developers.refinitiv.com/discussion/comment/41293#Comment_41293

How can i pull up the extraction notes? I checked the response returned and I couldn't find that msg/info

Christiaan Meihsl

https://community.developers.refinitiv.com/discussion/comment/41301#Comment_41301

@nelson.lam,

The extraction notes should be in the response to your extraction request, in the object "Notes" which is in the JSON formatted response body, as illustrated in the first box in my initial answer to your query.

If you cannot see them, then please post your request body and endpoint, so that I can check it.

nelson.lam

https://community.developers.refinitiv.com/discussion/comment/41293#Comment_41293

@Christiaan Meihsl,

Using the ValidationOptions I was able to retrieve TickHistoryRaw data. Thanks for the solution. However when I checked the response for Notes (in Python) I can't find it as an Object at the top level like status_code, headers and content. When I ran json() to convert the body into json object I got the following error message,

File "C:\tools\miniconda3\envs\development_2\lib\json\decoder.py", line 357, in raw_decode
raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)

Christiaan Meihsl

https://community.developers.refinitiv.com/discussion/comment/41304#Comment_41304

@nelson.lam,

There seems to be an issue inside the code, specifically with the decoding of the response body, which is where the notes are.

To avoid missing errors during the prototyping phase I often use Postman to ensure I see all the relevant info (headers, body, status, errors, etc.).

Even if it is not for the same data, you might want to have a look at the Python code TRTH_OnDemand_IntradayBars described here, it uses the same workflow as your request, and extracts the notes in step 3.