Incomplete data on historic download API Datascope

When downloading historic data for certain futures contracts I'm receiving a 200 status code immediately but the message is empty. This only seems to be happening for the 3rd from front month of the futures contract. The product is LS Gas Oil TAS, the date is 17/12/19 (uk date format) and the RIC is LGOTH0. Another example is the same product on the 02/01/20 with the RIC of LGOTJ0. This is also happening for the LS Gas Oil Singapore product, an example being 01/04/20 with the RIC of LGOmmN0.
Any help would be great,
Thanks!
Best Answer
-
Hi @sanchezTJUK,
The URL remains the same:
{{protocol}}{{host}}{{api}}Extractions/ExtractWithNotes
Please find detailed info on HistoricalReference request in REST API Tutorial 10: On Demand Histo Reference extraction tutorial.
Alternatively, you can run Historical Search request as in REST API Tutorial 13: Historical Search. tutorial
0
Answers
-
Thanks for getting back to me!
The extraction request is below. We don't actually get any extraction notes back because the script errors with:
r3Json = json.loads(r3.text.encode('ascii', 'ignore'))
NameError: name 'r3' is not definedThe extraction notes seem to come out of the r3Json variable.
I've not tried this extraction on the GUI but I'll give that a go now.
requestUrl='https://hosted.datascopeapi.reuters.com/RestApi/v1/Extractions/ExtractRaw';
requestHeaders={
"Prefer":"respond-async",
"Content-Type":"application/json",
"Authorization": "token " + token
}
requestBody={
"ExtractionRequest": {
"@odata.type": "#ThomsonReuters.Dss.Api.Extractions.ExtractionRequests.TickHistoryIntradaySummariesExtractionRequest",
"ContentFieldNames": ["Quote - Bid Price","Quote - Bid Size","Quote - Ask Price","Quote - Ask Size"],
"IdentifierList": {
"@odata.type": "#ThomsonReuters.Dss.Api.Extractions.ExtractionRequests.InstrumentIdentifierList",
"InstrumentIdentifiers": [{
"Identifier": "LGOTJ0",
"IdentifierType": "Ric"
}],
"UseUserPreferencesForValidationOptions": "true"
},
"Condition": {
"MessageTimeStampIn": "GmtUtc",
"ReportDateRangeType": "Range",
"QueryStartDate": "2020-01-02T00:00:00.000Z",
"QueryEndDate": "2020-01-03T00:00:00.000Z",
"SummaryInterval": "OneMinute",
"TimebarPersistence":"true",
"DisplaySourceRIC":"true"
}
}
}0 -
My reply wouldn't post so I've added another post below. Thanks
0 -
Hi @sanchezTJUK,
There are two things that may be helpful.
"r3 is not defined" is likely related to a python module missing. Please check what is r3 imported as, and if that module is installed.
Another thing to check, I just tested this request on it's own, "naked" and submitting it from Postman tool. It did not go through, the fields do not appear to be valid.
I checked with the list of valid fields for TickHistoryIntradaySummaries, these fields do not appear on the list.
When I replace the fields with valid ones for the request type, the request gest accepted ( status code 202)
{
"ExtractionRequest": {
"@odata.type": "#ThomsonReuters.Dss.Api.Extractions.ExtractionRequests.TickHistoryIntradaySummariesExtractionRequest",
"ContentFieldNames": [
"Close Ask",
"Close Bid"
],
"IdentifierList": {
"@odata.type": "#ThomsonReuters.Dss.Api.Extractions.ExtractionRequests.InstrumentIdentifierList",
"InstrumentIdentifiers": [{
"Identifier": "LGOTJ0",
"IdentifierType": "Ric"
}],
"UseUserPreferencesForValidationOptions": "true"
},
"Condition": {
"MessageTimeStampIn": "GmtUtc",
"ReportDateRangeType": "Range",
"QueryStartDate": "2020-01-02T00:00:00.000Z",
"QueryEndDate": "2020-01-03T00:00:00.000Z",
"SummaryInterval": "OneMinute",
"TimebarPersistence":"true",
"DisplaySourceRIC":"true"
}
}
}Does the above run for you?
You may find useful tutorial REST API Tutorial 6: On Demand intraday bars extraction, it goes over the approach to Intraday in detail, using Postman, and if you use Python, the approach would be consistent. Tutorial Programming without SDK discusses Python, and includes download link to working python examples.
You may wish, as a quicker approach:
1. Use Postman tool to fully test and tune TickHistoryIntradaySummariesExtractionRequest that you require.
2. Download the Python examples ( they are linked from tutorial and also in RTH-> Downloads -> RTH Python Code Samples), and then swap you required request tuned in step 1 into working python code, instead of the included request.
Hope this helps
0 -
I'm really sorry, i got that wrong, the extraction request is from:
"@odata.type": "#ThomsonReuters.Dss.Api.Extractions.ExtractionRequests.TickHistoryTimeAndSalesExtractionRequest"
A lot of my code is defined under variables so I needed to try and recreate from an original.
Thanks for the advice on using postman, but this is an issue because the same code and requests work with Brent Crude products and with LS Gas Oil months 1 and 2...
Regarding r3, this is not a module that is imported. It is a variable that is assigned to part of the json message that is not being delivered.
0 -
Replacing the request type, I get back
"
"error": {
"message": "Malformed request payload: For the property name \"SummaryInterval\" in the JSON request the value could not be parsed successfully. Please check the casing or spelling of the property."
Without this condition clause, I am able to get results back.
...
#RIC,Alias Underlying RIC,Domain,Date-Time,GMT Offset,Type,Bid Price,Bid Size,Ask Price,Ask Size
LGOTJ0,,Market Price,2020-01-02T07:24:56.596063285Z,+0,Quote,-0.25,10,,
LGOTJ0,,Market Price,2020-01-02T07:25:02.700408234Z,+0,Quote,-0.25,10,0.25,10
LGOTJ0,,Market Price,2020-01-02T10:47:24.335364201Z,+0,Quote,-0.25,90,0.25,10
LGOTJ0,,Market Price,2020-01-02T10:47:25.611314961Z,+0,Quote,-0.25,90,0.25,90
LGOTJ0,,Market Price,2020-01-02T15:44:43.453059247Z,+0,Quote,0,137,0.25,90
LGOTJ0,,Market Price,2020-01-02T15:44:57.936016337Z,+0,Quote,0,137,0.25,105
LGOTJ0,,Market Price,2020-01-02T15:46:25.500950349Z,+0,Quote,0,137,0.25,125
LGOTJ0,,Market Price,2020-01-02T16:08:52.450662974Z,+0,Quote,0,12,0.25,125
LGOTJ0,,Market Price,2020-01-02T16:11:23.754852370Z,+0,Quote,0,12,0.25,105
LGOTJ0,,Market Price,2020-01-02T16:18:44.751242199Z,+0,Quote,0,12,0.25,90
LGOTJ0,,Market Price,2020-01-02T16:18:48.603454951Z,+0,Quote,0,16,0.25,90
LGOTJ0,,Market Price,2020-01-02T16:22:10.061362751Z,+0,Quote,0,16,0.25,10
...Please confirm the exact request - glad to help test on my side?
0 -
Thanks for being so patient Zoya,
I got a couple of bits wrong on the original extraction request I sent you. Here is a revised copy that I've double checked:
requestUrl='https://hosted.datascopeapi.reuters.com/RestApi/v1/Extractions/ExtractRaw';
requestHeaders={
"Prefer":"respond-async",
"Content-Type":"application/json",
"Authorization": "token " + token
}
requestBody={
"ExtractionRequest": {
"@odata.type": "#ThomsonReuters.Dss.Api.Extractions.ExtractionRequests.TickHistoryTimeAndSalesExtractionRequest",
"ContentFieldNames": ["Quote - Bid Price","Quote - Bid Size","Quote - Ask Price","Quote - Ask Size"],
"IdentifierList": {
"@odata.type": "#ThomsonReuters.Dss.Api.Extractions.ExtractionRequests.InstrumentIdentifierList",
"InstrumentIdentifiers": [{
"Identifier": "LGOTJ9",
"IdentifierType": "Ric"
}],
"UseUserPreferencesForValidationOptions": "true"
},
"Condition": {
"MessageTimeStampIn": "GmtUtc",
"ApplyCorrectionsAndCancellations": "false",
"ReportDateRangeType": "Range",
"QueryStartDate": "2019-01-02T00:00:00.000Z",
"QueryEndDate": "2019-01-03T00:00:00.000Z",
"DisplaySourceRIC":"false"
}
}
}Could you maybe try it at your end with LGOTG9 and LGOTH9 because I know these two both work at my end.
Thanks again!
0 -
Hello @sanchezTJUK,
On my side, this request has resulted in notes
"
"Notes": [
"Extraction Services Version 14.5.42174 (21810bfe19a8), Built Feb 4 2021 20:25:13\nUser ID: 19945\nExtraction ID: 2000000226250284\nSchedule: 0x0771fae45477448a (ID = 0x0000000000000000)\nInput List (1 items): (ID = 0x0771fae45477448a) Created: 02/16/2021 16:41:12 Last Modified: 02/16/2021 16:41:12\nReport Template (4 fields): _OnD_0x0771fae45477448a (ID = 0x0771fae45497448a) Created: 02/16/2021 16:41:07 Last Modified: 02/16/2021 16:41:07\nSchedule dispatched via message queue (0x0771fae45477448a)\nSchedule Time: 02/16/2021 16:41:08\nProcessing started at 02/16/2021 16:41:08\nProcessing completed successfully at 02/16/2021 16:41:12\nExtraction finished at 02/16/2021 16:41:12 UTC, with servers: x03q13\nHistorical Instrument <RIC,LGOTJ9> expanded to 0 RICS.\nReport suppressed because there are no instruments\n"
]"
0 -
Therefore, at that time interval, the instrument appears not to have been valid.
Next, I have checked when the instrument was valid (it is an expired future), by running HistoricalReference extraction request:
{
"ExtractionRequest": {
"@odata.type": "#ThomsonReuters.Dss.Api.Extractions.ExtractionRequests.HistoricalReferenceExtractionRequest",
"ContentFieldNames": [
"RIC",
"Start date",
"Change Date",
"Expiration Date"
],
"IdentifierList": {
"@odata.type": "#ThomsonReuters.Dss.Api.Extractions.ExtractionRequests.InstrumentIdentifierList",
"InstrumentIdentifiers": [
{
"Identifier": "LGOTJ9",
"IdentifierType": "Ric"
}
],
"ValidationOptions": {
"AllowHistoricalInstruments": true
},
"UseUserPreferencesForValidationOptions": false
},
"Condition": {
"StartDate": "2007-01-01T00:00:00.000Z",
"EndDate": "2020-08-01T23:59:59.000Z"
}
}
}Resulting in
{
"@odata.context": "https://hosted.datascopeapi.reuters.com/RestApi/v1/$metadata#ThomsonReuters.Dss.Api.Extractions.ExtractionRequests.ExtractionResult",
"Contents": [
{
"IdentifierType": "Ric",
"Identifier": "LGOTJ9",
"RIC": "LGOTJ9",
"Start date": null,
"Change Date": "2019-01-10",
"Expiration Date": "2019-04-11"
},
{
"IdentifierType": "Ric",
"Identifier": "LGOTJ9",
"RIC": "LGOTJ9^1",
"Start date": null,
"Change Date": "2019-04-15",
"Expiration Date": "2019-04-11"
}
],I am not a content expert, but having read this, I would expect valid data 2019-01-10 - 2019-04-11
So I have next run:
{
"ExtractionRequest": {
"@odata.type": "#ThomsonReuters.Dss.Api.Extractions.ExtractionRequests.TickHistoryTimeAndSalesExtractionRequest",
"ContentFieldNames": ["Quote - Bid Price","Quote - Bid Size","Quote - Ask Price","Quote - Ask Size"],
"IdentifierList": {
"@odata.type": "#ThomsonReuters.Dss.Api.Extractions.ExtractionRequests.InstrumentIdentifierList",
"ValidationOptions": {
"AllowHistoricalInstruments": true
},
"UseUserPreferencesForValidationOptions": false,
"InstrumentIdentifiers": [{
"Identifier": "LGOTJ9",
"IdentifierType": "Ric"
}]
},
"Condition": {
"MessageTimeStampIn": "GmtUtc",
"ApplyCorrectionsAndCancellations": "false",
"ReportDateRangeType": "Range",
"QueryStartDate": "2019-01-02T00:00:00.000Z",
"QueryEndDate": "2019-04-11T00:00:00.000Z",
"DisplaySourceRIC":"false"
}
}
}And that have resulted in :
#RIC,Domain,Date-Time,GMT Offset,Type,Bid Price,Bid Size,Ask Price,Ask Size
LGOTJ9,Market Price,2019-01-10T02:37:16.198936629Z,+0,Quote,-0.25,500,,
LGOTJ9,Market Price,2019-01-10T02:37:20.331555914Z,+0,Quote,-0.25,500,0.25,500
LGOTJ9,Market Price,2019-01-10T08:14:59.704940277Z,+0,Quote,-0.25,525,0.25,500
LGOTJ9,Market Price,2019-01-10T08:15:04.505137097Z,+0,Quote,-0.25,525,0.25,525
LGOTJ9,Market Price,2019-01-10T16:14:12.123438568Z,+0,Quote,-0.25,525,0.25,500
LGOTJ9,Market Price,2019-01-10T16:21:34.416871597Z,+0,Quote,-0.25,325,0.25,500
...
LGOTJ9,Market Price,2019-04-10T15:29:58.146792544Z,+1,Quote,-0.5,32,0.25,25
LGOTJ9,Market Price,2019-04-10T15:29:59.102828245Z,+1,Quote,-0.5,32,0.25,15
LGOTJ9,Market Price,2019-04-10T15:29:59.126852382Z,+1,Quote,-0.25,10,0.25,15
LGOTJ9,Market Price,2019-04-10T15:29:59.226886984Z,+1,Quote,-0.5,37,0.25,15
LGOTJ9,Market Price,2019-04-10T15:29:59.554835560Z,+1,Quote,-0.5,42,0.25,15
LGOTJ9,Market Price,2019-04-10T15:29:59.686869731Z,+1,Quote,-0.25,10,0.25,15
LGOTJ9,Market Price,2019-04-10T15:30:00.114938765Z,+1,Quote,,,0.25,15So this is the validity of the instrument and the content that is available for it, as a result.
If you would like a more in-depth consultation on content, why the future was only available at this time, for a customer, the best approach is to directly refer the question to RTH Content experts via Refinitiv Helpdesk Online -> Content -> RTH.
Hope this helps
0 -
Thanks a lot for your time, really appreciate it.
0 -
Hi Zoya,
If I want to run a historical reference request like you've done above, does my request url have to change?
0
Categories
- All Categories
- 3 Polls
- 6 AHS
- 36 Alpha
- 166 App Studio
- 6 Block Chain
- 4 Bot Platform
- 18 Connected Risk APIs
- 47 Data Fusion
- 34 Data Model Discovery
- 690 Datastream
- 1.4K DSS
- 629 Eikon COM
- 5.2K Eikon Data APIs
- 11 Electronic Trading
- 1 Generic FIX
- 7 Local Bank Node API
- 3 Trading API
- 2.9K Elektron
- 1.4K EMA
- 255 ETA
- 559 WebSocket API
- 39 FX Venues
- 15 FX Market Data
- 1 FX Post Trade
- 1 FX Trading - Matching
- 12 FX Trading – RFQ Maker
- 5 Intelligent Tagging
- 2 Legal One
- 25 Messenger Bot
- 3 Messenger Side by Side
- 9 ONESOURCE
- 7 Indirect Tax
- 60 Open Calais
- 279 Open PermID
- 45 Entity Search
- 2 Org ID
- 1 PAM
- PAM - Logging
- 6 Product Insight
- Project Tracking
- ProView
- ProView Internal
- 23 RDMS
- 2K Refinitiv Data Platform
- 716 Refinitiv Data Platform Libraries
- 4 LSEG Due Diligence
- LSEG Due Diligence Portal API
- 4 Refinitiv Due Dilligence Centre
- Rose's Space
- 1.2K Screening
- 18 Qual-ID API
- 13 Screening Deployed
- 23 Screening Online
- 12 World-Check Customer Risk Screener
- 1K World-Check One
- 46 World-Check One Zero Footprint
- 45 Side by Side Integration API
- 2 Test Space
- 3 Thomson One Smart
- 10 TR Knowledge Graph
- 151 Transactions
- 143 REDI API
- 1.8K TREP APIs
- 4 CAT
- 27 DACS Station
- 121 Open DACS
- 1.1K RFA
- 106 UPA
- 194 TREP Infrastructure
- 229 TRKD
- 918 TRTH
- 5 Velocity Analytics
- 9 Wealth Management Web Services
- 95 Workspace SDK
- 11 Element Framework
- 5 Grid
- 19 World-Check Data File
- 1 Yield Book Analytics
- 48 中文论坛