Can extract only part of =ECI data even though entire data can be extracted via GUI of same TRTH ID

Please advise why we are unable to download entire data through API.

Hi,

We can only extract part of historical time and sales data for USCONC=ECI
and USCPF=ECI through TRTHv2 API.

I can extract entire historical records of over 700 rows through GUI of same TRTH ID, however only several
rows are extracted through API.

Only below 4 rows are returned even start date is May 15 and end date is Jun 18.

2015-05-08T12:11:11.160864350Z Trade 0.2 6
2015-05-08T17:42:53.187461400Z Trade 0.3 0.1 11
2015-05-11T11:41:25.784174700Z Trade 14 
2015-05-15T12:15:51.700763210Z Trade 0.2 0 32

TRTH ID: 9010877

Language: Ruby

TickHistoryTimeAndSales

IdentifierType=Ric、Identtifier=
USCPF=ECI

ValidationOptions: AllowOpenAccessInstrument true

Condition: MessageTimeSStampIn: LocalExchangeTime

ReportDateRangeType: Range

QueryStartDate 2015-05-01T00:00:00Z

QueryEndDate 2018-06-01T00:00:00Z

Preview: Content (later it was changed to None)

ExtractBy Ric

DisplaySourceRIC True

We picked up output csv results with returned JobID after ExtractionRaw.

We confirm that Extraction finished in relevant Notes file.

@{"@odata.context"=>"https://hosted.datascopeapi.reuters.com/RestApi/v1/$metadata#RawExtractionResults/$entity",
"JobId"=>"0x0636bbdb9d1b2f76",
"Notes"=>[
"Extraction Services Version 12.0.38980 (0bb9cdd6e646), Built Jun 6 2018 17:45:37
Processing started at 06/13/2018 14:42:44.
User ID: 9010877\r\nExtraction ID: 2000000031122766
Schedule: 0x0636bbdb9d1b2f76 (ID = 0x0000000000000000)
Input List (1 items): (ID = 0x0636bbdb9d1b2f76) Created: 06/13/2018 14:42:43 Last Modified: 06/13/2018 14:42:43
Instrument <RIC,USCONC=ECI> expanded to 1 RIC: USCONC=ECI.
Total instruments after instrument expansion = 1
Report Template (373 fields): _OnD_0x0636bbdb9d1b2f76 (ID = 0x0636bbdba00b2f76) Created: 06/13/2018 14:42:38 Last Modified: 06/13/2018 14:42:38
Schedule dispatched via message queue (0x0636bbdb9d1b2f76)
Schedule Time: 06/13/2018 14:42:40
Range Query from 2015-04-30T15:00:00.000 to 2018-05-31T15:00:00.000 (UTC)
Processing completed successfully at 06/13/2018 14:42:56, taking 12.685 Secs.
Extraction finished at 06/13/2018 05:42:56 UTC, with servers: x07q13, QSSHA1 (0.1 secs), TRTH (11.2 secs)"]}

Regards,

Find more posts tagged with

tick-history-rest-api

economic-data

Accepted answers

Christiaan Meihsl

@hiroko.goto1,

Browsers are not appropriate HTTP clients to use with TRTH.

Postman can be used for testing (note that it uses pure HTTP, not our .Net SDK).

Data requests need to be done using the GUI or the API, and if the API is used, programming must be done correctly, using the appropriate libraries. We deliver samples in several languages to illustrate how that is done.

There are no "anomalies" of TRTH gzip files, but you must be aware that TRTH uses concatenated gzip, which can cause issues if the HTTP client does not handle them properly. For more info on this topic see this advisory.

All comments

hiroko.goto1

Hi Christiaan, how can I edit this to public?

Christiaan Meihsl

@hiroko.goto1, you do not mention which fields you are requesting, but I suspect the GUI request and the REST API request do not contain the same field list, hence the difference of results you observe. I see for instance from your results that you are not requesting the "Trade - Forecast" field, which contains data.

I just tested. If I request all fields that contain data, I get the same results for a TRTH T&S query for instruments USCONC=ECI and USCPF=ECI, for the period from 1 May 2015 till 31 may 2018.

GUI request

I first made a TRTH T&S query with a template that contained all possible fields, for the same 2 instruments and date range. That delivered a fairly small file, in which there were 1486 rows (including the header row). Each instrument delivered slightly more than 700 rows. Most were of type Trade, a few were of type Reference Change.

Out of the 367 fields I requested, only the following fields delivered data:

#RIC, Domain, Date-Time, Type, Qualifiers, Actual, Prior, Revised, Forecast, Frcst High, Frcst Low, No. Frcts, Change Type, Old Value, New Value

API REST request

I then used Postman to query the same 2 instruments and date range, and requested all the fields that delivered data in the GUI request.

Endpoint: https://hosted.datascopeapi.reuters.com/RestApi/v1/Extractions/ExtractRaw

Request body:

{
  "ExtractionRequest": {
    "@odata.type": "#ThomsonReuters.Dss.Api.Extractions.ExtractionRequests.TickHistoryTimeAndSalesExtractionRequest",
    "ContentFieldNames": [ "Trade - Qualifiers", "Trade - Actual", "Trade - Prior", "Trade - Revised",
      "Trade - Forecast", "Trade - Forecast High", "Trade - Forecast Low", "Trade - Number of Forecasts",
      "Reference Change - Change Type", "Reference Change - Old Value", "Reference Change - New Value" ],
    "IdentifierList": {
      "@odata.type": "#ThomsonReuters.Dss.Api.Extractions.ExtractionRequests.InstrumentIdentifierList", 
      "InstrumentIdentifiers": [
      { "Identifier": "USCONC=ECI", "IdentifierType": "Ric" },
      { "Identifier": "USCPF=ECI", "IdentifierType": "Ric" }
      ],
      "ValidationOptions": { "AllowOpenAccessInstruments": true }
    },
    "Condition": {
      "MessageTimeStampIn": "LocalExchangeTime",
      "ApplyCorrectionsAndCancellations": false,
      "ReportDateRangeType": "Range",
      "QueryStartDate": "2015-05-01T00:00:00.000Z",
      "QueryEndDate": "2018-06-01T00:00:00.000Z",
      "DisplaySourceRIC": true
    }
  }
}

Extraction notes:

    "Notes": [
        "Extraction Services Version 12.0.38980 (0bb9cdd6e646), Built Jun  6 2018 17:41:16\nUser ID: 33314\nExtraction ID: 2000000031852333\nSchedule: 0x0638b40200cb2f96 (ID = 0x0000000000000000)\nInput List (2 items):  (ID = 0x0638b40200cb2f96) Created: 19062018 10:35:55 AM Last Modified: 19062018 10:35:55 AM\nReport Template (11 fields): _OnD_0x0638b40200cb2f96 (ID = 0x0638b40205ab2f96) Created: 19062018 10:33:18 AM Last Modified: 19062018 10:33:18 AM\nSchedule dispatched via message queue (0x0638b40200cb2f96), Data source identifier (3AF920E032FD4DABB3B9406498E8FF29)\nSchedule Time: 19062018 10:33:19 AM\nProcessing started at 19062018 10:33:19 AM\nProcessing completed successfully at 19062018 10:35:55 AM\nExtraction finished at 19062018 08:35:55 AM UTC, with servers: tm04n01, TRTH (148.346 secs)\nInstrument <RIC,USCONC=ECI> expanded to 1 RIC: USCONC=ECI.\nInstrument <RIC,USCPF=ECI> expanded to 1 RIC: USCPF=ECI.\nTotal instruments after instrument expansion = 2\nRange Query from 2015-04-30T22:00:00.000 to 2018-05-31T22:00:00.000 (UTC)\nQuota Message: INFO: Tick History Cash Quota Count Before Extraction: 26446; Instruments Approved for Extraction: 0; Tick History Cash Quota Count After Extraction: 26446, 2644.6% of Limit; Tick History Cash Quota Limit: 1000\nQuota Message: ERROR: The RIC 'USCONC=ECI' in the request would exceed your quota limits. Adjust your input list to continue.\nQuota Message: ERROR: The RIC 'USCPF=ECI' in the request would exceed your quota limits. Adjust your input list to continue.\nQuota Message: WARNING: Tick History Cash Quota has been reached or exceeded\nQuota Message: Note: Quota has exceeded, however, it is not being enforced at this time but you can still make your extractions and instruments are still being counted. Please contact your Account Manager for questions. \nManifest: #RIC,Domain,Start,End,Status,Count\nManifest: USCONC=ECI,Market Price,2015-05-12T14:00:57.840610750Z,2018-05-29T15:00:46.257032016Z,Active,776\nManifest: USCPF=ECI,Market Price,2015-05-08T12:11:11.160864350Z,2018-05-10T12:30:02.537586691Z,Active,709\n"
    ]

The data file contains exactly the same as what was retrieved using the GUI, 1486 lines in total.

Here are the first lines:

#RIC,Alias Underlying RIC,Domain,Date-Time,Type,Qualifiers,Actual,Prior,Revised,Forecast,Frcst High,Frcst Low,No. Frcts,Change Type,Old Value,New Value
USCONC=ECI,,Market Price,2015-05-12T14:00:57.840610750Z,Trade,Apr[CTBTR_1LL];[GV3_TEXT],,,,,,,,,,
USCONC=ECI,,Market Price,2015-05-12T14:00:59.163783520Z,Trade,May[CTBTR_1LL];[GV3_TEXT],,95.2,,,,,,,,
USCONC=ECI,,Market Price,2015-05-15T12:15:51.968994250Z,Trade,May[CTBTR_1LL];[GV3_TEXT],,,,97,98.1,95,5,,,

Here are the first lines for USCPF=ECI:

USCPF=ECI,,Market Price,2015-05-08T12:11:11.160864350Z,Trade,Mar[CTBTR_1LL];%[GV3_TEXT],0.20000000000000001,,,,,,6,,,
USCPF=ECI,,Market Price,2015-05-08T12:30:42.175861660Z,Trade,Mar[CTBTR_1LL];%[GV3_TEXT],,,,,,,,,,
USCPF=ECI,,Market Price,2015-05-08T12:30:43.118808740Z,Trade,Apr[CTBTR_1LL];%[GV3_TEXT],,,,,,,,,,
USCPF=ECI,,Market Price,2015-05-08T17:42:53.187461400Z,Trade,Apr[CTBTR_1LL];%[GV3_TEXT],,,,0.2,0.3,0.1,11,,,
USCPF=ECI,,Market Price,2015-05-11T11:41:25.784174700Z,Trade,Apr[CTBTR_1LL];%[GV3_TEXT],,,,,,,14,,,
USCPF=ECI,,Market Price,2015-05-15T12:15:51.700763210Z,Trade,Apr[CTBTR_1LL];%[GV3_TEXT],,,,,0.2,0,32,,,

And here are the last:

USCPF=ECI,,Market Price,2018-05-08T09:45:39.457826208Z,Trade,Apr[CTBTR_1LL];%[GV3_TEXT];Apr 2018[FCAST_PRD],,,,,,,71,,,
USCPF=ECI,,Market Price,2018-05-10T12:30:01.762965071Z,Trade,Apr[CTBTR_1LL];%[GV3_TEXT];Apr 2018[FCAST_PRD],+0.1,,,,,,,,,
USCPF=ECI,,Market Price,2018-05-10T12:30:02.537586691Z,Trade,Apr[CTBTR_1LL];%[GV3_TEXT];Apr 2018[FCAST_PRD],+0.1,,,,,,,,,

hiroko.goto1

Hi Christiaan,

Thank you for your support on this issue.

We tried REST API extraction several times and still unable to extract all available data rows for USCONC=ECI RIC.

We can still able to extract only several raws of historical records.

Can you please look at the following TRTH ID and two Job IDs and advise what is wrong?

TRTH ID: xxxxxxx

password: xxxxxxxx

JSON　（Specified Prefer: respond-async in Request header）

{"ExtractionRequest":{
  "@odata.type":"#ThomsonReuters.Dss.Api.Extractions.ExtractionRequests.TickHistoryTimeAndSalesExtractionRequest",
  "ContentFieldNames":[
    "Reference Change - Change Type","Reference Change - New Value","Reference Change - Old Value",
    "Trade - Actual","Trade - Forecast","Trade - Forecast High","Trade - Forecast Low",
    "Trade - Number of Forecasts","Trade - Prior","Trade - Qualifiers","Trade - Revised"
  ],
  "IdentifierList":{
    "@odata.type":"#ThomsonReuters.Dss.Api.Extractions.ExtractionRequests.InstrumentIdentifierList",
    "InstrumentIdentifiers":[
      {"Identifier":"USCPF=ECI","IdentifierType":"Ric"},
      {"Identifier":"USCONC=ECI","IdentifierType":"Ric"}
    ],
    "ValidationOptions":{"AllowOpenAccessInstruments":true}
  },
  "Condition":{
    "MessageTimeStampIn":"LocalExchangeTime",
    "ApplyCorrectionsAndCancellations":"false",
    "ReportDateRangeType":"Range",
    "QueryStartDate":"2015-05-01T00:00:00.000Z",
    "QueryEndDate":"2018-06-01T00:00:00.000Z",
    "DisplaySourceRIC":"true"
  }
 }
}

Resopnse 201：Accepted

Responded JOBID => 0x063a8e835b1b2f86

https://hosted.datascopeapi.reuters.com/RestApi/v1/Extractions/RawExtractionResults('0x063a8e835b1b2f86')/$value

responded data was as follows:

#RIC,Alias Underlying RIC,Domain,Date-Time,Type,Qualifiers,Actual,Prior,Revised,Forecast,Frcst High,Frcst Low,No. Frcts,Change Type,Old Value,New Value
USCONC=ECI,,Market Price,2015-05-12T14:00:57.840610750Z,Trade,Apr[CTBTR_1LL];[GV3_TEXT],,,,,,,,,,
USCONC=ECI,,Market Price,2015-05-12T14:00:59.163783520Z,Trade,May[CTBTR_1LL];[GV3_TEXT],,95.2,,,,,,,,
USCONC=ECI,,Market Price,2015-05-15T12:15:51.968994250Z,Trade,May[CTBTR_1LL];[GV3_TEXT],,,,97,98.1,95,5,,,
USCONC=ECI,,Market Price,2015-05-15T18:11:42.735467740Z,Trade,May[CTBTR_1LL];[GV3_TEXT],,,,95.2,,91,11,,,
USCONC=ECI,,Market Price,2015-05-18T09:54:05.613603120Z,Trade,May[CTBTR_1LL];[GV3_TEXT],,,,96.1,,,12,,,<br>

Even though we didnt specify Prefer: respond-async, extracted values were identical to above.

Job ID extraction: 0x063a8d06f81b2f96

Regards,

Christiaan Meihsl

@hiroko.goto1,

I cannot look at a specific JobID, I don't have that level of access.

For the dev team, to be able to analyze a specific request, please set a unique Client-Session-Id (it must be unique for every single request), and give us the Client-Session-Id and returned corresponding Request-Execution-Correlation-Id for the call to debug. Details can be found here.

3 comments on your latest post of today:

I guess the response you received was 202: Accepted, and not 201 (which is Created).
The request and response data seem not to match, the field order is not the same.
The first 5 lines of data you posted for USCONC=ECI seem correct, I receive the same.

New test today:

I just tried with the request as you posted it earlier today, which looks very much like the one I posted 6 days ago, except for the order of fields and instruments. I got a 202 accepted, and waited a while for the request to complete, regularly polling the location URL until it delivered the JobId. Using that I retrieved the data, again I received 1486 lines of data (1 header, 776 for USCONC=ECI and 709 for USCPF=ECI). The first 6 lines exactly match what you posted. All this seems fine to me.

Please answer these queries if you still think there is an issue:

Do you receive data for both instruments, or only one ?
How many lines of data do you receive for each instrument ?
Please post the extraction notes received with the JobId.

hiroko.goto1

Hi Christiaan,

Thank you. Let me respond back to your questions/requests.

Please find the result of request as follows:

JSON Request:

{"ExtractionRequest":{
  "@odata.type":"#ThomsonReuters.Dss.Api.Extractions.ExtractionRequests.TickHistoryTimeAndSalesExtractionRequest",
  "ContentFieldNames":["Reference Change - Change Type","Reference Change - New Value","Reference Change - Old Value","Trade - Actual","Trade - Forecast","Trade - Forecast High","Trade - Forecast Low","Trade - Number of Forecasts","Trade - Prior","Trade - Qualifiers","Trade - Revised"],
  "IdentifierList":{
    "@odata.type":"#ThomsonReuters.Dss.Api.Extractions.ExtractionRequests.InstrumentIdentifierList",
    "InstrumentIdentifiers":[
      {"Identifier":"USCPF=ECI","IdentifierType":"Ric"},
      {"Identifier":"USCONC=ECI","IdentifierType":"Ric"}
    ],
    "ValidationOptions":{"AllowOpenAccessInstruments":true}
  },
  "Condition":{
    "MessageTimeStampIn":"LocalExchangeTime",
    "ApplyCorrectionsAndCancellations":"false",
    "ReportDateRangeType":"Range",
    "QueryStartDate":"2015-05-01T00:00:00.000Z",
    "QueryEndDate":"2018-06-01T00:00:00.000Z",
    "DisplaySourceRIC":"true"
  }
}}

ClientSessionId and CorrelationId returned by above request.

ClientSessionId:f9a189ff-3054-4ea2-829c-f148914aa465

CorrelationId:dccd3c2c-9855-4d7b-aaf9-7d66b89d2546

JSON Response:

@{"@odata.context"=>"https://hosted.datascopeapi.reuters.com/RestApi/v1/$metadata#RawExtractionResults/$entity",
"JobId"=>"0x063b2ee5ab2b2f86", "Notes"=>["Extraction
Services Version 12.0.39050 (262c8b8409ca), Built Jun 12 2018 20:14:54\nUser
ID: xxxxxxx\nExtraction ID: 2000000032748502\nSchedule: 0x063b2ee5ab2b2f86 (ID
= 0x0000000000000000)\nInput List (2 items):  (ID = 0x063b2ee5ab2b2f86)
Created: 06/27/2018 10:32:26 Last Modified: 06/27/2018 10:32:26\nReport
Template (11 fields): _OnD_0x063b2ee5ab2b2f86 (ID = 0x063b2ee5b00b2f86)
Created: 06/27/2018 10:28:47 Last Modified: 06/27/2018 10:28:47\nSchedule
dispatched via message queue (0x063b2ee5ab2b2f86), Data source identifier
(F5BBEEBBCD9045D4822338B255021ED8)\nSchedule Time: 06/27/2018
10:28:50\nProcessing started at 06/27/2018 10:28:50\nProcessing completed
successfully at 06/27/2018 10:32:26\nExtraction finished at 06/27/2018 01:32:26
UTC, with servers: tm14n03, TRTH (193.115 secs)\nInstrument
<RIC,USCPF=ECI> expanded to 1 RIC: USCPF=ECI.\nInstrument
<RIC,USCONC=ECI> expanded to 1 RIC: USCONC=ECI.\nTotal instruments after
instrument expansion = 2\nRange Query from 2015-04-30T15:00:00.000 to 2018-05-31T15:00:00.000
(UTC)\nQuota Message: INFO: Tick History Cash Quota Count Before Extraction:
47031; Instruments Approved for Extraction: 0; Tick History Cash Quota Count
After Extraction: 47031, 9406.2% of Limit; Tick History Cash Quota Limit:
500\nQuota Message: ERROR: The RIC 'USCONC=ECI' in the request would exceed
your quota limits. Adjust your input list to continue.\nQuota Message: ERROR:
The RIC 'USCPF=ECI' in the request would exceed your quota limits. Adjust your
input list to continue.\nQuota Message: WARNING: Tick History Cash Quota has
been reached or exceeded\nQuota Message: Note: Quota has exceeded, however, it
is not being enforced at this time but you can still make your extractions and
instruments are still being counted. Please contact your Account Manager for
questions. \nManifest: #RIC,Domain,Start,End,Status,Count\nManifest:
USCONC=ECI,Market
Price,2015-05-12T14:00:57.840610750Z,2018-05-29T15:00:46.257032016Z,Active,776\nManifest:
USCPF=ECI,Market Price,2015-05-08T12:11:11.160864350Z,2018-05-10T12:30:02.537586691Z,Active,709\n"]}

All data rows fetched by this JobID (we havent gotten any SessionId abd CorrelationId for this request)

#RIC Alias Underlying RIC Domain Date-Time Type Qualifiers Actual Prior Revised Forecast   Frcst High Frcst Low No. Frcts Change Type Old Value New Value
USCONC=ECI Market Price 2015-05-12T14:00:57.840610750Z Trade Apr[CTBTR_1LL];[GV3_TEXT]
USCONC=ECI Market Price 2015-05-12T14:00:59.163783520Z Trade May[CTBTR_1LL];[GV3_TEXT] 95.2
USCONC=ECI Market Price 2015-05-15T12:15:51.968994250Z Trade May[CTBTR_1LL];[GV3_TEXT] 97 98.1 95 5
USCONC=ECI Market Price 2015-05-15T18:11:42.735467740Z Trade May[CTBTR_1LL];[GV3_TEXT] 95.2 91 11
USCONC=ECI Market Price 2015-05-18T09:54:05.613603120Z Trade May[CTBTR_1LL];[GV3_TEXT] 96.1 12

Do we receive data for both instruments, or only one ? -> As you see above, we only got 5 rows as above for USCONC=ECI.
How many lines of data do you receive for each instrument ? -> Only 5 rows.
Please post the extraction notes received with the JobId.-> Pls see notes file as above. As you see, we receive an error that 'WARNING: Tick History Cash Quota has been reached or exceeded\nQuota Message: Note: Quota has exceeded, however, it is not being enforced at this time but you can still make your extractions and instruments are still being counted.', however this is an internal ID for test and I was told before that internal TRTH IDs will not be affected by Quota even though we receive an error in notes file. The message also says it is not being enforced at this time, and we can still extract full records of USCONC=ECI etc via TRTH GUI of the same user ID. Therefore I dont think this quota message would be an issue.

Thank you and regards,

Hiroko

Christiaan Meihsl

@hiroko.goto1, this is weird, I do not understand why you only receive 5 rows. I copied your request as is into Postman, received similar quota warnings in the extraction notes, and then received 1486 lines of response.

Are you using this endpoint: https://hosted.datascopeapi.reuters.com/RestApi/v1/Extractions/ExtractRaw ?
Are you using Postman ?

If you are not using Postman:

Can you try inside Postman and tell me what you get ?
Can you send me the code you are using ?

hiroko.goto1

Hi Christiaan,

Thank you.

Although our tech here was able to extract
full data of =ECI via Postman, there are some points we need to confirm.

We tried the followings:

Used Firefox: the first 669 bytes were
received (same as previous 5 rows)

Used Chrome: the first 669 bytes were retrieved
as CSV, and rests were in binary.

Used Edge: never completed downloads

It seems that a tool which is used .NET Framework
like PostMan can extract correctly compressed gunzip files, but we cannot
extract gunzpi files by other HTTP client.
Have you heard of any anomalies of TRTH gunzip files? It seems that only compress/extraction through
.NET Framework is successful.

Would you try the followings?

1)Please Return response data that can be expanded with major browsers
like Chrome, Firefox, Edge

2) If gzip is not
specified by Accept-Encoding, please return response data in plaintext
if possibile

Regards,

Christiaan Meihsl

@hiroko.goto1,

Browsers are not appropriate HTTP clients to use with TRTH.

Postman can be used for testing (note that it uses pure HTTP, not our .Net SDK).

hiroko.goto1

Hi Christiaan,

Thank you very much!

We finally understood what the issue is.

Our tech here appreciate your great support.

Please close the case for now.

Thank you and best regards,

Hiroko

EXPLORE OUR SITES