question

Upvotes
Accepted
1 0 1 3

TRTH v2 file content issue

Hello,

I made a multi RIC request (described below) :

{
"ExtractedFileId": "VjF8MHgwNWY2MmEyY2RhMGIyZjk2fA",
"ReportExtractionId": "2000000007224811",
"ScheduleId": "0x05f629f161ab2f76",
"FileType": "Full",
"ExtractedFileName": "_OnD_0x05f629f161ab2f76.csv.gz",
"LastWriteTimeUtc": "2017-11-24T15:05:28.000Z",
"ContentsExists": true,
"Size": 2329027
}

When downloading i get a file containing only one ric.

Could you advise please ?

tick-history-rest-api
icon clock
10 |1500

Up to 2 attachments (including images) can be used with a maximum of 512.0 KiB each and 1.0 MiB total.

@said.aitabaid-ext, can you please post the end point and body of your request, as well as the output data, and extraction notes if possible. They will help us understand the issue and deliver a response.

hello and thanks for the reply

below the requested information

Sincerly

Upvotes
Accepted
13.7k 26 8 12

@said.aitabaid-ext, using AWS downloads might solve the issue, see this similar query.

icon clock
10 |1500

Up to 2 attachments (including images) can be used with a maximum of 512.0 KiB each and 1.0 MiB total.

@said.aitabaid-ext Have you tried AWS downloads yet? Can it solve this problem?

Upvotes
1 0 1 3

{
"ExtractionRequest" : {
"@odata.type" : "#ThomsonReuters.Dss.Api.Extractions.ExtractionRequests.TickHistoryTimeAndSalesExtractionRequest",
"ContentFieldNames" : [ "Quote - Bid Price", "Quote - Bid Size", "Quote - Number of Buyers", "Quote - Ask Price", "Quote - Ask Size", "Quote - Number of Sellers", "Trade - Price", "Trade - Volume" ],
"IdentifierList" : {
"@odata.type" : "#ThomsonReuters.Dss.Api.Extractions.ExtractionRequests.InstrumentIdentifierList",
"InstrumentIdentifiers" : [ {
"Identifier" : "EDF.PA",
"IdentifierType" : "Ric"
}, {
"Identifier" : "AIR.PA",
"IdentifierType" : "Ric"
}, {
"Identifier" : "ALCLS.PA",
"IdentifierType" : "Ric"
} ],
"ValidationOptions" : {
"AllowHistoricalInstruments" : true
},
"UseUserPreferencesForValidationOptions" : false
},
"Condition" : {
"MessageTimeStampIn" : "GmtUtc",
"ApplyCorrectionsAndCancellations" : true,
"ReportDateRangeType" : "Range",
"QueryStartDate" : "2017-11-15T23:00:00.000Z",
"QueryEndDate" : "2017-11-16T22:59:00.000Z",
"DisplaySourceRIC" : true,
"SortBy" : "SingleByRic",
"ExtractBy" : "Ric"
}
}
}

icon clock
10 |1500

Up to 2 attachments (including images) can be used with a maximum of 512.0 KiB each and 1.0 MiB total.

Upvotes
1 0 1 3

{"@odata.context":"https://hosted.datascopeapi.reuters.com/RestApi/v1/$metadata#RawExtractionResults/$entity","JobId":"0x05f7201dd5ab3036","Notes":["Extraction Services Version 11.2.37811 (51c11525fb7b), Built Nov 9 2017 14:30:01\nUser ID: 9013258\nExtraction ID: 2000000007804573\nSchedule: 0x05f7201dd5ab3036 (ID = 0x0000000000000000)\nInput List (3 items): (ID = 0x05f7201dd5ab3036) Created: 27-11-2017 15:45:34 Last Modified: 27-11-2017 15:45:34\nReport Template (8 fields): _OnD_0x05f7201dd5ab3036 (ID = 0x05f7201e2c6b3036) Created: 27-11-2017 15:43:40 Last Modified: 27-11-2017 15:43:40\nSchedule dispatched via message queue (0x05f7201dd5ab3036), Data source identifier (3912B9CF912C4B83AD83598C5E90B9D8)\nSchedule Time: 27-11-2017 15:43:43\nProcessing started at 27-11-2017 15:43:43\nProcessing completed successfully at 27-11-2017 15:45:34\nExtraction finished at 27-11-2017 14:45:34 UTC, with servers: tm04n01, TRTH (100.038 secs)\nInstrument <RIC,EDF.PA> expanded to 1 RIC: EDF.PA.\nInstrument <RIC,AIR.PA> expanded to 1 RIC: AIR.PA.\nInstrument <RIC,ALCLS.PA> expanded to 1 RIC: ALCLS.PA.\nQuota Message: INFO: Tick History Cash Quota Count Before Extraction: 4633; Instruments Approved for Extraction: 3; Tick History Cash Quota Count After Extraction: 4633, 61.6910785619174% of Limit; Tick History Cash Quota Limit: 7510\nManifest: #RIC,Domain,Start,End,Status,Count\nManifest: AIR.PA,Market Price,2017-11-16T03:07:08.554095130Z,2017-11-16T16:37:57.427417214Z,Active,198943\nManifest: ALCLS.PA,Market Price,2017-11-16T03:07:08.834758571Z,2017-11-16T16:50:55.413547815Z,Active,8609\nManifest: EDF.PA,Market Price,2017-11-16T06:15:00.311632845Z,2017-11-16T16:35:05.145556641Z,Active,63331\n"]}

icon clock
10 |1500

Up to 2 attachments (including images) can be used with a maximum of 512.0 KiB each and 1.0 MiB total.

Upvote
78.2k 246 52 72

This could be the problem mentioned in this advisory.

This advisory enables you to avoid a third-party decompression incompatibility that may be providing you with incomplete data, and may be doing so in a way that can be difficult to notice.

This advisory applies to all Tick History REST API extractions that return compressed data using any of the following report template types: Tick History Time and Sales, Tick History Market Depth, Tick History Intraday Summaries, and Tick History Raw.

You need to disable decompression in the HTTP client while downloading. Therefore, the application will get a raw gzip file instead of CSV data. Then, the application needs to decompress the gzip file in order to get the CSV data.

With the above extraction, you should see 270884 rows of data including a header row.

icon clock
10 |1500

Up to 2 attachments (including images) can be used with a maximum of 512.0 KiB each and 1.0 MiB total.

Hello,

I followed the instruction in the advisory (below the code used for the call )

String urlGet = basePath + "/Extractions/RawExtractionResults('0x05f7201dd5ab3036')/$value";
CloseableHttpClient httpclient = HttpClients.custom()
                .setRoutePlanner(routePlanner)
                .setSSLSocketFactory(sslsf)
                .disableContentCompression()
                .build();
HttpGet getCall = new HttpGet(urlGet);
  getCall.addHeader(HttpHeaders.AUTHORIZATION, "Token "+ sessionToken );   
CloseableHttpResponse response = httpclient.execute(getCall);

Upvotes
1 0 1 3
try {
                

GzipCompressorInputStream gis = new GzipCompressorInputStream( response
.getEntity().getContent(), true);

/*InputStream entityInputStream = response
.getEntity().getContent();*/
IOUtils.copy(gis,
fileOutputStream);
gis.close();
fileOutputStream.close();
} finally {
response.close();
}

I'm getting the following :

java.io.IOException: Input is not in the .gz format
at org.apache.commons.compress.compressors.gzip.GzipCompressorInputStream.init(GzipCompressorInputStream.java:162)

Could you advise please ?

Sincerly

icon clock
10 |1500

Up to 2 attachments (including images) can be used with a maximum of 512.0 KiB each and 1.0 MiB total.

Please try this code.

        InputStream is = responseGetData.getEntity().getContent();           
        BufferedReader br = new BufferedReader(new InputStreamReader(new GzipCompressorInputStream(new BufferedInputStream(is), true)));        
        String line = "";        
        while ((line = br.readLine()) != null) {            
           System.out.println(line);        
        }

Hello,

I'm still getting the same issue : java.io.IOException: Input is not in the .gz format.

The contentEncoding attribute in the response wrapped entity is null.

Could you advise please ?

Regards

Attached also a curl call for the same file

log.jpg (214.4 KiB)
Upvotes
1 1 1 0

Due to an outage of the Thomson Reuters Developer Community email notification system from 2:30 PM CDT on November 29th until 9:00 AM CDT on November 30th we are posting to all questions updated during this timeframe in order to ensure all customers receive all relevant email updates. Please review this question, and its associated answers, in order to ensure you are aware of any possible updates that may have occurred during this outage. We apologize for the delay and any inconvenience this has caused.

Matthew Logterman, J.D.

Sr. Product Manager, Thomson Reuters Developer Community

icon clock
10 |1500

Up to 2 attachments (including images) can be used with a maximum of 512.0 KiB each and 1.0 MiB total.

Upvotes
13.7k 26 8 12

@said.aitabaid-ex, are you using AWS downloads? I ask due to your comment stating: The contentEncoding attribute in the response wrapped entity is null. As explained in this article, Contrary to a non-AWS download, if you use AWS it does not return the Content-Encoding:gzip header in the response. Instead it sends this response header Content-Type:application/gzip.

That said, we have a Java sample, DSS2OnDemandTicksTRTH, available under the downloads tab, that downloads the result file and decompresses it. Although we recommend first saving the compressed data to file, and then reading and decompressing that, which the sample does, it also shows how to download and decompress on the fly (which seems to be what you want to do).

In the sample, see method awsRetrieveDataDecompressAndDisplayContents for an AWS download, or method retrieveDataDecompressAndDisplayContents for a non-AWS download. Note these 2 methods are different because AWS implies a redirection, which needs to be handled manually.

icon clock
10 |1500

Up to 2 attachments (including images) can be used with a maximum of 512.0 KiB each and 1.0 MiB total.

Upvotes
1 0 1 3

Hello,

We are not using aws downloads. Actually i used the

retrieveDataDecompressAndDisplayContents method from the

DSS2OnDemandTicksTRTH but still have the issue (

Input is not in the .gz format).

Could you please help ?

Regards


icon clock
10 |1500

Up to 2 attachments (including images) can be used with a maximum of 512.0 KiB each and 1.0 MiB total.

@said.aitabaid-ext, I shall try to reproduce your issue, and will come back to you soon.

I assume that your company's proxy may decompress the data and then send the csv data back to the application. It is similar to this answer.

You can verify this from a header in the HTTP response header that returns CSV data. If it contains the Via header, it means that the application is behind the proxy. The workaround is downloading a file from AWS.

Upvotes
13.7k 26 8 12

@said.aitabaid-ext, I tried to reproduce your issue, but could not, I receive GZIP compressed data as expected, and can save it directly, or decompress it on the fly.

Attached (dss2ondemandtickstrth.zip) is the Java code I used, a modified version of DSS2OnDemandTicksTRTH using your instrument list, range and parameters). It generates the same request as what you posted. My extraction notes are very similar to yours. The result contains data for the 3 RICs.

If this sample does not help, then I think we'd need to look at your code. If you don't want to post it here we can take that action offline.


icon clock
10 |1500

Up to 2 attachments (including images) can be used with a maximum of 512.0 KiB each and 1.0 MiB total.

Investigation taken offline. Might be related to the code, libraries or proxy. Will post explanation / solution when found.

Write an Answer

Hint: Notify or tag a user in this post by typing @username.

Up to 2 attachments (including images) can be used with a maximum of 512.0 KiB each and 1.0 MiB total.