Why not all instruments received if AutoDecompression is to true on ExtractionContext for TickHis...

...toryTimeAndSaleExtractionRequest?
Hi,
We are trying to migrate to TickHistory v2 using the REST API in .Net C#.
Currently we are retrieving a gzip file containing the following information for each Equity Spot. But when I set AutoDecompression option on ExtractionContext to true, then in the received file (.csv file) I do not receive all the instruments I have specified. But if I do not set this option AutoDecompression then I do receive all the instruments in the received zip file (gz file for example). The template I am using is TickHistoryTimeAndSaleExtractionRequest.
Do you know the reason for this?
Thanks.
Best Answer
-
@mojtaba.danai, to help answer your queries, I have just updated .Net Tutorial 5. It now shows how you can:
- Save the extracted data directly as a compressed file, on hard disk.
- Save the extracted data as a CSV file, for later usage (out of scope here).
- Read and decompress the data from the compressed file that was saved to hard disk.
- Read and decompress the data on the fly, as it is received from the server. This variant also saves the data as a second CSV file (identical to the one created by the other variant).
It also shows how to download the data from AWS (the Amazon Web Service cloud), which is faster than the standard download.You can download this new code, it is part of the .Net SDK Tutorials code package. The code uses the .Net SDK.
Note: on the fly processing is not recommended for large data sets. Instead, we recommend to save the compressed data to file, and then to read,
decompress and treat it from there.0
Answers
-
This problem has been mentioned in this document.
Most Tick History reports deliver output as a gzip file. If a report is large, it delivers its output as
several smaller gzip files concatenated into a single large gzip file.
If your HTTP client does not support concatenated gzip files, you may not receive some of your
output. At this time, most HTTP clients do not fully support concatenated gzip files.0 -
@Jirapongse
Thanks for the reply. As mentioned we are using the REST API in .Net C#. How do we check out HTTP client through .Net is supporting concatenated gzip files?
If we are using the .NET C# , how this is using the HTTP client?
If the decompression flag is not set, then we do get a large gz file (zip file) containing every thing. And if the decompression flag is set to true then the file of course is not compressed and contains only some of the instruments.
0 -
We have been using HttpClient or HttpClientHandler in our code. Actually we wish to download the whole zip file and decompress it afterwards after the download is finished. We have been using System.IO.Compression.GzipStream in another PowerShell script to decompress the downloaded file from Reuters FTP (version 1 of the api).
As mentioned we are suing ExtractionsContext to save the gzip file:
using (var response = _extractionContext.RawExtractionResultOperations.GetReadStream(result)) { using (var fileStream = File.Create(fullFileName)) { response.Stream.CopyTo(fileStream); } }
as mentioned in Reuters sample code provided in the TickHistoryExamples.cs file.
I am not sure we have to use HttpClientHandler or HttpClient.
There is no sample for code using this.
And as mentioned what if we do wish to decompress the downloaded file using System.IO.Compression.GzipStream in another PowerShell script?
Thanks
0 -
Internally, ExtractionsContext uses .NET HTTP Client (HttpClient and HttpClientHandler) to send and receive the response.
To avoid this problem and get the data in gzip file, the application needs to set AutomaticDecompression to false in ExtractionsContext to disable AutomaticDecompression in .NET HTTP Client.
extractionsContext.Options.AutomaticDecompression = false;
Therefore, if you are using ExtractionsContext, you don't need to use HttpClientHandler or HttpClient anymore.
0 -
@mojtaba.danai, have you tried the code of .Net Tutorial 5 ? It uses our .Net SDK, thus simplifying your coding work (there is no need to manage the HTTP connection yourself), and illustrates both saving the compressed data to file, and decompressing on the fly. It follows the recommendations outlined in the advisory. I believe it works for large data sets as well.
That said, we recommend you save the compressed data as a file on local hard disk, and then read it from there, instead of decompressing it on the fly.
0 -
I am using the following code to get the gz file at the moment:
using (var response = _extractionContext.RawExtractionResultOperations.GetReadStream(result)) { GZipStream using (var fileStream = File.Create(fullFileName)) { response.Stream.CopyTo(fileStream); } }
fullFileName is the name of the gz file (abcd.csv.gz).
- Do you know where I can download the sample code for decompressing the gz file runtime using the ExtractionsContext?
- Can I keep the above code and make decompression afterwards?
- Or What mehtod shall use on extractionsContext to decompress the file?
0 -
Unfortunately the Reuters documentation on this is very poor. We are not sure how to decompress the gz file? Please see above comment
0 -
Unfortunately the Reuters documentation on this is very poor. We are not sure how to decompress the gz file? Please see below comment
0 -
It was mentioned in the advisory on page 4 (Decompress the Gzip file). It recommends using SharpZipLib to decompress your gzip output files. It is available from
ICSharpCode and NuGet.You can apply the above methods to your code. For example, if you need to save the output in a CSV file, you can use the below code:
var response = extractionsContext.RawExtractionResultOperations.GetReadStream(result);
using (var gzip = new GZipInputStream(response.Stream))
{
using (var outputStream = File.Create("output.csv"))
{
gzip.CopyTo(outputStream);
}
}0 -
@mojtaba.danai, I shall modify .Net SDK Tutorial Code sample 5 now to answer your queries. I'll post it as soon as it is ready. It will be available under the downloads tab.
0 -
It would be nice if we could use the standard .NET Ssystem.IO.Compression library to unzip the files, but unfortunately this doesn't work. Presumably this is why TR have included a third-party library in their tutorial code.
The root of the problem seems to be that the TR code that compresses the data file incorrectly records the size of the original file (this is stored in the last 4 byes of a gzip file). 7Zip, SharpZipLib and other utilities are robust enough to handle this error, but the Microsoft library isn't.
It would be great if TR could fix their compression code so that the gzip file is correctly formed and we wouldn't have to use third-party libraries.
0
Categories
- All Categories
- 3 Polls
- 6 AHS
- 36 Alpha
- 166 App Studio
- 6 Block Chain
- 4 Bot Platform
- 18 Connected Risk APIs
- 47 Data Fusion
- 34 Data Model Discovery
- 688 Datastream
- 1.4K DSS
- 625 Eikon COM
- 5.2K Eikon Data APIs
- 11 Electronic Trading
- 1 Generic FIX
- 7 Local Bank Node API
- 3 Trading API
- 2.9K Elektron
- 1.4K EMA
- 255 ETA
- 557 WebSocket API
- 38 FX Venues
- 14 FX Market Data
- 1 FX Post Trade
- 1 FX Trading - Matching
- 12 FX Trading – RFQ Maker
- 5 Intelligent Tagging
- 2 Legal One
- 23 Messenger Bot
- 3 Messenger Side by Side
- 9 ONESOURCE
- 7 Indirect Tax
- 60 Open Calais
- 276 Open PermID
- 44 Entity Search
- 2 Org ID
- 1 PAM
- PAM - Logging
- 6 Product Insight
- Project Tracking
- ProView
- ProView Internal
- 22 RDMS
- 1.9K Refinitiv Data Platform
- 695 Refinitiv Data Platform Libraries
- 4 LSEG Due Diligence
- LSEG Due Diligence Portal API
- 4 Refinitiv Due Dilligence Centre
- Rose's Space
- 1.2K Screening
- 18 Qual-ID API
- 13 Screening Deployed
- 23 Screening Online
- 12 World-Check Customer Risk Screener
- 1K World-Check One
- 46 World-Check One Zero Footprint
- 45 Side by Side Integration API
- 2 Test Space
- 3 Thomson One Smart
- 10 TR Knowledge Graph
- 151 Transactions
- 143 REDI API
- 1.8K TREP APIs
- 4 CAT
- 27 DACS Station
- 121 Open DACS
- 1.1K RFA
- 105 UPA
- 194 TREP Infrastructure
- 229 TRKD
- 918 TRTH
- 5 Velocity Analytics
- 9 Wealth Management Web Services
- 92 Workspace SDK
- 11 Element Framework
- 5 Grid
- 18 World-Check Data File
- 1 Yield Book Analytics
- 48 中文论坛