Why not all instruments received if AutoDecompression is to true on ExtractionContext for TickHistoryTimeAndSaleExtractionRequest?

This problem has been mentioned in this document.

Most Tick History reports deliver output as a gzip file. If a report is large, it delivers its output as several smaller gzip files concatenated into a single large gzip file. If your HTTP client does not support concatenated gzip files, you may not receive some of your output. At this time, most HTTP clients do not fully support concatenated gzip files.

@ jirapongse.phuriphanvichai

We have been using HttpClient or HttpClientHandler in our code. Actually we wish to download the whole zip file and decompress it afterwards after the download is finished. We have been using System.IO.Compression.GzipStream in another PowerShell script to decompress the downloaded file from Reuters FTP (version 1 of the api).

As mentioned we are suing ExtractionsContext to save the gzip file:

using (var response = _extractionContext.RawExtractionResultOperations.GetReadStream(result)) { using (var fileStream = File.Create(fullFileName)) { response.Stream.CopyTo(fileStream); } }

as mentioned in Reuters sample code provided in the TickHistoryExamples.cs file.

I am not sure we have to use HttpClientHandler or HttpClient.

There is no sample for code using this.

And as mentioned what if we do wish to decompress the downloaded file using System.IO.Compression.GzipStream in another PowerShell script?

Thanks

extractionsContext.Options.AutomaticDecompression = false;

var response = extractionsContext.RawExtractionResultOperations.GetReadStream(result);
using (var gzip = new GZipInputStream(response.Stream))
 {
     using (var outputStream = File.Create("output.csv"))
     {
         gzip.CopyTo(outputStream);
     }
 }

@mojtaba.danai, have you tried the code of .Net Tutorial 5 ? It uses our .Net SDK, thus simplifying your coding work (there is no need to manage the HTTP connection yourself), and illustrates both saving the compressed data to file, and decompressing on the fly. It follows the recommendations outlined in the advisory. I believe it works for large data sets as well.

That said, we recommend you save the compressed data as a file on local hard disk, and then read it from there, instead of decompressing it on the fly.

It would be nice if we could use the standard .NET Ssystem.IO.Compression library to unzip the files, but unfortunately this doesn't work. Presumably this is why TR have included a third-party library in their tutorial code.

The root of the problem seems to be that the TR code that compresses the data file incorrectly records the size of the original file (this is stored in the last 4 byes of a gzip file). 7Zip, SharpZipLib and other utilities are robust enough to handle this error, but the Microsoft library isn't.

It would be great if TR could fix their compression code so that the gzip file is correctly formed and we wouldn't have to use third-party libraries.