Hello @Sven Thomas ,

TickHistory result should be gzipped, and it is not adaptable, as Tick History does not allow to customize compression format, please see DSS Key Mechanisms -> Streaming-> Compression for more information.

Please see this previous discussion thread and the answer from Veerapath at the bottom, hoep this will be of help.

If not, please include the details, on how to reproduce multipart gzip that causes the issue for you, so we can try to reproduce the same on our side.

One of the factors that may be relevant is that oversized tick history requirements that are implemented with custom request approach and not VBD, often have to be partitioned into sub-requests, so we would collect all of the zipped sub-results that are to be accumulated for the use case, first, and consequently decompress locally, all of the pieces, outside of DSS NET SDK. NET decompression is usually used as on-the-fly automatic decompression.

Hi @zoya faberov ...with 'share the request' you mean the source code ?

var extractionRequest = new TickHistoryTimeAndSalesExtractionRequest()
{
    IdentifierList = new InstrumentIdentifierList
    {
        InstrumentIdentifiers = new DataScope.Select.Api.Content.InstrumentIdentifier[]
        {
            new DataScope.Select.Api.Content.InstrumentIdentifier { Identifier = "ABI.BR", IdentifierType = IdentifierType.Ric },
            new DataScope.Select.Api.Content.InstrumentIdentifier { Identifier = "AFPD.S", IdentifierType = IdentifierType.Ric },
            new DataScope.Select.Api.Content.InstrumentIdentifier { Identifier = "AMPF.MI", IdentifierType = IdentifierType.Ric },
            new DataScope.Select.Api.Content.InstrumentIdentifier { Identifier = "ATCOa.ST", IdentifierType = IdentifierType.Ric },
            new DataScope.Select.Api.Content.InstrumentIdentifier { Identifier = "BAR.BR", IdentifierType = IdentifierType.Ric },
            new DataScope.Select.Api.Content.InstrumentIdentifier { Identifier = "CHRH.CO", IdentifierType = IdentifierType.Ric },
            new DataScope.Select.Api.Content.InstrumentIdentifier { Identifier = "FFIH0", IdentifierType = IdentifierType.Ric },
            new DataScope.Select.Api.Content.InstrumentIdentifier { Identifier = "FGBMH0", IdentifierType = IdentifierType.Ric },
            new DataScope.Select.Api.Content.InstrumentIdentifier { Identifier = "FGBSH0", IdentifierType = IdentifierType.Ric },
            new DataScope.Select.Api.Content.InstrumentIdentifier { Identifier = "IMCD.AS", IdentifierType = IdentifierType.Ric },
            new DataScope.Select.Api.Content.InstrumentIdentifier { Identifier = "LVMH.PA", IdentifierType = IdentifierType.Ric },
            new DataScope.Select.Api.Content.InstrumentIdentifier { Identifier = "OREP.PA", IdentifierType = IdentifierType.Ric },
            new DataScope.Select.Api.Content.InstrumentIdentifier { Identifier = "PHG.AS", IdentifierType = IdentifierType.Ric },
            new DataScope.Select.Api.Content.InstrumentIdentifier { Identifier = "STXEH0", IdentifierType = IdentifierType.Ric },
            new DataScope.Select.Api.Content.InstrumentIdentifier { Identifier = "UNA.AS", IdentifierType = IdentifierType.Ric }
        },
        ValidationOptions = new InstrumentValidationOptions { AllowHistoricalInstruments = true },
        UseUserPreferencesForValidationOptions = false
    },
    Condition = new TickHistoryTimeAndSalesCondition()
    {
        ReportDateRangeType = ReportDateRangeType.Range,
        ExtractBy = TickHistoryExtractByMode.Ric,
        MessageTimeStampIn = TickHistoryTimeOptions.GmtUtc,
        SortBy = TickHistorySort.SingleByRic,
        TimeRangeMode = TickHistoryTimeRangeMode.Inclusive,
        QueryStartDate = new DateTimeOffset(new DateTime(2019,12,20,9,2,34, DateTimeKind.Utc)),
                    QueryEndDate = new DateTimeOffset(new DateTime(2019,12,20,9,11,27, DateTimeKind.Utc)),
        DaysAgo = null
    },
    ContentFieldNames = new[]
    {
        RefinitivContentFieldNames.TimeAndSales.BidPrice,
        RefinitivContentFieldNames.TimeAndSales.AskPrice
    }
};


extractionsContext.Options.AutomaticDecompression = true;
            
RawExtractionResult extractionResult = await extractionsContext.ExtractRawAsync(extractionsContext, extractionRequest, cancellation);

extractionsContext is of type IExtractionsContext on which I subsequently call GetReadStremAsync and of course created with our RTH account credentials, well you know what I mean.

Is that enough info for you to run the verification ?

Many thanks already !

Hi @zoya faberov , digged a bit further into this ...

Turned off AutomaticDecompression and persisted the returned response stream to a file (gzip).

Magically , the unzipped file contains all expected lines ... the sum of all manifest Count values.

So now my best guess it has something to do with AutomaticDecompression. Came across the following thread on the forum ... Missing completed extraction returned via API - Forum | Refinitiv Developer Community

The thread mentions an advisory but the link seems outdated.

Once again ... any help/info ... mucho appreciado !!!

Hello @Sven Thomas ,

Thanks for sharing your request and for letting us know.

You are absolutely correct, disabling decompression is often helpful for longer extraction times and larger results.

The other feature that is often of help is Asyc, as described in detail in Key Mechanisms -> Async.

So having run the request on our side and having part missing, I did both, i.e.

  //  ExtractionsContext.Preferences.WaitSeconds = 5;
    //        ExtractionsContext.Options.AutomaticDecompression = true;

And

  //Start the job
            var job = extractionsContext.ExtractRawStart(extractionRequest);
            Status.Notify(extractionsContext, null, "MonitorJob", MethodType.Operation, Publish.Primary);
            while (job != null && job.Status != JobStatus.Completed && job.Status != JobStatus.Error
                   && job.Status != JobStatus.Cancelled && job.Status != JobStatus.Purged)
            {
                //Retrieve an updated status of the job
                job = extractionsContext.MonitorJob(job);
            }

            //Output
            if (job != null && job.Result != null && job.Status == JobStatus.Completed)
            {
...

on my side, have verified the complete result for your request as well.

But thanks to your information, we also know that just not decompressing for this request sizing is sufficient, and start+monitor job is not strictly required.

The other note that you may find useful, is once we have the jobId, it may be helpful to quick-check job status with Postman to be completed=200 and quick-get-result with Postman on jobId as well. Compare the result= it was much larger for your request. This way, I knew that the job submitted with .NET SDK was completing successfully, no issue with request or permissions, and that the result as retrieved via with .NET SDK on the spot was not the complete result.

Hope that this information helps

Hi @zoya faberov

Sorry but I'm not quite following your suggestion to use Async mechanism ... I'm actually using ExtractRawAsync already. When this method returns an ExtractionResult I would expect it to be complete for both AutomaticDecompression true or false. Seems like a flaw in the DSS api to me and apparently reported before here I see now.

Anyway, I will work on the solution without autodecompressing and un-g-zipping myself.

Many thanks !

CC @aleksandra.kluczniak

Good afternoon @zoya faberov

With AutomaticDecompression set to false and decompressing on our end I now run into another issue. The TRTH api seems to return multipart gzip data which is not supported by standard .Net System.IO.Compression.GZipStream. As far as I know only 3rd party library SharpZipLib does support this. Do you perhaps have more experience with this or can you recommend other decompression possibility for multipart gzip ?

Are there any plans on Refinitiv side for other types of data compression ?