Are there any TRTH On Demand limitations on results returned? Only seeing 22K lines returned for ...

@Troy.Pfiffner,

Issue analysis

I made a very similar request in Postman. Request body:

{
  "ExtractionRequest": {
    "@odata.type": "#ThomsonReuters.Dss.Api.Extractions.ExtractionRequests.TickHistoryTimeAndSalesExtractionRequest",
    "ContentFieldNames": [ "Trade - Price", "Trade - Volume" ],
    "IdentifierList": {
      "@odata.type": "#ThomsonReuters.Dss.Api.Extractions.ExtractionRequests.InstrumentIdentifierList", 
      "InstrumentIdentifiers": [{ "Identifier": "UNG", "IdentifierType": "Ric" }]
    },
    "Condition": {
      "MessageTimeStampIn": "LocalExchangeTime",
      "ApplyCorrectionsAndCancellations": false,
      "ReportDateRangeType": "Range",
      "QueryStartDate": "2018-04-27T11:00:00.000Z",
      "QueryEndDate": "2018-05-24T23:00:00.000Z",
      "DisplaySourceRIC": true
    }
  }
}

After polling the location URL to get the JobId, the delivered data contained slightly more than 100k records, covering the entire query range from 27 April till 24 May.

When I ran your code I got 24332 records, the first and last were:

UNG,Market Price,2018-04-27T06:05:20.697783330-04,Trade,22.84,1200
UNG,Market Price,2018-05-03T10:35:00.733435679-04,Trade,22.08,500

The results stop before the end of the requested range. This is typical when decompressing TRTH downloaded data on the fly with standard libraries. See this advisory for details and explanations.

Solutions

The recommended way to proceed is to save the compressed data from the stream directly into a gzip file, and then read and decompress the data from the file for further treatment.

Your code decompress data on the fly when receiving it, without the intermediary step of saving it to disk. This workflow is not recommended. It fails silently when the data set is above a certain size, due to the use of core libraries which do not support merged files. It is possible (but not recommended) to do on the fly decompression using libraries that handle it reliably, like SharpZipLib.

As stated, on the fly data treatment is not recommended for large data sets, but here is how you could do it. Add the library:

using ICSharpCode.SharpZipLib.GZip;

Change the code to retrieve the data:

if (extractionResult.Status == ThomsonReuters.Dss.Api.Jobs.JobStatus.Completed)
{
    var streamResponse = extractionsContext.GetReadStream(extractionResult.Result);
    {
        using (var gzip = new GZipInputStream(streamResponse.Stream))
        {
            using (var reader = new StreamReader(gzip, Encoding.UTF8))
            {
                var result = reader.ReadLine();
                if (string.IsNullOrEmpty(result))
                    System.Diagnostics.Debug.WriteLine("No raw results returned");
                else
                {
                    System.Diagnostics.Debug.WriteLine(result);
                    int lineCount = 0;
                    for (; !reader.EndOfStream; lineCount++)
                    {
                        result = reader.ReadLine();
                        if (lineCount == 0)
                            System.Diagnostics.Debug.WriteLine(result);
                    }
                    System.Diagnostics.Debug.WriteLine(result);
                    System.Diagnostics.Debug.WriteLine("Records: " + lineCount);
                }
            }
        }
    }
    //Output Notes
    System.Diagnostics.Debug.WriteLine("NOTES:");
    foreach (var note in extractionResult.Result.Notes)
        System.Diagnostics.Debug.WriteLine(note);
}

When I ran this code I got 101772 records, the first and last record were:

UNG,Market Price,2018-04-27T07:00:00.020488185-04,Trade,22.8,1100
UNG,Market Price,2018-05-23T18:30:00.020374970-04,Trade,23.92,0

You will find a whole set of relevant C# code in the .Net SDK Tutorial 5, covering various use cases (the tutorial code is available for download). That tutorial uses Gzip decompression, and SharpZipLib.

On the same topic you might also want to look at this thread.

Side comment

The extraction notes reveal that an embargo was applied, which removed some of the data:

Some data suppressed for release cycle(s) C3, C4, PE 64. Request occurred during embargo. Data currently available through 05/21/2018 20:00:00. See Release Cycle schedule in the user documentation for details.

I get the same type of warning, but it only impacts the last day of the request (today).