TRTH slow download speed

@RTPKRSQT48, this month I did some download performance tests (in Python), without using the X-Direct-Download:
true header. On average, downloading a 323MB file took approximately 5 minutes, i.e. I'm observing ~1MB/s, which is faster than what you observed. I am behind a proxy, but have an excellent Internet connection and bandwidth. From your query I gather the large files you are downloading are between 100 and 300 MB (please correct me if I'm wrong).

Could you share your code, and tell us what type of data you are downloading (VBD or custom files) ?

That might help us to find a solution.

@RTPKRSQT48, my Python tests were also with T&S data, over the Internet (I work at Thomson Reuters but don't have direct access to TRTH servers) using my office connection. I did not test GUI performance.

Ok, you are using code from the C# example app. Let me test to see what performance I get from here in C#. I'll use the .Net C# Tutorial 5 code (available under the downloads tab), it saves in gzip.

When you say 80% is missing from large files: what is the size of those files (not truncated) ? On this topic, did you see this advisory ?

GUI: I don't believe you can activate X-direct-download.

@RTPKRSQT48, I made a test using the .Net C# Tutorial 5 code, changing the request to retrieve 1 year of 1 minute bars for more than 8200 instruments (input file: TRTH_API_1942_input_file.csv), without X-Direct-download.

Result was correctly saved as a gzip file of 3.8GB (containing an 88GB CSV with more than 1200 million lines).

Download performance (from Thomson Reuters, via Internet through our proxy): ~640KB/sec

The essentials of the download code:

extractionsContext.Options.AutomaticDecompression = false;
RawExtractionResult extractionResult = extractionsContext.ExtractRaw(extractionRequest);
DssStreamResponse streamResponse = extractionsContext.GetReadStream(extractionResult);
using (var fileStream = File.Create(dataOutputFile))
    streamResponse.Stream.CopyTo(fileStream);

For the entire code, please refer to the original sample (available under the downloads tab).

@RTPKRSQT48, I'm happy that using AWS by adding "x-direct-download" solves the issue for you.

But I am still intrigued at why, when not using AWS, I get better performance than you. If you have time and do not mind doing this, could you try running the latest .Net C# Tutorial 5 code (modifying it only to retrieve more data than what it does by default), and compare its download performance with that of your code (based on that of the C# example app) ? It might help us understand if the issue is related to the way it is coded.

From the ScheduleName, you need to use the following endpoint to the ScheduleId.

GET https://hosted.datascopeapi.reuters.com/RestApi/v1/Extractions/ScheduleGetByName(ScheduleName='example-eod')

HTTP/1.1 200 OK
{
    "@odata.context": "https://hosted.datascopeapi.reuters.com/RestApi/v1/$metadata#Schedules/$entity",
    "ScheduleId": "0x0580701ad3ab1f86",
    "Name": "example-eod",
    "OutputFileName": "",
    …
}

Then, after getting the SchduleId, you can use the
following endpoints to get the extractions.

• Extractions/Schedules({id})/CompletedExtractions
  returns the list of completed extractions for this schedule

• Extractions/Schedules({id})/LastExtraction
  returns information about the last completed extraction for this schedule

GET https://hosted.datascopeapi.reuters.com/RestApi/v1/Extractions/Schedules('0x0580702a9d0b1f86')/LastExtraction

HTTP/1.1 200 OK
{
    "@odata.context": "https://hosted.datascopeapi.reuters.com/RestApi/v1/$metadata#ReportExtractions/$entity",
    "ReportExtractionId": "6651428",
    "ScheduleId": "0x0580702a9d0b1f86",
    "Status": "Completed",
    "DetailedStatus": "Done",
    "ExtractionDateUtc": "2016-11-23T20:45:37.476Z",
    "ScheduleName": "example-eod",
    "IsTriggered": false,
    "ExtractionStartUtc": "2016-11-23T20:45:52.000Z",
    "ExtractionEndUtc": "2016-11-23T20:45:56.000Z"
}

After that, from ReportExtractionId,
you can get the list of files(Data file and Note file) belong to this
extraction report.

GET https://hosted.datascopeapi.reuters.com/RestApi/v1/Extractions/ReportExtractions('6651453')/Files

HTTP/1.1 200 OK
{
    "@odata.context": "https://hosted.datascopeapi.reuters.com/RestApi/v1/$metadata#ExtractedFiles",
    "value": [
        {
            "ExtractedFileId": "VjF8fDMzNjIzNTE",
            "ReportExtractionId": "6651453",
            "ScheduleId": "0x05807049631b1f86",
            "FileType": "Full",
            "ExtractedFileName": "9001552.ImmediateNonCustomized.20161123.144755.6651453.x04n05.csv",
            "LastWriteTimeUtc": "2016-11-23T20:47:57.567Z",
            "ContentsExists": true,
            "Size": 122,
           "ReceivedDateUtc": "2016-11-23T20:47:57.567Z"
        },
        {
            "ExtractedFileId": "VjF8fDMzNjIzNTA",
            "ReportExtractionId": "6651453",
            "ScheduleId": "0x05807049631b1f86",
            "FileType": "Note",
            "ExtractedFileName": "9001552.ImmediateNonCustomized.20161123.144755.6651453.x04n05.csv.notes.txt",
            "LastWriteTimeUtc": "2016-11-23T20:47:57.573Z",
            "ContentsExists": true,
            "Size": 1777,
            "ReceivedDateUtc": "2016-11-23T20:47:57.573Z"
        }
    ]
} 

From ExtractedFileId, you can download a file with this endpoint: Extractions/ExtractedFiles({id})/$value.

GET https://hosted.datascopeapi.reuters.com/RestApi/v1/Extractions/ExtractedFiles('VjF8fDMyMDk1NDc2OQ')/$value

There are other endpoints that can be useful, such as ReportExtractionGetCompletedByDateRangeByScheduleId
or ReportExtractionGetCompletedByScheduleId. For more information,
please refer to API Reference Tree .
Please focus on Extractions => Schedule , Extractions =>
Extraction => ReportExtraction, and Extractions =>
Extraction =>ExtractedFile endpoints.