Missing completed extraction returned via API

benedict.chiu

Hi,

I have a few scheduled extracts that I want to pull out via the C# .NET SDK. I can retrieve the completed extractions if they are dated after 2018-12-07. (Note: It's currently 2019-01-07 here)

No results is returned if I look for anything before the aforementioned date, but works fine if I use any dates after. (For the dates that have a trigger anyway)

Via the Web UI, I can see and download the extracts prior to 2018-12-07. (I used an arbitrary 2018-12-04).

The earliest extract for this schedule should have been 2018-10-10.

There are no paging available, and I didn't play with the maxPage property (defaults to 250 apparently?)... and I'm only getting 21 results.

Is there some settings that I'm missing...?

Thanks

Regards,

EDIT: I've added images for clarity:

1. From the web UI, there are 26 items available (between the date range for 2018-12-01 to 2019-01-08).

Also, there are files older than 2018-12-08...

Please note, the reports are only running between Mon-Fri (inclusive)

2. From the code (I'm using GetCompletedByScheduleId for simplicity sake)... there are only 20 records, and no paging is available.

Furthermore, the oldest record is dated 2018-12-10

veerapath.rungruengrayubkul

Hi @benedict.chiu,

I have done some tests and found similar result. According to API Reference Tree, the ReportExtractions and ExtractedFiles exist in the DataScope Select's databases for a limited time. Currently, that time window is 45 days. However, it seems that some ReportExtractions within this period are not listed in results of GetCompletedByScheduleId.

For workaround, you can use ExtractFiles/GetAll to get the same result as Web UI. The GetAll retrieves all types of ExtractedFiles (i.e. result and Note files), so you need to filter only the FileType = ExtractedFileType.Full. Also, the result also includes ExtractedFile of on demand extraction which is not listed on Web UI. You may need to filter the file started with "_OnD_" prefix out.

Below is the sample code.

var firstPage = ExtractionsContext.ExtractedFileOperations.GetAll();
//Output
System.Console.WriteLine("FIRST PAGE:");
foreach (var extractedFile in firstPage)
{
    if (extractedFile.FileType == ExtractedFileType.Full)
    System.Console.WriteLine("Filename: {0}, Size: {1}, ReceivedDate: {2}",
    extractedFile.ExtractedFileName, extractedFile.Size, extractedFile.ReceivedDateUtc);
}

veerapath.rungruengrayubkul

Hi @benedict.chiu

Do you mean that you can get extracted result but it doesn't have all extracts? If so, this could be the incomplete output issue stated in this advisory.

Most Tick History reports deliver output as a gzip file. If a report is large, it delivers its output as
several smaller gzip files concatenated into a single large gzip file. The issue is that some HTTP client does not support concatenated gzip files.

To avoid this issue, you need to disable AutomaticDecompression, download gzip file, and then decompress the gzip file using function which supports concatenated gzip file. Sample C# codes are provided in page 4-5 of the advisory. For further information, please refer to the .Net SDK Tutorial 5 which illustrates various ways of downloading and saving extraction data from TRTH.

benedict.chiu

https://community.developers.refinitiv.com/discussion/comment/36865#Comment_36865

Hi,

I've updated the question with some screenshots for clarity, but I don't believe the issue is related to the advisory... thanks!

benedict.chiu

https://community.developers.refinitiv.com/discussion/comment/37046#Comment_37046

Thanks a lot for your assistance. Much appreciated!