question

Upvotes
Accepted
vaughan.rees avatar image
1 ●0 ●0 ●2

Trial Licence IntraDay Summary Limit

Hi there, we are currently using a trial licence (until we receive our full licence). I have used the C# template code to extract Intraday Summary data (all fields) at second level, creating one file per year of data. We have found an issue where each file only has data up to (approx.) 12th Jan (approx. 145MB in size) . Even when I extract data for two/five/ten years the file still remains virtually the same size, proving that no additional data is included. For comparison, when one year of data is downloaded from the portal it is 1.2GB in size. Please help as this is an urgent issue that we require a quick resolution for.

Thanks in advance.

Vaughan

apitick-history-rest-apihistorical
icon clock
10 |1500

Up to 2 attachments (including images) can be used with a maximum of 512.0 KiB each and 1.0 MiB total.

wasin.w avatar image   wasin.w

Hello @vaughan.rees

Thank you for your participation in the forum. Is the reply below satisfactory in resolving your query?

If so please can you click the 'Accept' text next to the appropriate reply. This will guide all community members who have a similar question.

Thanks,

AHS

wasin.w avatar image   wasin.w

Hello @vaughan.rees

Thank you for your participation in the forum. Is the reply below satisfactory in resolving your query?

If so please can you click the 'Accept' text next to the appropriate reply. This will guide all community members who have a similar question.

Thanks,

AHS

Christiaan Meihsl avatar image   Christiaan Meihsl

@vaughan.rees, please be informed that a reply has been verified as correct in answering the question, and has been marked as such. Thanks, AHS

2 Answers

· Write an Answer
Upvotes
Accepted
Christiaan Meihsl avatar image
13.7k ●26 ●8 ●12

@vaughan.rees, as Veerapath states, this looks like an issue with the retrieval of concatenated gzip files, which the TRTH servers generate for large data sets.

Depending on how you retrieve the data, how you code that retrieval, and which library you use, you can run into a situation where not all data is downloaded, without generating an error. The larger the data set, the higher the probability you will run into this issue. Decompressing data on the fly makes things worse, that is why we strongly recommend you first download and save the compressed data, and only open it up (from local hard disk) afterwards, for whatever treatment you want to apply to it.

Pages 4 and 5 of the advisory Veerapath mentioned give some information on how to avoid this issue when coding with C#. For further information, please refer to the .Net SDK Tutorial 5. Its downloadable C# code sample contains code that illustrates various ways of downloading and saving your extraction data from TRTH. It also shows how to download the data from AWS (the Amazon Web Service cloud), which is faster than the standard download. On the topic of AWS downloads, see this advisory.

In a nutshell, the recommended method is to download and save the compressed data, from AWS, without decompressing on the fly.

If this answer does not suffice, please post the code you use to retrieve and save the data, that could be useful for us to help you more.

icon clock
10 |1500

Up to 2 attachments (including images) can be used with a maximum of 512.0 KiB each and 1.0 MiB total.

vaughan.rees avatar image vaughan.rees

Thanks for the detail Christiaan, very much appreciated.

Upvotes
veerapath.rungruengrayubkul avatar image
11.3k ●25 ●9 ●14

Hi @vaughan.rees

It looks like the incomplete output issue, where some HTTP clients do not fully support concatenated gzip files. Please check this documentation ADVISORY: AVOID INCOMPLETE OUTPUT - DOWNLOAD THEN DECOMPRESS for more information.

icon clock
10 |1500

Up to 2 attachments (including images) can be used with a maximum of 512.0 KiB each and 1.0 MiB total.

vaughan.rees avatar image vaughan.rees

Many thanks Veerpath, this has helped me!

Write an Answer

Hint: Notify or tag a user in this post by typing @username.

Up to 2 attachments (including images) can be used with a maximum of 512.0 KiB each and 1.0 MiB total.