question

Upvotes
Accepted
dirk avatar image
5 ●1 ●1 ●1

Is it possible to split a query into one file per day or month?

I would like to extract times and sales (TAS) data for the whole history of a RIC, say HOZ5. To ultimately work with it, I need small files, say one file per day. There are two options to get that

1) make a new request for each day

2) make a single request and later split the gigantic outputfile by other means

The advantage of a single request is that goes much quicker (about 5 minutes no matter whether it is for a day or a whole history). But it is tedious to later split the file (they become too gigantic). So is there (in the context of the C# RestAPI) a third way, in which the output are already separated by, say day?

tick-history-rest-api
icon clock
10 |1500

Up to 2 attachments (including images) can be used with a maximum of 512.0 KiB each and 1.0 MiB total.

2 Answers

· Write an Answer
Upvotes
Accepted
Christiaan Meihsl avatar image
13.7k ●26 ●8 ●12

@dirk,

The API does not deliver a capability to make a single request that results in a set of smaller files.

If you need years of data, we do not recommend making hundreds of requests for a single day each. The overhead takes time, making it much longer for you to get all the data, and it also places additional load on our servers. Se our best practices.

One could be tempted to split the data during the retrieval time, by decompressing it on the fly and saving it to separate files (say 1 per year). But decompressing the data stream on the fly is not recommended either, see this advisory.

If it is too tedious to manage the whole set of data in one file, my recommendation would be to make a set of medium sized queries (maybe one query for one year of data), to have file sizes that are more easily manageable, without lengthening the extraction time too much. If you need a file for each trading day, you will have to process the medium sized files to make the smaller ones.

In the end it is a trade-off between extraction speed and file size.

icon clock
10 |1500

Up to 2 attachments (including images) can be used with a maximum of 512.0 KiB each and 1.0 MiB total.

Upvotes
dirk avatar image
5 ●1 ●1 ●1

Thank you

icon clock
10 |1500

Up to 2 attachments (including images) can be used with a maximum of 512.0 KiB each and 1.0 MiB total.

Write an Answer

Hint: Notify or tag a user in this post by typing @username.

Up to 2 attachments (including images) can be used with a maximum of 512.0 KiB each and 1.0 MiB total.