Is it possible to split a query into one file per day or month?

dirk
dirk Newcomer
in TRTH

I would like to extract times and sales (TAS) data for the whole history of a RIC, say HOZ5. To ultimately work with it, I need small files, say one file per day. There are two options to get that

1) make a new request for each day

2) make a single request and later split the gigantic outputfile by other means

The advantage of a single request is that goes much quicker (about 5 minutes no matter whether it is for a day or a whole history). But it is tedious to later split the file (they become too gigantic). So is there (in the context of the C# RestAPI) a third way, in which the output are already separated by, say day?

Tagged:

Best Answer

  • Christiaan Meihsl
    Christiaan Meihsl LSEG
    Answer ✓

    @dirk,

    The API does not deliver a capability to make a single request that results in a set of smaller files.

    If you need years of data, we do not recommend making hundreds of requests for a single day each. The overhead takes time, making it much longer for you to get all the data, and it also places additional load on our servers. Se our best practices.

    One could be tempted to split the data during the retrieval time, by decompressing it on the fly and saving it to separate files (say 1 per year). But decompressing the data stream on the fly is not recommended either, see this advisory.

    If it is too tedious to manage the whole set of data in one file, my recommendation would be to make a set of medium sized queries (maybe one query for one year of data), to have file sizes that are more easily manageable, without lengthening the extraction time too much. If you need a file for each trading day, you will have to process the medium sized files to make the smaller ones.

    In the end it is a trade-off between extraction speed and file size.

Answers

Categories