file size information

Options
s61670
s61670 Explorer
in TRTH

help

How do request file size on a Refinitiv server via REST API?

At the forum gave advice, to look through the DataScope. What if I download a historical data on individual days and such files thousands? For example, I want to download RIC for 9 years. But download not 1 file, download for 1 day (StartDate_EndDat=1 day)

Creating thousands of reports per DataScope for thousands of days to view the size of each file is not effective.

How do I request file size information through the REST API?

I want to automate the entire process in order to know for sure that when downloading information is not lost.

For example, I downloaded RIC, it weighs 23 Gb, but I know that other RIC by this symbol, the average file size is 50 Gb.

Every time look at DataScope? Not an effective waste of time ((Must be some way through REST API

image

download speed is not stable.

I have 10 Gb\s Internet.

The actual speed download can be 90 mb\s and then up to 0.01 mb\s

image


The forum wrote that historical data is stored on AWS

If the Internet speed is high, then you can download very quickly.

Practice shows that the information is not true.

During working days, download speed decreases to 0.01 mb\s - 2 mb\s for many hours, even if 10 gb\s Internet.

It is not possible to download historical data efficiently and quickly.

Tagged:

Best Answer

  • zoya faberov
    zoya faberov ✭✭✭✭✭
    Answer ✓

    Hello @s61670,

    So in my understanding you are looking to download TickHistoryRaw content, for instrument ESH1, of Domain MarketPrice and RawLegacyMarketDepth. This data is every tick that market has issued, so it is indeed large.

    Let me suggest, and you can tune/improve the strategy.

    If you request one day, 24 hours worth of TickHistoryRaw content for MarketPrice, for 3 consecutive business days (for example, for 2021.03.29, for 2021, 03.30 and for 2021.03.31), and once you have the 3 results back, average the size of them, you should have a rough idea on the expected size of this content for 1 day, same goes for RawLegacyMarketPrice.

    Next, you can look to split your required request into increments, define how many days per request makes the most sense.

    You can further decide if you like to design concurrency into your download strategy, let's say two threads, running concurrently, see Tick History REST API User Guide -> Concurrent Requests to learn more.

    If you would like to learn more of the embargo rules for this instrument, which looks to be important for your design, one option is to contact RTH content experts via Refinitiv Helpdesk Online -> Content -> RTH with your specific questions, to get the addressed.

    Request processing time and download speed depend on many factors, and you should aim to design your solution as asynchronous, issue a request and keep checking the status till the response is ready, without reliance on the result being available at a specific time.

    Hope this information is helpful to you.

Answers

  • zoya faberov
    zoya faberov ✭✭✭✭✭

    Hello @s61670,

    I believe that in order to create a well-designed, fully automated solution, you would have to have a rough idea how large is each of the data sets that you require, at the time of design of the solution, and also to check for and handle any errors during run, that can prevent the download from completing.

    The speed of the download, as you observe, you are absolutely correct, can differ. And as an example for a proactive design strategy, for exceptionally large requirements, it sometimes may make sense to run those well outside of the relevant market's opening and closing hours, to proactively decrease the chance of heavy load on the service coinciding with your job requests.

    ---

    Let me step back at this point, and try to find out more about your overall Tick History data requirements, in order to try to help?

    • What content set(s) are your looking to download?
    • Does your requirement include a a substantial amount of historical data, to be pulled only once, and then to be updated with the significantly smaller daily changes/deltas?
    • Please include any other info, detail or limitation, that are part of the requirements that you work to meet, that can be relevant to the design and the solution?
    • I am going to also alert your Refinitiv account team, so that they can be of most help as well, to help with working out the best solution for your requirements.
  • s61670
    s61670 Explorer
    https://community.developers.refinitiv.com/discussion/comment/76345#Comment_76345

    We are interested in chain RICs: ES and URO (old RIC ESCC), Market Price and LegacyMarketDepth

    Does your requirement include a a substantial amount of historical data, to be pulled only once, and then to be updated with the significantly smaller daily changes/deltas?

    Yeah, you're right.



  • s61670
    s61670 Explorer
    https://community.developers.refinitiv.com/discussion/comment/76345#Comment_76345

    For example, ESH1, we plan to download every single day (Market_Price + RawLegacyMarketDepth).

    The option to download ESH1 one file (the first day _ day_Expiry) does not interest us.

    1. Now the situation is such that the server has an embargo time when we cannot send requests for downloads.

    2. We also don't understand how we can control file size information.

    3. Download speed is unstable.

  • s61670
    s61670 Explorer
    https://community.developers.refinitiv.com/discussion/comment/76345#Comment_76345

    single day = 1file

    ESH1 = a few hundred days, we want a few hundred files.

Categories