sometimes VDB files are not downloaded completely. is this due to network issue ?

samara
samara Explorer
in TRTH

we are trying to download VBD files in 10 threads , sometimes it is failing to download complete file. is this due to network issue . we are using below command to download the file.

if it is due to network issue , how to avoid the network failures ?

curl -H "Prefer:respond-async" -H "Content-Type:application/json" -H "Authorization: Token XXXXX" -H "Accept-Charset: UTF-8" --proxy XXXXXXX "https://hosted.datascopeapi.reuters.com/RestApi/v1/StandardExtractions/UserPackageDeliveries('0x067c39d073d0330a')/\$value" -o OTC-2019-01-15-NORMALIZEDMP-Data-1-of-1.csv.gz

Tagged:

Best Answer

  • veerapath.rungruengrayubkul
    veerapath.rungruengrayubkul LSEG
    Answer ✓

    @samara

    According to Best Practices and Limits for Tick History, below is the limitation for file download.

    You can have up to four extraction downloads in progress at one time. This total applies to any combination of custom reporting and Venue by Day downloads, via any combination of the REST API and via the GUI.

    There is an exception to extraction file download limits: there is no limit on the number of files that you can download directly from Amazon Web Services (AWS) at one time. Downloading directly from Amazon is described in the Tick History REST API Users Guide and the Tick History User Guide.

    As there is no limit on the number of files downloaded directly from AWS, please change your CURL script to download files from AWS instead.

    Please add the “X-Direct-Download : true” as a header of the VBD file download request. This request will be
    redirected to AWS via HTTP 302 redirect.

    Below are sample commands.

    • -Get redirect URL, and then store it in an environment variable
    redirect_url="$(curl -I -Ls -H 'Authorization: Token_xxxxxxxxxxxxxxxxxxxxx' -H 'X-Direct-Download:true' -X GET https://hosted.datascopeapi.reuters.com/RestApi/v1/StandardExtractions/UserPackageDeliveries('<UserPackageDeliveriesID>')/$value -o /dev/null -w %{url_effective})"
    • Directly download a VBD file from the redirect URL.
    curl -k -X GET $redirect_url -o OTC-2019-01-15-NORMALIZEDMP-Data-1-of-1.csv.gz

Categories