sometimes VDB files are not downloaded completely. is this due to network issue ?

we are trying to download VBD files in 10 threads , sometimes it is failing to download complete file. is this due to network issue . we are using below command to download the file.

if it is due to network issue , how to avoid the network failures ?

curl -H "Prefer:respond-async" -H "Content-Type:application/json" -H "Authorization: Token XXXXX" -H "Accept-Charset: UTF-8" --proxy XXXXXXX "https://hosted.datascopeapi.reuters.com/RestApi/v1/StandardExtractions/UserPackageDeliveries('0x067c39d073d0330a')/\$value" -o OTC-2019-01-15-NORMALIZEDMP-Data-1-of-1.csv.gz

Find more posts tagged with

tick-history-rest-api

Accepted answers

veerapath.rungruengrayubkul

@samara

According to Best Practices and Limits for Tick History, below is the limitation for file download.

You can have up to four extraction downloads in progress at one time. This total applies to any combination of custom reporting and Venue by Day downloads, via any combination of the REST API and via the GUI.

There is an exception to extraction file download limits: there is no limit on the number of files that you can download directly from Amazon Web Services (AWS) at one time. Downloading directly from Amazon is described in the Tick History REST API Users Guide and the Tick History User Guide.

As there is no limit on the number of files downloaded directly from AWS, please change your CURL script to download files from AWS instead.

Please add the “X-Direct-Download : true” as a header of the VBD file download request. This request will be
redirected to AWS via HTTP 302 redirect.

Below are sample commands.

-Get redirect URL, and then store it in an environment variable

redirect_url="$(curl -I -Ls -H 'Authorization: Token_xxxxxxxxxxxxxxxxxxxxx' -H 'X-Direct-Download:true' -X GET https://hosted.datascopeapi.reuters.com/RestApi/v1/StandardExtractions/UserPackageDeliveries('&lt;UserPackageDeliveriesID&gt;')/$value -o /dev/null -w %{url_effective})"

Directly download a VBD file from the redirect URL.

curl -k -X GET $redirect_url -o OTC-2019-01-15-NORMALIZEDMP-Data-1-of-1.csv.gz

All comments