Can we able to download huge size TRTH VBD files in small chunks to avoid network contention?
Some of the TRTH VBD files (Raw files) are more than 5 GB in size. The downloading of these files causes network bandwidth contention.
1. Is it possble to download the file in smal chunks (500 MB each) by curl command?
or
2. TRTH can able to split the huge size files in to max 500 MB each
Best Answer
-
Refer to Key Mechanisms regarding Streaming, you can use range request (HTTP header Range) to get small chunks of data.
For example, if we would like this file:
{
"PackageDeliveryId": "0x05d0f50a992b2f96",
"UserPackageId": "0x04f21a8d26059cb1",
"SubscriptionId": "0x0400dc1d24a00cb4",
"Name": "NAS-2017-07-31-NORMALIZEDMP-Report-1-of-1.csv.gz",
"ReleaseDateTime": "2017-08-01T04:00:00.000Z",
"FileSizeBytes": 776125,
"Frequency": "Daily",
"ContentMd5": ""Its size is 776KB. We need to 200KB for each chunk. Therefore, we need to send 4 range requests. The following is the 4 curl commands.
curl -k -X GET -H "Range: bytes=0-200000" -H "Authorization: Token <token>" -o part1 https://hosted.datascopeapi.reuters.com/RestApi/v1/StandardExtractions/UserPackageDeliveries('0x05d0f50a992b2f96')/$value
curl -k -X GET -H "Range: bytes=200001-400000" -H "Authorization: Token <token>" -o part2 https://hosted.datascopeapi.reuters.com/RestApi/v1/StandardExtractions/UserPackageDeliveries('0x05d0f50a992b2f96')/$value
curl -k -X GET -H "Range: bytes=400001-600000" -H "Authorization: Token <token>" -o part3 https://hosted.datascopeapi.reuters.com/RestApi/v1/StandardExtractions/UserPackageDeliveries('0x05d0f50a992b2f96')/$value
curl -k -X GET -H "Range: bytes=600001-" -H "Authorization: Token <token>" -o part4 https://hosted.datascopeapi.reuters.com/RestApi/v1/StandardExtractions/UserPackageDeliveries('0x05d0f50a992b2f96')/$valueAfter that, we need to merge all four parts to a single file.
$cat part1 part2 part3 part4 > NAS-2017-07-31-NORMALIZEDMP-Report-1-of-1.csv.gz
0
Answers
-
Does the final merged file pass the MD5 Check?
0 -
Hi @Ayan
Yes, it should.
HTTP range requests does not alter the content of the file so the merged file should be exactly the same as original file.
0 -
We are facing issue while trying to gunzip the files, getting error stating "Not a gz file". But if we merger them using cat command it in a single file we are able to unzip that single file. Is there anyway to download files in chunk and the chunked files can be unzipped?
0 -
Hi @Ayan
I believe you misunderstand how range requests work.
Range request or byte serving does not repackage the content into a smaller chunk. What it does is it send a specific part of a file as requested. It does not alter or recompress the file, which means if the initial file is compressed, you have to combine every part of the file first before you can decompress it.As for the question;
No, currently it is not possible to unzipped part of a VBD file.
0
Categories
- All Categories
- 3 Polls
- 6 AHS
- 36 Alpha
- 166 App Studio
- 6 Block Chain
- 4 Bot Platform
- 18 Connected Risk APIs
- 47 Data Fusion
- 34 Data Model Discovery
- 687 Datastream
- 1.4K DSS
- 622 Eikon COM
- 5.2K Eikon Data APIs
- 10 Electronic Trading
- Generic FIX
- 7 Local Bank Node API
- 3 Trading API
- 2.9K Elektron
- 1.4K EMA
- 254 ETA
- 557 WebSocket API
- 38 FX Venues
- 14 FX Market Data
- 1 FX Post Trade
- 1 FX Trading - Matching
- 12 FX Trading – RFQ Maker
- 5 Intelligent Tagging
- 2 Legal One
- 23 Messenger Bot
- 3 Messenger Side by Side
- 9 ONESOURCE
- 7 Indirect Tax
- 60 Open Calais
- 276 Open PermID
- 44 Entity Search
- 2 Org ID
- 1 PAM
- PAM - Logging
- 6 Product Insight
- Project Tracking
- ProView
- ProView Internal
- 22 RDMS
- 1.9K Refinitiv Data Platform
- 677 Refinitiv Data Platform Libraries
- 4 LSEG Due Diligence
- LSEG Due Diligence Portal API
- 4 Refinitiv Due Dilligence Centre
- Rose's Space
- 1.2K Screening
- 18 Qual-ID API
- 13 Screening Deployed
- 23 Screening Online
- 12 World-Check Customer Risk Screener
- 1K World-Check One
- 46 World-Check One Zero Footprint
- 45 Side by Side Integration API
- 2 Test Space
- 3 Thomson One Smart
- 10 TR Knowledge Graph
- 151 Transactions
- 143 REDI API
- 1.8K TREP APIs
- 4 CAT
- 27 DACS Station
- 121 Open DACS
- 1.1K RFA
- 104 UPA
- 194 TREP Infrastructure
- 229 TRKD
- 918 TRTH
- 5 Velocity Analytics
- 9 Wealth Management Web Services
- 91 Workspace SDK
- 11 Element Framework
- 5 Grid
- 18 World-Check Data File
- 1 Yield Book Analytics
- 48 中文论坛