How to download venue by detail files in .gz format for TRTH v2 API?

Hi,
I am able to use the UserPackageDeliveries/$value endpoint to get a response from the API from the .gz venue by day files but this response is in form of a csv object. This is huge in terms of size and hence the download time is also pretty high. Is there a way to directly download the .gz files for a subscription ID or package deliveryID directly?
I am using Powershell for this.
Thanks,
Ashish
Best Answer
-
Normally the data will be delivered in gzip encoding stream.
Are you using the Poweshell's Invoke-WebRequest orInvoke-RestMethod? As far as I know, the commands will return decompressed response data. It seems that the commands automatically decompress response. If you want to retreive the gzip data directly, you can use other command.
Below is the sample code. I use the System.Net.WebRequest to download data and then save the data to file.
$GetResultUrl = "https://hosted.datascopeapi.reuters.com/RestApi/v1/StandardExtractions/UserPackageDeliveries('" + $PackageDeliveryId + ''')/$value'
# Creating a new HttpWebRequest object.
[System.Net.HttpWebRequest]$oHttpWebRequest = [System.Net.WebRequest]::Create($GetResultUrl)
# This may be superflous if your HTTP server doesn't use compression.
$oHttpWebRequest.AutomaticDecompression = [System.Net.DecompressionMethods]::None
$oHttpWebRequest.Headers.Add("Authorization",'Token'+$token)
$oHttpWebRequest.Headers.Add("X-Direct-Download",'true')
$sr = $oHttpWebRequest.GetResponse().GetResponseStream()
$Wrt = [System.IO.File]::Create("D:\output.gz")
$Buffer = New-Object Byte[] 1024
Do {
$BytesRead = $sr.Read($Buffer, 0, $Buffer.Length)
$Wrt.Write($Buffer, 0, $BytesRead)
} While ($BytesRead -gt 0)
$sr.close()
$sr.Dispose()
$Wrt.Flush()
$Wrt.Close()
$Wrt.Dispose()Moreover, new DataScope Select 11.1 REST API now supports downloading Venue by Day files faster by retrieving them directly from the Amazon cloud in which they are hosted. Below is the information from the Thomson Reuters Tick History 11.1 REST API User Guide / Version 3.0.
0
Answers
-
Hey,
Thanks for the response, I shall try this asap. Also, I did try the X-Direct-Download:True but for some reason it showed up an error "Forbidden". Do you have any idea on this error? Do I need some sort of access on AWS in order to download files from there directly?
Thanks,
Ashish
0 -
@ashish.singh2 I
have found the forbidden error, once I use the Invoke-WebRequest with
the X-Direct-Download:true header. However, the header works fine with
my sample code. Could you try the code?0 -
My trial has expired for now, I will try your code as soon as I get it renewed. Thanks a lot for all the help
0 -
Hi,
I tried the above method. But I need to provide proxy as well in order to access this. I was providing proxy via -proxy when using invoke-restmethod. I tried adding proxy to this by $oHttpWebRequest.Proxy.Add but it doesn't seem to be working. Can you please help with this?
Thanks,
Ashish
0 -
Hi,
Please try the following code. The HttpWebRequest is a class in .Net framework. You can find more information in the following resources.
$webproxy = "<proxy address>"
$proxy = new-object System.Net.WebProxy
$proxy.Address = $webproxy
#$account = new-object System.Net.NetworkCredential($user,[Runtime.InteropServices.Marshal]::PtrToStringAuto([Runtime.InteropServices.Marshal]::SecureStringToBSTR($pwd)), "")
#$proxy.credentials = $account
$oHttpWebRequest.Proxy = $proxy;
$response = $oHttpWebRequest.GetResponse()0 -
Hi,
I tried adding proxy through the way you suggested. It throws the following error:
Exception setting "Proxy": "This operation cannot be performed after the
request has been submitted."Thanks,
Ashish
0 -
It seems like the "Proxy" property has been assigned after the GetResponse() call. Could you please verify the calling sequence?
It will be helpful, if you can provide your snippet code.
0 -
Hi,
Sorry about that, I forgot to create a new request and tried to process the same request again with proxy which resulted in that error. I am able to download the file in gz now. Thanks a lot for all the help.
Regards,
Ashish
0 -
Thank you for the update. Glad to know that you are able to download the file now.
Regards,
Veerapath
0
Categories
- All Categories
- 3 Polls
- 6 AHS
- 36 Alpha
- 166 App Studio
- 6 Block Chain
- 4 Bot Platform
- 18 Connected Risk APIs
- 47 Data Fusion
- 34 Data Model Discovery
- 685 Datastream
- 1.4K DSS
- 620 Eikon COM
- 5.2K Eikon Data APIs
- 10 Electronic Trading
- Generic FIX
- 7 Local Bank Node API
- 3 Trading API
- 2.9K Elektron
- 1.4K EMA
- 254 ETA
- 557 WebSocket API
- 38 FX Venues
- 14 FX Market Data
- 1 FX Post Trade
- 1 FX Trading - Matching
- 12 FX Trading – RFQ Maker
- 5 Intelligent Tagging
- 2 Legal One
- 23 Messenger Bot
- 3 Messenger Side by Side
- 9 ONESOURCE
- 7 Indirect Tax
- 60 Open Calais
- 276 Open PermID
- 44 Entity Search
- 2 Org ID
- 1 PAM
- PAM - Logging
- 6 Product Insight
- Project Tracking
- ProView
- ProView Internal
- 22 RDMS
- 1.9K Refinitiv Data Platform
- 662 Refinitiv Data Platform Libraries
- 4 LSEG Due Diligence
- LSEG Due Diligence Portal API
- 4 Refinitiv Due Dilligence Centre
- Rose's Space
- 1.2K Screening
- 18 Qual-ID API
- 13 Screening Deployed
- 23 Screening Online
- 12 World-Check Customer Risk Screener
- 1K World-Check One
- 46 World-Check One Zero Footprint
- 45 Side by Side Integration API
- 2 Test Space
- 3 Thomson One Smart
- 10 TR Knowledge Graph
- 151 Transactions
- 143 REDI API
- 1.8K TREP APIs
- 4 CAT
- 27 DACS Station
- 121 Open DACS
- 1.1K RFA
- 104 UPA
- 193 TREP Infrastructure
- 229 TRKD
- 917 TRTH
- 5 Velocity Analytics
- 9 Wealth Management Web Services
- 90 Workspace SDK
- 11 Element Framework
- 5 Grid
- 18 World-Check Data File
- 1 Yield Book Analytics
- 46 中文论坛