TRTH v2 using Unirest java http client vbd gzip download issue
We’re having issues while trying to programmatically run DSS StandardExtractions UserPackageDelivery using Unirest java http client.
When trying to stream the file contents using ‘StandardExtractions/UserPackageDeliveries({id})/$value’ request we do not get the file in compressed ‘gzip’ format as advised by the API documentation.
While sending the http GET request we’re sending the header ‘Accept-Encoding: gzip’ as advised. We can also see the response header correctly puts ‘Content-Encoding=gzip’, the response body however appears to be plain text format. How can we receive the streamed file contents in gzip format for StandardExtractions?
I think that you are correct. From my test, the response always contain "Content-Encoding: gzip" no matter the "Accept-Encoding: gzip" is present, or not. It conflicts with the statement mentioned in the documents. I will contact the development team to verify it.
For now, to get the raw gz file, you can use this kind of Java code.
HttpClient client = HttpClients.custom().disableContentCompression().build();
HttpGet httpget = new HttpGet("https://hosted.datascopeapi.reuters.com/RestApi/v1/StandardExtractions/UserPackageDeliveries('0x05d0f50a992b2f96')/$value");
httpget.addHeader("Authorization", "Token <token>");
CloseableHttpResponse response;
try {
response = (CloseableHttpResponse) client.execute(httpget);
InputStream is = response.getEntity().getContent();
FileOutputStream fos = new FileOutputStream(new File("c:\\output.cvs.gz"));
int inByte;
while((inByte = is.read()) != -1)
fos.write(inByte);
is.close();
fos.close();
} catch (ClientProtocolException e) {
// TODO Auto-generated catch block
e.printStackTrace();
} catch (IOException e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
Refer to Unirest code HttpReponse.java at line 84, it decompresses the data if the data is gzipped.
To get the gzip file, you need to use the solution mentioned in this question.
I have used client other than Unirest with closeable http client like this:
CloseableHttpClient client = HttpClients.custom().disableContentCompression().build(); Unirest.setHttpClient(client);
Also, I have tried using java library directly too with same result:
HttpURLConnection conn= (HttpURLConnection)myURL.openConnection();
conn.setRequestProperty("Authorization", "Token " + getAuthToken());
conn.setRequestProperty("Accept-Encoding", "gzip");
conn.setRequestMethod("GET");
conn.getInputStream();
I do see the Unirest code where it checks the content-encoding:gzip, however I am not expecting 'content-encoding:gzip' to be set on the response as per : https://hosted.datascopeapi.reuters.com/RestApi.Help/Home/KeyMechanisms?ctx=Extractions&tab=0&uid=Streaming its mentioned that if request header does not have 'Accept-Encoding:gzip' set then the response header wouldn't have 'content-encoding:gzip' set for response header.
@Arvind Kaushik, thanks to some out of band information received separately by email I shall answer this one.
If I was well informed, you retrieved the list of user package deliveries with a GET to this URL:
https://hosted.datascopeapi.reuters.com/RestApi/v1/StandardExtractions/UserPackageDeliveryGetUserPackageDeliveriesByPackageId(PackageId='0x0460dc1d24a62cb1')
{
"@odata.context": "https://hosted.datascopeapi.reuters.com/RestApi/v1/$metadata#UserPackageDeliveries",
"value": [
{
"PackageDeliveryId": "MHgwNDYwZGMxZDI0YTYyY2IxfEZGRS1MZWdhY3l8SW5zaWRlcnNcVVNcTW9kZWx2M3xXZWVrbHl8SVRNdjNXZWVrbHkuMjAxNzAyMDI",
"UserPackageId": "0x0460dc1d24a62cb1",
"SubscriptionId": "0x0400dc1d24a00cb3",
"Name": "ITMv3Weekly.20170202",
"ReleaseDateTime": "2017-02-02T11:31:35.000Z",
"FileSizeBytes": 314925,
"Frequency": "Weekly"
},
etc.
The issue is with the UserPackageId 0x0460dc1d24a62cb1, which is not for TRTH VBD, it is for Insider data. That is why you get something different from what is shown in step 4 of TRTH REST API Tutorial 2 (get list of user package deliveries).
See the value of UserPackageId in this extract from the first step of TRTH REST API Tutorial 2 (list all user packages):
…
{
"UserPackageId": "0x04f21a8d20c59cb1",
"PackageId": "0x04f21a8d20c59cb1",
"PackageName": "KAR - Karachi Stock Exchange",
"SubscriptionId": "0x0400dc1d24a00cb4",
"SubscriptionName": "TRTH Venue by Day"
},
{
"UserPackageId": "0x0460dc1d24a62cb1",
"PackageId": "0x0460dc1d24a62cb1",
"PackageName": "US Insider Trading Model v3",
"SubscriptionId": "0x0400dc1d24a00cb3",
"SubscriptionName": "Insider"
},
…
Use a PackageId that is related to "SubscriptionName": "TRTH Venue by Day" and you will receive TRTH data as described in the tutorial, in gzip format.
I'm not sure what your source of information is, but I have tried using number of different requests with similar results. e.g
https://hosted.datascopeapi.reuters.com/RestApi/v1/StandardExtractions/UserPackageDeliveryGetUserPackageDeliveriesByDateRange(SubscriptionId='0x0400dc1d24a00cb4',FromDate=2017-07-30,ToDate=2017-07-31)
{
"@odata.context": "https://hosted.datascopeapi.reuters.com/RestApi/v1/$metadata#UserPackageDeliveries",
"value": [
{
"PackageDeliveryId": "0x05d0a0119a2b3016",
"UserPackageId": "0x04f21a8d1f759cb1",
"SubscriptionId": "0x0400dc1d24a00cb4",
"Name": "IOM-2017-07-30-NORMALIZEDMP-Report-4-of-6.csv.gz",
"ReleaseDateTime": "2017-07-31T03:00:00.000Z",
"FileSizeBytes": 841968,
"Frequency": "Daily",
"ContentMd5": ""
},
https://hosted.datascopeapi.reuters.com/RestApi/v1/StandardExtractions/UserPackageDeliveries('0x05d0a0119a2b3016')/$value
@Arvind Kaushik, I'm assuming you want to save the data, not decompress it on the fly. If that is the case, can you try this:
String urlGet = urlHost + "/StandardExtractions/UserPackageDeliveries('"+FileId+"')/$value";
try {
URL myURL = new URL(urlGet);
HttpURLConnection myURLConnection = (HttpURLConnection)myURL.openConnection();
myURLConnection.setRequestProperty("Authorization", "Token "+sessionToken);
myURLConnection.setRequestProperty("Accept-Encoding", "gzip");
myURLConnection.setRequestProperty("Accept-Charset", "UTF-8");
myURLConnection.setRequestMethod("GET");
try( DataInputStream readerIS = new DataInputStream( myURLConnection.getInputStream())) {
Files.copy (readerIS, Paths.get (filename));
}
I have not had the occasion to try this with VBD, but I think it should work.
You mean without specifying 'Accept-Encoding:gzip' so that the 'Content-Encoding:gzip' is not returned in the response ?
I shall give it a try.
@Arvind Kaushik, oops, sorry, I forgot that header, please include header 'Accept-Encoding:gzip'.
I'll edit my response above to add this line.
@Christiaan Meihsl
I did try this with Postman, however I still see the content-encoding:gzip being returned in the response header.
Request Headers: cache-control:"no-cache" Postman-Token:"ea9146cd-c631-4161-b248-06b476adcb68" Prefer:"respond-async" Authorization:"Basic OTAxMjYyMzppaHNtQHJraXQ=" User-Agent:"PostmanRuntime/6.1.6" Accept:"*/*" Host:"hosted.datascopeapi.reuters.com" cookie:"DSSAPI-COOKIE=R3148268809" accept-encoding:"gzip, deflate" Response Headers: set-cookie:"DSSAPI-COOKIE=R3148268809; path=/" cache-control:"no-cache" pragma:"no-cache" content-length:"841968" content-type:"text/plain" content-encoding:"gzip"
@Arvind Kaushik, my suggestion was actually to try the piece of Java code I posted.
Sorry, I think I have given Christiaan wrong information. Anyway, I did a quick test with Python using the URL your provided and was able to retrieve the file as gz file. Have you tested with the Java solution suggested by Jiraponse?
under https://hosted.datascopeapi.reuters.com/RestApi.Help/Home/KeyMechanisms?ctx=Extractions&tab=0&uid=Streaming its mentioned that if request header does not have 'Accept-Encoding:gzip' set then the response header wouldn't have 'content-encoding:gzip' set for response header.
Standard Extractions feeds content can deliver very large files that would increase the burdens on our servers and bandwidth were the files not compressed. In this situation we always deliver the file in a compressed format, although if the Accept-Encoding: gzip is not present, the Content-Encoding: gzip header will not be included in the response, but the file will always be compressed as gzip if that subscription delivers gzipped content.
I do not see this happening for my testing though. I always get content-encoding:gzip causing the stream to be unzipped, contrary to what I want.