TRTH v2 using Unirest java http client vbd gzip download issue

I think that you are correct. From my test, the response always contain "Content-Encoding: gzip" no matter the "Accept-Encoding: gzip" is present, or not. It conflicts with the statement mentioned in the documents. I will contact the development team to verify it.

For now, to get the raw gz file, you can use this kind of Java code.

		HttpClient client = HttpClients.custom().disableContentCompression().build();

		HttpGet httpget = new HttpGet("https://hosted.datascopeapi.reuters.com/RestApi/v1/StandardExtractions/UserPackageDeliveries('0x05d0f50a992b2f96')/$value");
		httpget.addHeader("Authorization", "Token <token>");
		CloseableHttpResponse response;
		try {
			response = (CloseableHttpResponse) client.execute(httpget);
			InputStream is = response.getEntity().getContent();
			FileOutputStream fos = new FileOutputStream(new File("c:\\output.cvs.gz"));
			int inByte;
			while((inByte = is.read()) != -1)
			     fos.write(inByte);
			is.close();
			fos.close();
		} catch (ClientProtocolException e) {
			// TODO Auto-generated catch block
			e.printStackTrace();
		} catch (IOException e) {
			// TODO Auto-generated catch block
			e.printStackTrace();
		}

All comments

Jirapongse

Refer to Unirest code HttpReponse.java at line 84, it decompresses the data if the data is gzipped.

To get the gzip file, you need to use the solution mentioned in this question.

@Arvind Kaushik, thanks to some out of band information received separately by email I shall answer this one.

If I was well informed, you retrieved the list of user package deliveries with a GET to this URL:

https://hosted.datascopeapi.reuters.com/RestApi/v1/StandardExtractions/UserPackageDeliveryGetUserPackageDeliveriesByPackageId(PackageId='0x0460dc1d24a62cb1')

The result is:

{
    "@odata.context": "https://hosted.datascopeapi.reuters.com/RestApi/v1/$metadata#UserPackageDeliveries",
    "value": [
        {
            "PackageDeliveryId": "MHgwNDYwZGMxZDI0YTYyY2IxfEZGRS1MZWdhY3l8SW5zaWRlcnNcVVNcTW9kZWx2M3xXZWVrbHl8SVRNdjNXZWVrbHkuMjAxNzAyMDI",
            "UserPackageId": "0x0460dc1d24a62cb1",
            "SubscriptionId": "0x0400dc1d24a00cb3",
            "Name": "ITMv3Weekly.20170202",
            "ReleaseDateTime": "2017-02-02T11:31:35.000Z",
            "FileSizeBytes": 314925,
            "Frequency": "Weekly"
        },
etc.

The issue is with the UserPackageId 0x0460dc1d24a62cb1, which is not for TRTH VBD, it is for Insider data. That is why you get something different from what is shown in step 4 of TRTH REST API Tutorial 2 (get list of user package deliveries).

See the value of UserPackageId in this extract from the first step of TRTH REST API Tutorial 2 (list all user packages):

… 
{ 
 "UserPackageId": "0x04f21a8d20c59cb1", 
 "PackageId": "0x04f21a8d20c59cb1", 
 "PackageName": "KAR - Karachi Stock Exchange", 
 "SubscriptionId": "0x0400dc1d24a00cb4", 
 "SubscriptionName": "TRTH Venue by Day" 
}, 
{ 
 "UserPackageId": "0x0460dc1d24a62cb1", 
 "PackageId": "0x0460dc1d24a62cb1", 
 "PackageName": "US Insider Trading Model v3", 
 "SubscriptionId": "0x0400dc1d24a00cb3", 
 "SubscriptionName": "Insider" 
},
…

Use a PackageId that is related to "SubscriptionName": "TRTH Venue by Day" and you will receive TRTH data as described in the tutorial, in gzip format.

I have used client other than Unirest with closeable http client like this:

CloseableHttpClient client = HttpClients.custom().disableContentCompression().build();
Unirest.setHttpClient(client);

Also, I have tried using java library directly too with same result:

	            HttpURLConnection conn= (HttpURLConnection)myURL.openConnection();
	            conn.setRequestProperty("Authorization", "Token " + getAuthToken());
	            conn.setRequestProperty("Accept-Encoding", "gzip");
	            conn.setRequestMethod("GET");
	            conn.getInputStream();

@Christiaan Meihsl

I'm not sure what your source of information is, but I have tried using number of different requests with similar results. e.g

https://hosted.datascopeapi.reuters.com/RestApi/v1/StandardExtractions/UserPackageDeliveryGetUserPackageDeliveriesByDateRange(SubscriptionId='0x0400dc1d24a00cb4',FromDate=2017-07-30,ToDate=2017-07-31)

	{
    "@odata.context": "https://hosted.datascopeapi.reuters.com/RestApi/v1/$metadata#UserPackageDeliveries",
    "value": [
        {
            "PackageDeliveryId": "0x05d0a0119a2b3016",
            "UserPackageId": "0x04f21a8d1f759cb1",
            "SubscriptionId": "0x0400dc1d24a00cb4",
            "Name": "IOM-2017-07-30-NORMALIZEDMP-Report-4-of-6.csv.gz",
            "ReleaseDateTime": "2017-07-31T03:00:00.000Z",
            "FileSizeBytes": 841968,
            "Frequency": "Daily",
            "ContentMd5": ""
        },

https://hosted.datascopeapi.reuters.com/RestApi/v1/StandardExtractions/UserPackageDeliveries('0x05d0a0119a2b3016')/$value

@Arvind Kaushik, I'm assuming you want to save the data, not decompress it on the fly. If that is the case, can you try this:

String urlGet = urlHost + "/StandardExtractions/UserPackageDeliveries('"+FileId+"')/$value";
try {
    URL myURL = new URL(urlGet);
    HttpURLConnection myURLConnection = (HttpURLConnection)myURL.openConnection();
    myURLConnection.setRequestProperty("Authorization", "Token "+sessionToken);
    myURLConnection.setRequestProperty("Accept-Encoding", "gzip");
    myURLConnection.setRequestProperty("Accept-Charset", "UTF-8");
    myURLConnection.setRequestMethod("GET");

    try( DataInputStream readerIS = new DataInputStream( myURLConnection.getInputStream())) {
        Files.copy (readerIS, Paths.get (filename));
    }

I have not had the occasion to try this with VBD, but I think it should work.

@Christiaan Meihsl

You mean without specifying 'Accept-Encoding:gzip' so that the 'Content-Encoding:gzip' is not returned in the response ?

I shall give it a try.

@Christiaan Meihsl

I did try this with Postman, however I still see the content-encoding:gzip being returned in the response header.

Request Headers:
cache-control:"no-cache"
Postman-Token:"ea9146cd-c631-4161-b248-06b476adcb68"
Prefer:"respond-async"
Authorization:"Basic OTAxMjYyMzppaHNtQHJraXQ="
User-Agent:"PostmanRuntime/6.1.6"
Accept:"*/*"
Host:"hosted.datascopeapi.reuters.com"
cookie:"DSSAPI-COOKIE=R3148268809"
accept-encoding:"gzip, deflate"
Response Headers:
set-cookie:"DSSAPI-COOKIE=R3148268809; path=/"
cache-control:"no-cache"
pragma:"no-cache"
content-length:"841968"
content-type:"text/plain"
content-encoding:"gzip"

@Arvind Kaushik, oops, sorry, I forgot that header, please include header 'Accept-Encoding:gzip'.

I'll edit my response above to add this line.

steven.peng

Sorry, I think I have given Christiaan wrong information. Anyway, I did a quick test with Python using the URL your provided and was able to retrieve the file as gz file. Have you tested with the Java solution suggested by Jiraponse?

Jiraponse?? do you have a link?

under https://hosted.datascopeapi.reuters.com/RestApi.Help/Home/KeyMechanisms?ctx=Extractions&tab=0&uid=Streaming its mentioned that if request header does not have 'Accept-Encoding:gzip' set then the response header wouldn't have 'content-encoding:gzip' set for response header.

Standard Extractions feeds content can deliver very large files that would increase the burdens on our servers and bandwidth were the files not compressed. In this situation we always deliver the file in a compressed format, although if the Accept-Encoding: gzip is not present, the Content-Encoding: gzip header will not be included in the response, but the file will always be compressed as gzip if that subscription delivers gzipped content.

I do not see this happening for my testing though. I always get content-encoding:gzip causing the stream to be unzipped, contrary to what I want.

I do see the Unirest code where it checks the content-encoding:gzip, however I am not expecting 'content-encoding:gzip' to be set on the response as per : https://hosted.datascopeapi.reuters.com/RestApi.Help/Home/KeyMechanisms?ctx=Extractions&tab=0&uid=Streaming its mentioned that if request header does not have 'Accept-Encoding:gzip' set then the response header wouldn't have 'content-encoding:gzip' set for response header.

Jirapongse

For now, to get the raw gz file, you can use this kind of Java code.

		HttpClient client = HttpClients.custom().disableContentCompression().build();

		HttpGet httpget = new HttpGet("https://hosted.datascopeapi.reuters.com/RestApi/v1/StandardExtractions/UserPackageDeliveries('0x05d0f50a992b2f96')/$value");
		httpget.addHeader("Authorization", "Token <token>");
		CloseableHttpResponse response;
		try {
			response = (CloseableHttpResponse) client.execute(httpget);
			InputStream is = response.getEntity().getContent();
			FileOutputStream fos = new FileOutputStream(new File("c:\\output.cvs.gz"));
			int inByte;
			while((inByte = is.read()) != -1)
			     fos.write(inByte);
			is.close();
			fos.close();
		} catch (ClientProtocolException e) {
			// TODO Auto-generated catch block
			e.printStackTrace();
		} catch (IOException e) {
			// TODO Auto-generated catch block
			e.printStackTrace();
		}