Discover Refinitiv
MyRefinitiv Refinitiv Perspectives Careers
Created with Sketch.
All APIs Questions & Answers  Register |  Login
Ask a question
  • Questions
  • Tags
  • Badges
  • Unanswered
Search:
  • Home /
  • TRTH /
avatar image
Question by Arvind Kaushik · Aug 01, 2017 at 12:11 PM · trth v2streamingvbd file downloadtrth vbdgzip

TRTH v2 using Unirest java http client vbd gzip download issue

We’re having issues while trying to programmatically run DSS StandardExtractions UserPackageDelivery using Unirest java http client.

When trying to stream the file contents using ‘StandardExtractions/UserPackageDeliveries({id})/$value’ request we do not get the file in compressed ‘gzip’ format as advised by the API documentation.

While sending the http GET request we’re sending the header ‘Accept-Encoding: gzip’ as advised. We can also see the response header correctly puts ‘Content-Encoding=gzip’, the response body however appears to be plain text format. How can we receive the streamed file contents in gzip format for StandardExtractions?

jirapongse.phuriphanvichai

People who like this

1 Show 0
Comment
10 |1500 characters needed characters left characters exceeded
▼
  • Viewable by all users
  • Viewable by moderators
  • Viewable by moderators and the original poster
  • Advanced visibility
Viewable by all users

Up to 2 attachments (including images) can be used with a maximum of 512.0 KiB each and 1.0 MiB total.

9 Replies

  • Sort: 
avatar image
REFINITIV
Best Answer
Answer by jirapongse.phuriphanvichai · Aug 03, 2017 at 12:20 AM
@Arvind Kaushik

I think that you are correct. From my test, the response always contain "Content-Encoding: gzip" no matter the "Accept-Encoding: gzip" is present, or not. It conflicts with the statement mentioned in the documents. I will contact the development team to verify it.

For now, to get the raw gz file, you can use this kind of Java code.

		HttpClient client = HttpClients.custom().disableContentCompression().build();
		
		HttpGet httpget = new HttpGet("https://hosted.datascopeapi.reuters.com/RestApi/v1/StandardExtractions/UserPackageDeliveries('0x05d0f50a992b2f96')/$value");
		httpget.addHeader("Authorization", "Token <token>");
		CloseableHttpResponse response;
		try {
			response = (CloseableHttpResponse) client.execute(httpget);
			InputStream is = response.getEntity().getContent();
			FileOutputStream fos = new FileOutputStream(new File("c:\\output.cvs.gz"));
			int inByte;
			while((inByte = is.read()) != -1)
			     fos.write(inByte);
			is.close();
			fos.close();
		} catch (ClientProtocolException e) {
			// TODO Auto-generated catch block
			e.printStackTrace();
		} catch (IOException e) {
			// TODO Auto-generated catch block
			e.printStackTrace();
		}
Comment

People who like this

0 Show 0 · Share
10 |1500 characters needed characters left characters exceeded
▼
  • Viewable by all users
  • Viewable by moderators
  • Viewable by moderators and the original poster
  • Advanced visibility
Viewable by all users

Up to 2 attachments (including images) can be used with a maximum of 512.0 KiB each and 1.0 MiB total.

avatar image
REFINITIV
Answer by jirapongse.phuriphanvichai · Aug 02, 2017 at 05:45 AM

@Arvind Kaushik

Refer to Unirest code HttpReponse.java at line 84, it decompresses the data if the data is gzipped.

To get the gzip file, you need to use the solution mentioned in this question.

Comment

People who like this

0 Show 2 · Share
10 |1500 characters needed characters left characters exceeded
▼
  • Viewable by all users
  • Viewable by moderators
  • Viewable by moderators and the original poster
  • Advanced visibility
Viewable by all users

Up to 2 attachments (including images) can be used with a maximum of 512.0 KiB each and 1.0 MiB total.

avatar image
Arvind Kaushik · Aug 02, 2017 at 09:22 AM 0
Share

I have used client other than Unirest with closeable http client like this:

CloseableHttpClient client = HttpClients.custom().disableContentCompression().build();
Unirest.setHttpClient(client);

Also, I have tried using java library directly too with same result:

	            HttpURLConnection conn= (HttpURLConnection)myURL.openConnection();
	            conn.setRequestProperty("Authorization", "Token " + getAuthToken());
	            conn.setRequestProperty("Accept-Encoding", "gzip");
	            conn.setRequestMethod("GET");
	            conn.getInputStream();



avatar image
Arvind Kaushik · Aug 02, 2017 at 10:52 AM 0
Share

I do see the Unirest code where it checks the content-encoding:gzip, however I am not expecting 'content-encoding:gzip' to be set on the response as per : https://hosted.datascopeapi.reuters.com/RestApi.Help/Home/KeyMechanisms?ctx=Extractions&tab=0&uid=Streaming its mentioned that if request header does not have 'Accept-Encoding:gzip' set then the response header wouldn't have 'content-encoding:gzip' set for response header.

avatar image
REFINITIV
Answer by Christiaan Meihsl · Aug 02, 2017 at 08:48 AM

@Arvind Kaushik, thanks to some out of band information received separately by email I shall answer this one.

If I was well informed, you retrieved the list of user package deliveries with a GET to this URL:

https://hosted.datascopeapi.reuters.com/RestApi/v1/StandardExtractions/UserPackageDeliveryGetUserPackageDeliveriesByPackageId(PackageId='0x0460dc1d24a62cb1')

The result is:

{
    "@odata.context": "https://hosted.datascopeapi.reuters.com/RestApi/v1/$metadata#UserPackageDeliveries",
    "value": [
        {
            "PackageDeliveryId": "MHgwNDYwZGMxZDI0YTYyY2IxfEZGRS1MZWdhY3l8SW5zaWRlcnNcVVNcTW9kZWx2M3xXZWVrbHl8SVRNdjNXZWVrbHkuMjAxNzAyMDI",
            "UserPackageId": "0x0460dc1d24a62cb1",
            "SubscriptionId": "0x0400dc1d24a00cb3",
            "Name": "ITMv3Weekly.20170202",
            "ReleaseDateTime": "2017-02-02T11:31:35.000Z",
            "FileSizeBytes": 314925,
            "Frequency": "Weekly"
        },
etc.

The issue is with the UserPackageId 0x0460dc1d24a62cb1, which is not for TRTH VBD, it is for Insider data. That is why you get something different from what is shown in step 4 of TRTH REST API Tutorial 2 (get list of user package deliveries).

See the value of UserPackageId in this extract from the first step of TRTH REST API Tutorial 2 (list all user packages):

… 
{ 
 "UserPackageId": "0x04f21a8d20c59cb1", 
 "PackageId": "0x04f21a8d20c59cb1", 
 "PackageName": "KAR - Karachi Stock Exchange", 
 "SubscriptionId": "0x0400dc1d24a00cb4", 
 "SubscriptionName": "TRTH Venue by Day" 
}, 
{ 
 "UserPackageId": "0x0460dc1d24a62cb1", 
 "PackageId": "0x0460dc1d24a62cb1", 
 "PackageName": "US Insider Trading Model v3", 
 "SubscriptionId": "0x0400dc1d24a00cb3", 
 "SubscriptionName": "Insider" 
},
…

Use a PackageId that is related to "SubscriptionName": "TRTH Venue by Day" and you will receive TRTH data as described in the tutorial, in gzip format.

Comment

People who like this

0 Show 0 · Share
10 |1500 characters needed characters left characters exceeded
▼
  • Viewable by all users
  • Viewable by moderators
  • Viewable by moderators and the original poster
  • Advanced visibility
Viewable by all users

Up to 2 attachments (including images) can be used with a maximum of 512.0 KiB each and 1.0 MiB total.

avatar image
Answer by Arvind Kaushik · Aug 02, 2017 at 09:25 AM

@Christiaan Meihsl

I'm not sure what your source of information is, but I have tried using number of different requests with similar results. e.g

https://hosted.datascopeapi.reuters.com/RestApi/v1/StandardExtractions/UserPackageDeliveryGetUserPackageDeliveriesByDateRange(SubscriptionId='0x0400dc1d24a00cb4',FromDate=2017-07-30,ToDate=2017-07-31)
	
	{
    "@odata.context": "https://hosted.datascopeapi.reuters.com/RestApi/v1/$metadata#UserPackageDeliveries",
    "value": [
        {
            "PackageDeliveryId": "0x05d0a0119a2b3016",
            "UserPackageId": "0x04f21a8d1f759cb1",
            "SubscriptionId": "0x0400dc1d24a00cb4",
            "Name": "IOM-2017-07-30-NORMALIZEDMP-Report-4-of-6.csv.gz",
            "ReleaseDateTime": "2017-07-31T03:00:00.000Z",
            "FileSizeBytes": 841968,
            "Frequency": "Daily",
            "ContentMd5": ""
        },
	
	https://hosted.datascopeapi.reuters.com/RestApi/v1/StandardExtractions/UserPackageDeliveries('0x05d0a0119a2b3016')/$value
Comment

People who like this

0 Show 0 · Share
10 |1500 characters needed characters left characters exceeded
▼
  • Viewable by all users
  • Viewable by moderators
  • Viewable by moderators and the original poster
  • Advanced visibility
Viewable by all users

Up to 2 attachments (including images) can be used with a maximum of 512.0 KiB each and 1.0 MiB total.

avatar image
REFINITIV
Answer by Christiaan Meihsl · Aug 02, 2017 at 09:59 AM

@Arvind Kaushik, I'm assuming you want to save the data, not decompress it on the fly. If that is the case, can you try this:

String urlGet = urlHost + "/StandardExtractions/UserPackageDeliveries('"+FileId+"')/$value";
try {
    URL myURL = new URL(urlGet);
    HttpURLConnection myURLConnection = (HttpURLConnection)myURL.openConnection();
    myURLConnection.setRequestProperty("Authorization", "Token "+sessionToken);
    myURLConnection.setRequestProperty("Accept-Encoding", "gzip");
    myURLConnection.setRequestProperty("Accept-Charset", "UTF-8");
    myURLConnection.setRequestMethod("GET");

    try( DataInputStream readerIS = new DataInputStream( myURLConnection.getInputStream())) {
        Files.copy (readerIS, Paths.get (filename));
    }

I have not had the occasion to try this with VBD, but I think it should work.

Comment

People who like this

0 Show 0 · Share
10 |1500 characters needed characters left characters exceeded
▼
  • Viewable by all users
  • Viewable by moderators
  • Viewable by moderators and the original poster
  • Advanced visibility
Viewable by all users

Up to 2 attachments (including images) can be used with a maximum of 512.0 KiB each and 1.0 MiB total.

avatar image
Answer by Arvind Kaushik · Aug 02, 2017 at 10:03 AM

@Christiaan Meihsl

You mean without specifying 'Accept-Encoding:gzip' so that the 'Content-Encoding:gzip' is not returned in the response ?

I shall give it a try.

Comment

People who like this

0 Show 1 · Share
10 |1500 characters needed characters left characters exceeded
▼
  • Viewable by all users
  • Viewable by moderators
  • Viewable by moderators and the original poster
  • Advanced visibility
Viewable by all users

Up to 2 attachments (including images) can be used with a maximum of 512.0 KiB each and 1.0 MiB total.

avatar image
REFINITIV
Christiaan Meihsl ♦♦ · Aug 02, 2017 at 10:12 AM 0
Share

@Arvind Kaushik, oops, sorry, I forgot that header, please include header 'Accept-Encoding:gzip'.

I'll edit my response above to add this line.

avatar image
Answer by Arvind Kaushik · Aug 02, 2017 at 10:10 AM

@Christiaan Meihsl

I did try this with Postman, however I still see the content-encoding:gzip being returned in the response header.

Request Headers:
cache-control:"no-cache"
Postman-Token:"ea9146cd-c631-4161-b248-06b476adcb68"
Prefer:"respond-async"
Authorization:"Basic OTAxMjYyMzppaHNtQHJraXQ="
User-Agent:"PostmanRuntime/6.1.6"
Accept:"*/*"
Host:"hosted.datascopeapi.reuters.com"
cookie:"DSSAPI-COOKIE=R3148268809"
accept-encoding:"gzip, deflate"
Response Headers:
set-cookie:"DSSAPI-COOKIE=R3148268809; path=/"
cache-control:"no-cache"
pragma:"no-cache"
content-length:"841968"
content-type:"text/plain"
content-encoding:"gzip"
Comment

People who like this

0 Show 1 · Share
10 |1500 characters needed characters left characters exceeded
▼
  • Viewable by all users
  • Viewable by moderators
  • Viewable by moderators and the original poster
  • Advanced visibility
Viewable by all users

Up to 2 attachments (including images) can be used with a maximum of 512.0 KiB each and 1.0 MiB total.

avatar image
REFINITIV
Christiaan Meihsl ♦♦ · Aug 03, 2017 at 03:06 AM 0
Share

@Arvind Kaushik, my suggestion was actually to try the piece of Java code I posted.

avatar image
REFINITIV
Answer by steven.peng · Aug 02, 2017 at 10:18 AM
@Arvind Kaushik

Sorry, I think I have given Christiaan wrong information. Anyway, I did a quick test with Python using the URL your provided and was able to retrieve the file as gz file. Have you tested with the Java solution suggested by Jiraponse?

Comment

People who like this

0 Show 1 · Share
10 |1500 characters needed characters left characters exceeded
▼
  • Viewable by all users
  • Viewable by moderators
  • Viewable by moderators and the original poster
  • Advanced visibility
Viewable by all users

Up to 2 attachments (including images) can be used with a maximum of 512.0 KiB each and 1.0 MiB total.

avatar image
Arvind Kaushik · Aug 02, 2017 at 10:24 AM 0
Share

Jiraponse?? do you have a link?

avatar image
Answer by Arvind Kaushik · Aug 02, 2017 at 10:37 AM

under https://hosted.datascopeapi.reuters.com/RestApi.Help/Home/KeyMechanisms?ctx=Extractions&tab=0&uid=Streaming its mentioned that if request header does not have 'Accept-Encoding:gzip' set then the response header wouldn't have 'content-encoding:gzip' set for response header.

Standard Extractions feeds content can deliver very large files that would increase the burdens on our servers and bandwidth were the files not compressed. In this situation we always deliver the file in a compressed format, although if the Accept-Encoding: gzip is not present, the Content-Encoding: gzip header will not be included in the response, but the file will always be compressed as gzip if that subscription delivers gzipped content.

I do not see this happening for my testing though. I always get content-encoding:gzip causing the stream to be unzipped, contrary to what I want.

Comment

People who like this

0 Show 0 · Share
10 |1500 characters needed characters left characters exceeded
▼
  • Viewable by all users
  • Viewable by moderators
  • Viewable by moderators and the original poster
  • Advanced visibility
Viewable by all users

Up to 2 attachments (including images) can be used with a maximum of 512.0 KiB each and 1.0 MiB total.

Watch this question

Add to watch list
Add to your watch list to receive emailed updates for this question. Too many emails? Change your settings >
12 People are following this question.

Related Questions

How many days of Historical VBD files can be downloaded

Corporate Actions: How to get the closest 'Shares amount' value to given date for a symbol

while Retrieving schedule id with schedule name request getting stuck

TRTH - direct copy from one AWS S3 bucket to another S3 bucket

Does this mean there is only the new GUI via DataScope Select in TRTH V2 and existing Tick History web access will be removed? (from Webinar May 31)

  • Feedback
  • Copyright
  • Cookie Policy
  • Privacy Statement
  • Terms of Use
  • Careers
  • Anonymous
  • Sign in
  • Create
  • Ask a question
  • Spaces
  • Alpha
  • App Studio
  • Block Chain
  • Bot Platform
  • Calais
  • Connected Risk APIs
  • DSS
  • Data Fusion
  • Data Model Discovery
  • Datastream
  • Eikon COM
  • Eikon Data APIs
  • Electronic Trading
    • Generic FIX
    • Local Bank Node API
    • Trading API
  • Elektron
    • EMA
    • ETA
    • WebSocket API
  • Legal One
  • Messenger Bot
  • Messenger Side by Side
  • ONESOURCE
    • Indirect Tax
  • Open PermID
    • Entity Search
  • Org ID
  • PAM
    • PAM - Logging
  • ProView
  • ProView Internal
  • Product Insight
  • Project Tracking
  • Refinitiv Data Platform
    • Refinitiv Data Platform Libraries
  • Rose's Space
  • Screening
    • Qual-ID API
    • Screening Deployed
    • Screening Online
    • World-Check One
    • World-Check One Zero Footprint
  • Side by Side Integration API
  • TR Knowledge Graph
  • TREP APIs
    • CAT
    • DACS Station
    • Open DACS
    • RFA
    • UPA
  • TREP Infrastructure
  • TRIT
  • TRKD
  • TRTH
  • Thomson One Smart
  • Transactions
    • REDI API
  • Velocity Analytics
  • Wealth Management Web Services
  • World-Check Data File
  • Explore
  • Tags
  • Questions
  • Badges