Discover Refinitiv
MyRefinitiv Refinitiv Perspectives Careers
Created with Sketch.
All APIs Questions & Answers  Register |  Login
Ask a question
  • Questions
  • Tags
  • Badges
  • Unanswered
Search:
  • Home /
  • TRTH /
avatar image
Question by kai fang · Oct 29, 2018 at 04:21 PM · dss-rest-apitick-history-rest-apiextraction

AWS File Extraction not working properly

I try to get the stream file of my schedule task, and read it into python, using the rest api call, using the following headers:

headers = {'Authorization': 'Token %s' % auth, 'Prefer': 'respond-async', 'Accept-Charset': 'UTF-8', 'Content-Type': 'application/json',"X-Direct-Download":"true"}

When i do not include X-Direct-Download, i only get part of the data, compared to the csv file i directly download in the web GUI, however, by adding this, i get some text result like the following:

'\x1f�\x08\x00\x00\x00\x00\x00\x00\x00�][s\x1c�n~?��u�\x1aw��\x17��\x12m1�(\x15I%�_T�M�RŖO,�R��\x01fv�;3��Pܥ\x1d\x17�ds���h4\x1a@��/���V�\x7f��������\x7f�yr�\u19db�\u05ef�'

which is clearly not correct. Is there any way to get correct full file as a stream in aws enviorment.

full code:

result = simple_get("/Extractions/ExtractedFiles('%s')/$value" % file_id, auth)

where:

def simple_get(endpoint, auth):

headers = {'Authorization': 'Token %s' % auth, 'Prefer': 'respond-async', 'Accept-Charset': 'UTF-8', 'Content-Type': 'application/json',"X-Direct-Download":"true"}

r = requests.get(url_base + endpoint, headers=headers)

return r

People who like this

0 Show 0
Comment
10 |1500 characters needed characters left characters exceeded
▼
  • Viewable by all users
  • Viewable by moderators
  • Viewable by moderators and the original poster
  • Advanced visibility
Viewable by all users

Up to 2 attachments (including images) can be used with a maximum of 512.0 KiB each and 1.0 MiB total.

3 Replies

  • Sort: 
avatar image
REFINITIV
Best Answer
Answer by Christiaan Meihsl · Oct 30, 2018 at 05:08 AM

@@kf2449, in addition to the response by Jirapongse:

What you observe is actually gzipped data:

'\x1f�\x08\x00\x00\x00\x00\x00\x00\x00�][s\x1c�n~?��u�\x1aw��\x17��\x12m1�(\x15I%�_T�M�RŖO,�R��\x01fv�;3��Pܥ\x1d\x17�ds���h4\x1a@��/���V�\x7f��������\x7f�yr�\u19db�\u05ef�'

When handling the data, you are getting different behaviors due to the response headers, which differ between TRTH and AWS downloads.

TRTH returns the following content related headers:

Content-Encoding:gzip
Content-Type:text/plain

AWS returns different headers:

Content-Disposition:attachment; filename=_OnD_0x05de99857e5b3036.csv.gz
Content-Type:application/gzip

Your application reacts differently to these (many code libraries automatically decide to decompress data (or not) based on response headers), that is why you observe different data formats. It is best to set the appropriate headers and parameters to avoid such behavior. If you follow what is explained in the links sent by Jirapongse you should be able to download the entire file, in the correct format.

You can also look at the Python code samples set available under the downloads tab, and explained in this document. One of the samples (TRTH_Get_Latest_Schedule_Files) should be of help.

Comment

People who like this

0 Show 0 · Share
10 |1500 characters needed characters left characters exceeded
▼
  • Viewable by all users
  • Viewable by moderators
  • Viewable by moderators and the original poster
  • Advanced visibility
Viewable by all users

Up to 2 attachments (including images) can be used with a maximum of 512.0 KiB each and 1.0 MiB total.

avatar image
REFINITIV
Answer by Jirapongse · Oct 30, 2018 at 04:46 AM

@kf2449

To avoid incomplete output, you need to download the Gzip file and then decompress it. It was mentioned in this documentation ADVISORY: AVOID INCOMPLETE OUTPUT - DOWNLOAD THEN DECOMPRESS. You can also refer to this article: How to Optimize TRTH (Tick History) file downloads for Python (and other languages) for Python.

To download from AWS, please refer to Boost TRTH downloads with AWS.

Comment

People who like this

0 Show 0 · Share
10 |1500 characters needed characters left characters exceeded
▼
  • Viewable by all users
  • Viewable by moderators
  • Viewable by moderators and the original poster
  • Advanced visibility
Viewable by all users

Up to 2 attachments (including images) can be used with a maximum of 512.0 KiB each and 1.0 MiB total.

avatar image
REFINITIV
Answer by veerapath.rungruengrayubkul · Oct 30, 2018 at 04:56 AM

@kf2449

The response text likely is in gzip compression format. Normally, TRTH extraction provides data in gzip compress format. The Python Requests library automatically decompress data if the response header contains Content-Encoding="gzip". However, AWS does not set this header, so the Requests call does not decompress. Application needs to decompress data in separated call. Please see code example in the "TRTH Python Code" downloaded from this link.

Comment

People who like this

0 Show 0 · Share
10 |1500 characters needed characters left characters exceeded
▼
  • Viewable by all users
  • Viewable by moderators
  • Viewable by moderators and the original poster
  • Advanced visibility
Viewable by all users

Up to 2 attachments (including images) can be used with a maximum of 512.0 KiB each and 1.0 MiB total.

Watch this question

Add to watch list
Add to your watch list to receive emailed updates for this question. Too many emails? Change your settings >
13 People are following this question.

Related Questions

Data Extraction issue with Reuters DSS C# API

ThomsonReuters Request Examples

Are there any TRTH On Demand limitations on results returned? Only seeing 22K lines returned for a given symbol

TuckHistory / retrieve scheduled extraction / REST/JSON

How to get single request for multiple symbols for RIC rename history using DSS / TRTH / HistoricalReference / HistoricalSearch

  • Copyright
  • Cookie Policy
  • Privacy Statement
  • Terms of Use
  • Anonymous
  • Sign in
  • Create
  • Ask a question
  • Spaces
  • Alpha
  • App Studio
  • Block Chain
  • Bot Platform
  • Connected Risk APIs
  • DSS
  • Data Fusion
  • Data Model Discovery
  • Datastream
  • Eikon COM
  • Eikon Data APIs
  • Electronic Trading
    • Generic FIX
    • Local Bank Node API
    • Trading API
  • Elektron
    • EMA
    • ETA
    • WebSocket API
  • Intelligent Tagging
  • Legal One
  • Messenger Bot
  • Messenger Side by Side
  • ONESOURCE
    • Indirect Tax
  • Open Calais
  • Open PermID
    • Entity Search
  • Org ID
  • PAM
    • PAM - Logging
  • ProView
  • ProView Internal
  • Product Insight
  • Project Tracking
  • RDMS
  • Refinitiv Data Platform
    • Refinitiv Data Platform Libraries
  • Rose's Space
  • Screening
    • Qual-ID API
    • Screening Deployed
    • Screening Online
    • World-Check One
    • World-Check One Zero Footprint
  • Side by Side Integration API
  • TR Knowledge Graph
  • TREP APIs
    • CAT
    • DACS Station
    • Open DACS
    • RFA
    • UPA
  • TREP Infrastructure
  • TRKD
  • TRTH
  • Thomson One Smart
  • Transactions
    • REDI API
  • Velocity Analytics
  • Wealth Management Web Services
  • Workspace SDK
    • Element Framework
    • Grid
  • World-Check Data File
  • 中文论坛
  • Explore
  • Tags
  • Questions
  • Badges