AWS server return different data type when using REST API with python2 instead of python3

timothy.liu.1010

Hi,

I am trying to use REST API in python to extract tick history. I was able to follow the example provided in the download section and extract tick data successfully (in .csv.gzip format) in my development environment, which is in python3. However, when I tested exactly the same code in my production environment (python2) I got an error saying the return data is not in gzip format.

I checked the `Content-Type` field in HTTP response, I found AWS return a xml data, instead of gzip, in my production environment. Message as below:

`Content response headers (AWS server): type: application/xml`

The only difference, as far as I can tell for now, is that I am running with python2. Would this be the problem?

Thanks so much for helping

Find more posts tagged with

tick-history-rest-api

Accepted answers

veerapath.rungruengrayubkul

Hi @timothy.liu.1010,

I have tried the TRTH_OnDemand_IntradayBars.py example which is modified to use TickHistoryTimeAndSales extraction with Python 2.7.9 version, but cannot find the issue. Could you provide the request message you are using? Is there any modification on the example except the request?

Are you able replicate the issue on demand? Could you 1) enable debug logging, 2) add the X-Client-Session-Id header, and 3) log the response header of the RawExtractionResults endpoint?

Please provide the application's console log.

Below is the sample of modified code on Step 5.

#Step 5: get the extraction results, using the received jobId.
...
#   Advisory: avoid incomplete output - decompress then download
# 1) enable debug logging
logging.basicConfig(level=logging.DEBUG)

requestUrl = "https://hosted.datascopeapi.reuters.com/RestApi/v1/Extractions/RawExtractionResults&quot; + "('" + jobId + "')" + "/$value"

#AWS requires an additional header: X-Direct-Download
if useAws:
    requestHeaders={
        "Prefer":"respond-async",
        "Content-Type":"text/plain",
        "Accept-Encoding":"gzip",
        "X-Direct-Download":"true",
        "Authorization": "token " + token,
        # 2) add the X-Client-Session-Id header
        "X-Client-Session-Id": "9FBAA1E1-DFF3-435C-A0CA-1C9A6C46C266"
    }
else:
    requestHeaders={
        "Prefer":"respond-async",
        "Content-Type":"text/plain",
        "Accept-Encoding":"gzip",
        "Authorization": "token " + token
    }

r5 = requests.get(requestUrl,headers=requestHeaders,stream=True)
#Ensure we do not automatically decompress the data on the fly:
r5.raw.decode_content = False
if useAws:
    # 3) log the response header
    for red in r5.history:
        print (red.headers)
    
    print ('Content response headers (AWS server): type: ' + r5.headers["Content-Type"] + '\n')
    #AWS does not set header Content-Encoding="gzip".

All comments

veerapath.rungruengrayubkul

Hi @timothy.liu.1010,

Could you share the snippet code and application log when the issue occurred, so that I will try to replicate the issue?

timothy.liu.1010

I don't know the best way to share snippet here, but my code basically follows the example file in official doc: TRTH_OnDemand_IntradayBars.py, except we are extracting Time and Sales, instead of Intraday Summary.

Which kind of log are you looking for? I can try to paste it here.

Thanks

veerapath.rungruengrayubkul

Hi @timothy.liu.1010,

Are you able replicate the issue on demand? Could you 1) enable debug logging, 2) add the X-Client-Session-Id header, and 3) log the response header of the RawExtractionResults endpoint?

Please provide the application's console log.

Below is the sample of modified code on Step 5.

#Step 5: get the extraction results, using the received jobId.
...
#   Advisory: avoid incomplete output - decompress then download
# 1) enable debug logging
logging.basicConfig(level=logging.DEBUG)

requestUrl = "https://hosted.datascopeapi.reuters.com/RestApi/v1/Extractions/RawExtractionResults&quot; + "('" + jobId + "')" + "/$value"

#AWS requires an additional header: X-Direct-Download
if useAws:
    requestHeaders={
        "Prefer":"respond-async",
        "Content-Type":"text/plain",
        "Accept-Encoding":"gzip",
        "X-Direct-Download":"true",
        "Authorization": "token " + token,
        # 2) add the X-Client-Session-Id header
        "X-Client-Session-Id": "9FBAA1E1-DFF3-435C-A0CA-1C9A6C46C266"
    }
else:
    requestHeaders={
        "Prefer":"respond-async",
        "Content-Type":"text/plain",
        "Accept-Encoding":"gzip",
        "Authorization": "token " + token
    }

r5 = requests.get(requestUrl,headers=requestHeaders,stream=True)
#Ensure we do not automatically decompress the data on the fly:
r5.raw.decode_content = False
if useAws:
    # 3) log the response header
    for red in r5.history:
        print (red.headers)
    
    print ('Content response headers (AWS server): type: ' + r5.headers["Content-Type"] + '\n')
    #AWS does not set header Content-Encoding="gzip".

EXPLORE OUR SITES