question

Upvotes
Accepted
9 5 3 8

TRTH: Python Extract market depth data

Hi,

I'm in trouble to extract data from http response object.

I can't figure out how to extract data using job ID and $value endpoint. Use case reference says users can download the report by folowing final step, but can't find any gzip file in my directory.

How can I retrieve or find data from report_gzip or anyother types of objjects?

Thanks,

report_gzip= requests.get( "https://hosted.datascopeapi.reuters.com/RestApi/v1/Extractions/RawExtractionResults('0x05c5d18a57bb2f96')/$value",
                            headers = header2, stream=True)
pythontick-history-rest-api
icon clock
10 |1500

Up to 2 attachments (including images) can be used with a maximum of 512.0 KiB each and 1.0 MiB total.

Upvotes
Accepted
11.3k 25 9 14

@YK

The raw extraction result will be streamed in gzip format, and then stored in the Response object. You can use this method of the Python Requests module to save raw gzip streamed response to file.

Below is the sample code. You can also see this question.

filename = "result.csv.gz"

with open(filename,'wb') as f:
   for data in report_gzip.raw.stream(decode_content=False):
       f.write(data)
icon clock
10 |1500

Up to 2 attachments (including images) can be used with a maximum of 512.0 KiB each and 1.0 MiB total.

Upvotes
9 5 3 8

Thank you, @veerapath.rungruengrayubkul

It works perfect on my environment. I was just wondering how to extract gzip file from response object, but it's now crystal clear.

Thanks again!

icon clock
10 |1500

Up to 2 attachments (including images) can be used with a maximum of 512.0 KiB each and 1.0 MiB total.

@YK

I have just realized that the sample code I provided might lead to the issue that data will not be completely decompressed if the extracted result is huge.

I have edited the sample code in my answer. Please use the new sample code instead.

@YK

If you want to process the extracted result instead of writing to a gzip file, you need to use gzip to decompress the data. Below is the sample code.

import io
import gzip

fd = io.BytesIO()
for data in report_gzip.raw.stream(decode_content=False):
    fd.write(data)


#decompress data
decompress_data = gzip.decompress(fd.getbuffer());
data = io.StringIO(decompress_data.decode('utf-8'))
    for line in data:
        print(line+'\n')

Hope this help.

Upvotes
9 5 3 8
@ veerapath.rungruengrayubkul

ok will use that sample code if I need to process the result, thank you!

icon clock
10 |1500

Up to 2 attachments (including images) can be used with a maximum of 512.0 KiB each and 1.0 MiB total.

Write an Answer

Hint: Notify or tag a user in this post by typing @username.

Up to 2 attachments (including images) can be used with a maximum of 512.0 KiB each and 1.0 MiB total.