Automatic Decompression in HTTP request for Tick History Time and Sales report

leonardo.martins

Hi there!

I'm trying to download the Tick History Time and Sales report via R library httr.

Apparently, the GET method decompress it automatcally, what could be a real problem for large reports.

Is there a way to work around this?

I read in this page that if the request doesn't have the "Accept-Encoding" header, the response wouldn't have the "Content-Encoding" header and the client shouldn't decompress it, but that is not true, the "Content-Encoding" is still present.

So I don't know if the problem is with the R or with the API.

Christiaan Meihsl

@leonardo.martins,

The TRTH servers deliver compressed (or not) data depending on several things, as described under heading "Compression" in this page. For an On Demand Time and Sales extraction I believe it should always deliver compressed data, whatever you set in the GET header.

What does the response header contain ? I would expect it to contain this:

Content-Encoding: gzip Content-Type: text/plain

If that is the case, the content is compressed.

But some HTTP clients automatically decompress data when they receive compressed data (Postman does that) I guess you have run into that as well. You must disable httr’s content decoding, using "config(http_content_decoding=0)" in the GET call.

Here is a code snippet that does it:

TRTHRawExtractionResults <- function(token,jobid,Path,Overwrite = TRUE) {
  url <- paste0("https://hosted.datascopeapi.reuters.com/RestApi/v1/Extractions/RawExtractionResults('",jobid,"')/$value")
  r <- httr::GET(url,add_headers(prefer = "respond-async",Authorization = token),config(http_content_decoding=0),write_disk(Path,Overwrite),progress())
  stop_for_status(r)
  return(r)
}

This is an extract from this article that describes an example of R code that does a time and sales extraction.

If that does not help, please post your code so we can have a look at it.

leonardo.martins

https://community.developers.refinitiv.com/discussion/comment/32916#Comment_32916

Christiaan Meihsl

Oct 4, 2018

@leonardo.martins,

The TRTH servers deliver compressed (or not) data depending on several things, as described under heading "Compression" in this page. For an On Demand Time and Sales extraction I believe it should always deliver compressed data, whatever you set in the GET header.

What does the response header contain ? I would expect it to contain this:

Content-Encoding: gzip Content-Type: text/plain

If that is the case, the content is compressed.

But some HTTP clients automatically decompress data when they receive compressed data (Postman does that) I guess you have run into that as well. You must disable httr’s content decoding, using "config(http_content_decoding=0)" in the GET call.

Here is a code snippet that does it:

TRTHRawExtractionResults <- function(token,jobid,Path,Overwrite = TRUE) {
  url <- paste0("https://hosted.datascopeapi.reuters.com/RestApi/v1/Extractions/RawExtractionResults('",jobid,"')/$value")
  r <- httr::GET(url,add_headers(prefer = "respond-async",Authorization = token),config(http_content_decoding=0),write_disk(Path,Overwrite),progress())
  stop_for_status(r)
  return(r)
}

This is an extract from this article that describes an example of R code that does a time and sales extraction.

If that does not help, please post your code so we can have a look at it.

Thank you very much! It its working perfectly now.