I'm encountering problems with request timeouts, more of my requests fail than succeed now.
I'm making a request to
and receiving a 202 with a Location header, and my application polls the TR system waiting for a response. But I find that I make my initial request around 23:00:00, then at first I get 202 responses when I poll, but they soon turn into timeouts, even with a 30 second timeout setting on my RestTemplate.
This is requesting 20 years of data, so if the problem is simply that the system cannot start responding within 30 seconds please let me know what a reasonable timeout setting would be, from your POV.
(Also I get a fair few instances of entirely failing to connect:
ResourceAccessException: I/O error on GET request for "https://hosted.datascopeapi.reuters.com/RestApi/v1/Extractions/ExtractWithNotesResult(ExtractionId='0x0673f e52bc7027d2')": Connect to hosted.datascopeapi.reuters.com:443 [hosted.datascopeapi.reuters.com/126.96.36.199] failed: Connection refused (Connection refused); nested exception is org.apache.http.conn.HttpHostConnectException: Connect to hosted.datascopeapi.reuters.com :443 [hosted.datascopeapi.reuters.com/188.8.131.52] failed: Connection refused (Connection refused)
I agree that the long delay before the bytes actually start flowing is undesirable and, while there is a reason behind it, I am not sure it is a very good reason... Extraction 0x0673fe52bc7027d2 produced a fairly large result file and currently, before the bytes can be sent, the entire raw results need to be converted to JSON text before the ExtractWithNotesResults call can start sending bytes. Our development plan includes adding support for streaming JSON results, but I cannot tell you where that is on the development timeline.
You will find that for large result set extractions, the ExtractRaw method is more responsive, although you will receive the data as a CSV file stream and not JSON. You would then deploy your own CSV processing once the bytes are streamed down.
I have some feedback from the development team:
It might be the speed of your network versus the local timeout time being set. We show a connection open for 2m17s and sending 69MB:
2018-12-21 13:36:27.123 2018-12-21 13:34:10.575 GET "9019523" 172.25.182.9 "184.108.40.206" 200 136548 "CiD/9019523/PhQNBQ.0x06744ee083d0280f/RA" 1736183808 12.82 31.82 80 69390327 882 "Apache-HttpClient/4.5.1 (Java/1.8.0_181)" /RestApi/v1/Extractions/ExtractWithNotesResult(ExtractionId='0x0673fe52bc7027d2') -
That’s about 500kB/sec.
How long is your local Apache-HttpClient timeout setting?