question

Upvotes
Accepted
0 0 0 2

Exception from get bundle data in datastream

Hi,


Im using the Datastream get_bundle_data to returns several items for a universe of stocks. We have set up a loop to cycle our universe. It works fine,but then randomly we get an error and it says it has did connected. Can you please advise me on the following. I should not we are within the max requests and on retry it works.


1. How can we handle this so it avoids this disconnection. We currently have the program use tenacity retry, a wrapper from python but this is less than ideal. We want to avoid these errors all together.

2. Can we use multithreading to make this faster, we don't have this in place currently but would like to do it. Or any other optimisations you can recommend based on screenshots.


here is the exception:


on 40: HTTPSConnectionPool(host='product.datastream.com', port=443): Read timed out.


on 40: _get_json_Response: Exception Occured:


on 40: HTTPSConnectionPool(host='product.datastream.com', port=443):


Read timed out.


on 40: get_bundle_data: Exception Occured


on 40: (<class 'TypeError>, TypeError("argument of type 'None Type' is not


Iterable"), <traceback object at 0x00000227252BCC40>)


on 40: None


on 40: Traceback (most recent call last):


File "C:\Users\sayersqvm\miniconda3/envs/qvmLive\lib\site-


packages\DatastreamDSWS\DS_Response.py", line 209, in get_bundle_data


if 'DataResponses' in json_Response:


TypeError: argument of type 'None Type' is not iterable


on 62: ("Connection aborted.", RemoteDisconnected('Remote end closed connection without response"))


on 62: _get_json_Response: Exception Occured:


on 62: ('Connection aborted.", RemoteDisconnected('Remote end closed


connection without response'))


on 62: get_bundle_data: Exception Occured


on 62: (<class 'TypeError>, TypeError("argument of type 'None Type' is not


Iterable"), <traceback object at 0x0000022724802040>)


on 62: None


Reply


31


Search

#technologyapibulk-download
icon clock
10 |1500

Up to 2 attachments (including images) can be used with a maximum of 512.0 KiB each and 1.0 MiB total.

Upvotes
Accepted
82.5k 278 53 77

@Blue.suit

I found that the default HTTP timeout of the DatastreamDSWS Python library is 180 seconds (3 minutes). Therefore, if you got an error after 3 minutes, it could relate to this setting.

You can change this timeout by using the configuration file (config.ini).

[app]
timeout=300
[url]
path=https://product.datastream.com

Then, set this configuration file in the config parameter.

ds = DSWS.Datastream(username = username, password = password, config='config.ini')

However, if it doesn't solve the issue, it could be a connection or server issue.

Regarding using multiple threads with DatastreamDSWS, please refer to this article: Load Data using Multiprocessing in Python.

icon clock
10 |1500

Up to 2 attachments (including images) can be used with a maximum of 512.0 KiB each and 1.0 MiB total.

Upvotes
14.9k 31 5 10

Hi @Blue.suit ,

Regarding an error you got, have you had a chance to check this thread?

The suggestion from the thread is you can use the requests Python library to check network settings. Please try the following code in the same environment and let us know the result

import requests
 
url = 'https://product.datastream.com/DSWSClient/V1/DSService.svc/rest/GetToken'
myobj = {"Password": "<username>", 
         "Properties": [{"Key": "__AppId", "Value": "PythonLib-1.1.0"}], 
         "UserName": "<password>"}
 
x = requests.post(url, json = myobj)
print(x.status_code)
print(x.text)

You may need to enable HTTP logging in Python.

import logging
import http.client
 
http.client.HTTPConnection.debuglevel = 1
 
logging.basicConfig()
logging.getLogger().setLevel(logging.DEBUG)
requests_log = logging.getLogger("requests.packages.urllib3")
requests_log.setLevel(logging.DEBUG)
requests_log.propagate = True

Hope this helps and please let me know in case you have any further questions

icon clock
10 |1500

Up to 2 attachments (including images) can be used with a maximum of 512.0 KiB each and 1.0 MiB total.

Upvotes
0 0 0 2

Sorry but this is not helpful. This example is not getting access at all. For our issue, we are able to get data, but it will intermittently freeze the call and result in an error. When it is retried, it works fine.

icon clock
10 |1500

Up to 2 attachments (including images) can be used with a maximum of 512.0 KiB each and 1.0 MiB total.

Write an Answer

Hint: Notify or tag a user in this post by typing @username.

Up to 2 attachments (including images) can be used with a maximum of 512.0 KiB each and 1.0 MiB total.