question

Upvotes
Accepted
_Tom_ avatar image
17 ●1 ●1 ●4

Python websocket connection to Refinitiv Realtime fails with SSL error

I am trying to connect to the Refinitiv Realtime Optimized platform in AWS using websockets in python.

I am using the market_price_rdpgw_service_discovery.py example from https://github.com/Refinitiv/websocket-api/tree/master/Applications/Examples/RDP/python#windowslinuxmacos


The HTTP requests for service discovery work and I can setup a token but the websocket connection fails:

Sending Refinitiv Data Platform service discovery request to https://api.refinitiv.com/streaming/pricing/v1/

Refinitiv Data Platform Service discovery succeeded. RECEIVED:
....

Connecting to WebSocket wss://eu-west-1-aws-3-sm.optimized-pricing-api.refinitiv.net:443/WebSocket for session1...
Traceback (most recent call last):
  File "/home/tom/anaconda3/envs/rrt_client/lib/python3.8/site-packages/websocket/_app.py", line 312, in run_forever
    self.sock.connect(
  File "/home/tom/anaconda3/envs/rrt_client/lib/python3.8/site-packages/websocket/_core.py", line 248, in connect
    self.sock, addrs = connect(url, self.sock_opt, proxy_info(**options),
  File "/home/tom/anaconda3/envs/rrt_client/lib/python3.8/site-packages/websocket/_http.py", line 121, in connect
    sock = _ssl_socket(sock, options.sslopt, hostname)
  File "/home/tom/anaconda3/envs/rrt_client/lib/python3.8/site-packages/websocket/_http.py", line 256, in _ssl_socket
    sock = _wrap_sni_socket(sock, sslopt, hostname, check_hostname)
  File "/home/tom/anaconda3/envs/rrt_client/lib/python3.8/site-packages/websocket/_http.py", line 231, in _wrap_sni_socket
    return context.wrap_socket(
  File "/home/tom/anaconda3/envs/rrt_client/lib/python3.8/ssl.py", line 500, in wrap_socket
    return self.sslsocket_class._create(
  File "/home/tom/anaconda3/envs/rrt_client/lib/python3.8/ssl.py", line 1040, in _create
    self.do_handshake()
  File "/home/tom/anaconda3/envs/rrt_client/lib/python3.8/ssl.py", line 1309, in do_handshake
    self._sslobj.do_handshake()
ssl.SSLEOFError: EOF occurred in violation of protocol (_ssl.c:1131)
EOF occurred in violation of protocol (_ssl.c:1131) for session1
WebSocket Closed for session1


I've tried disabling all SSL verification and fixing the version to TSL 1.2 but still get the same error on handshake with the socket.

 def connect(self):
        # Start websocket handshake
        ws_address = "wss://{}/WebSocket".format(self.host)
        print("Connecting to WebSocket " + ws_address + " for " + self.session_name + "...")
        self.web_socket_app = websocket.WebSocketApp(ws_address,
                                                     on_message=self._on_message,
                                                     on_error=self._on_error,
                                                     on_close=self._on_close,
                                                     on_open=self._on_open,
                                                     subprotocols=['tr_json2'])  
        sslopt = {
            'check_hostname': False,
            "verify_mode": ssl.CERT_NONE,
            "cert_reqs": ssl.CERT_NONE,
            "ssl_version": ssl.PROTOCOL_TLSv1_2,
        }
        wst = threading.Thread(target=self.web_socket_app.run_forever, kwargs={'sslopt': sslopt})
        wst.start()


Here are some of the relevant library version:

pip list |grep 'socket\|request\|http\|cert\|url'
certifi              2021.10.8
ndg-httpsclient      0.5.1
requests             2.22.0
requests-toolbelt    0.9.1
urllib3              1.25.11
websocket-client     1.1.0


Has anyone else seen this and been able to resolve?

This is running behind a corporate proxy which I have no control over, but the HTTP requests for service discovery are working which makes me thing the SSL setup is OK in python requests generally, but there is an issue with websockets.

pythonwebsocketsrefinitiv-realtime-optimisedssl
icon clock
10 |1500

Up to 2 attachments (including images) can be used with a maximum of 512.0 KiB each and 1.0 MiB total.

Shadab Hussain avatar image   Shadab Hussain

Hi @_Tom_,

Thank you for your participation in the forum. Is the reply below satisfactory in resolving your query?

If so please can you click the 'Accept' text next to the appropriate reply? This will guide all community members who have a similar question.


Thanks,

AHS

4 Answers

· Write an Answer
Upvotes
Accepted
Jirapongse avatar image
79.2k ●251 ●52 ●74

@_Tom_

I tested with the mitmdump (Man In The Middle Dump) proxy.

The code can run properly with the http_proxy environment variable.

1646044691882.png



1646044691882.png (79.3 KiB)
icon clock
10 |1500

Up to 2 attachments (including images) can be used with a maximum of 512.0 KiB each and 1.0 MiB total.

_Tom_ avatar image _Tom_
Thanks for checking the proxy setup, I've commented below on the real issue which was company specific network setup. But this will be useful for others.
Upvotes
umer.nalla avatar image
25.3k ●87 ●12 ●25

Hi @_Tom_

We cannot provide much Websocket API guidance in terms of dealing with corporate proxies as there can be many factors at play and variations depending on your local security implementations.

However, a few things to try

Certainly, I am aware of a couple of users where the 2nd option allowed to connect.

You should speak to your internal security team for further guidance. You can also raise a ticket with the RTO team - who may be able to provide some assistance to your security team.


icon clock
10 |1500

Up to 2 attachments (including images) can be used with a maximum of 512.0 KiB each and 1.0 MiB total.

Upvotes
_Tom_ avatar image
17 ●1 ●1 ●4

I've tried setting http_proxy and https_proxy, it works for the initial http requests but not for the websocket. If I set them incorrectly then I can break the normal requests, setting them correctly goes back to trying the websocket connection and failing:

Connecting to WebSocket wss://eu-west-1-aws-3-sm.optimized-pricing-api.refinitiv.net:443/WebSocket for session1...
Traceback (most recent call last):
  File "/home/tom/anaconda3/envs/rrt_client/lib/python3.8/site-packages/websocket/_app.py", line 312, in run_forever
    self.sock.connect(
  File "/home/tom/anaconda3/envs/rrt_client/lib/python3.8/site-packages/websocket/_core.py", line 248, in connect
    self.sock, addrs = connect(url, self.sock_opt, proxy_info(**options),
  File "/home/tom/anaconda3/envs/rrt_client/lib/python3.8/site-packages/websocket/_http.py", line 117, in connect
    sock = _tunnel(sock, hostname, port, auth)
  File "/home/tom/anaconda3/envs/rrt_client/lib/python3.8/site-packages/websocket/_http.py", line 287, in _tunnel
    raise WebSocketProxyException(
websocket._exceptions.WebSocketProxyException: failed CONNECT via proxy status: 504
failed CONNECT via proxy status: 504 for session1
WebSocket Closed for session1

Basic network connectivity works on port 443 based on telnet:

telnet eu-west-1-aws-3-sm.optimized-pricing-api.refinitiv.net 443
Trying 159.43.201.118...
Connected to eu-west-1-aws-3-sm.optimized-pricing-api.refinitiv.biz.
Escape character is '^]'.
icon clock
10 |1500

Up to 2 attachments (including images) can be used with a maximum of 512.0 KiB each and 1.0 MiB total.

Upvote
_Tom_ avatar image
17 ●1 ●1 ●4

For anyone else reading this thread with similar problems it turned out to be network configuration issues. The hosts returned from the service discovery call resolve to internal IPs that we didn't have routing for (e.g. refinitiv.net were CNAMEs for refinitiv.biz internal addresses).

We did have network connectivity setup for us-east region and just set that group of hosts. The EOF error from the websocket ssl handshake is really because the resolved IP is not a refinitiv RTO server due to the network setup.

icon clock
10 |1500

Up to 2 attachments (including images) can be used with a maximum of 512.0 KiB each and 1.0 MiB total.

Write an Answer

Hint: Notify or tag a user in this post by typing @username.

Up to 2 attachments (including images) can be used with a maximum of 512.0 KiB each and 1.0 MiB total.