Hello @bartwozniak
The websocket connection does maintain a ping/pong connectivity to server and after timeout period will throw and exception. At this point the library would try to recover the connection and resubscribe to the instruments.
I used the streaming events example from the LD library and simulated a connection break and received the websockets error followed by auto recovery:
Update : JPY=:{'BID': 148.82, 'ASK': 148.83} Update : JPY=:{'BID': 148.82, 'ASK': 148.83} Update : GBP=:{'BID': 1.2965, 'ASK': 1.2969} [OMMSTREAMING_PRICING_0.0] on_ws_error: Exception: [WinError 10054] An existing connection was forcibly closed by the remote host Refresh : Instrument BID ASK 0 EUR= 1.0848 1.085 1 GBP= 1.2965 1.2969 2 JPY= 148.82 148.83 3 BADRIC <NA> <NA> Refresh : Instrument BID ASK 0 EUR= 1.0848 1.085 1 GBP= 1.2965 1.2969 2 JPY= 148.82 148.84 3 BADRIC <NA> <NA> Refresh : Instrument BID ASK 0 EUR= 1.0848 1.085 1 GBP= 1.2965 1.2969 2 JPY= 148.82 148.84 3 BADRIC <NA> <NA> Update : GBP=:{'BID': 1.2965, 'ASK': 1.2969} Status : BADRIC:NotFound:**The record could not be found Update : JPY=:{'BID': 148.82, 'ASK': 148.84} Update : EUR=:{'BID': 1.0849, 'ASK': 1.085} …
How to best handle connection errors in Python?

Hi community!
I am connecting to pricing streams using the LSEG Python library (version lseg.data==2.0.1)
Here is a sample code. I am then able to register callbacks with the stream and listen to the data.
session = ld.session.platform.Definition( app_key=api_key, grant=ld.session.platform.ClientCredentials( client_id='...', client_secret='...' ), signon_control=False, ).get_session() session.open() streams = ld.content.pricing.Definition( universe=REQUESTED_RICS, fields=REQUESTED_FIELDS ).get_stream(session) streams.open()
My question is about handling connection problems. When I simulate internet connectivity going down on my box, the library does not throw any errors and the websocket connection does not seem to time out. How to best handle such issues?
Ideal for my use case would be to get an exception so that I can exit and launch a new process right away.
Thanks,
Bart
Answers
-
Hello @bartwozniak
The websocket connection does maintain a ping/pong connectivity to server and after timeout period will throw and exception. At this point the library would try to recover the connection and resubscribe to the instruments.
I used the streaming events example from the LD library and simulated a connection break and received the websockets error followed by auto recovery:
Update : JPY=:{'BID': 148.82, 'ASK': 148.83} Update : JPY=:{'BID': 148.82, 'ASK': 148.83} Update : GBP=:{'BID': 1.2965, 'ASK': 1.2969} [OMMSTREAMING_PRICING_0.0] on_ws_error: Exception: [WinError 10054] An existing connection was forcibly closed by the remote host Refresh : Instrument BID ASK 0 EUR= 1.0848 1.085 1 GBP= 1.2965 1.2969 2 JPY= 148.82 148.83 3 BADRIC <NA> <NA> Refresh : Instrument BID ASK 0 EUR= 1.0848 1.085 1 GBP= 1.2965 1.2969 2 JPY= 148.82 148.84 3 BADRIC <NA> <NA> Refresh : Instrument BID ASK 0 EUR= 1.0848 1.085 1 GBP= 1.2965 1.2969 2 JPY= 148.82 148.84 3 BADRIC <NA> <NA> Update : GBP=:{'BID': 1.2965, 'ASK': 1.2969} Status : BADRIC:NotFound:**The record could not be found Update : JPY=:{'BID': 148.82, 'ASK': 148.84} Update : EUR=:{'BID': 1.0849, 'ASK': 1.085} …
0 -
Thank you for your answer @Gurpreet.
What version of the library did you use? I am trying to reproduce this locally. Unfortunately, I observe that the websocket client used in the library is not configured with ping timeouts.
0 -
I am also using the latest version of lseg-data = 2.0.1.
The ping/pong messages are the part of websockets protocol specification - which is used in the streaming events example. The library would not be able to work without implementing this functionality.
0 -
@Gurpreet thank you again for getting back to me on this issue.
I looked into the library code and I noticed that for the
platform
session the ws client runs usingrun_forever
method on theWebSocketApp
. The source is located atlseg/data/delivery/_stream/stream_connection.py
line 276. run_forever is invoked without any ping settings while the documentation say that if ping interval is set to 0 (default) then no pings will be sent. Please see here for detailsAre you sure that the pings are exchanged? When I simulate my internet connection going down, I do not get any timeouts from the websocket for at least 30 minutes.
0 -
The TR_JSON2 websocket protocol defines its own ping/pong messages and not the ping as defined in the Websocket protocol specification.
Here is a raw package of the above request which shows the streaming messages and the pings:
0 -
I understand, thank you. What is the timeout on the ping set to and can I make it shorter than 30 minutes I observed? I would like the library to notify me faster when there is a problem with the connection.
0 -
Hello @bartwozniak
The ping timeout can be adjustment on the server side (ADS on RTO) only.
Alternatively, you may consider switch to the core
which is a WebSocket transport layer of the . However, you need to implement everything such as connection management, session management, authentication, subscription management, etc.) by yourself with the WebSocket API. It means you can get all status and handle them based on your preference.The Python WebSocket API for RTO connection examples are available at https://github.com/Refinitiv/websocket-api/tree/master/Applications/Examples/RTO/python page.
0 -
Thank you for extra info.
Where exactly is this configuration for ADS? I can't find it.
So the server will timeout, but what about my client application if I get disconnected from the internet? I don't think the client will know as I have verified experimentally. Is that expected?
0 -
Websockets is based on TCP protocol and if your machine disconnects from internet, the application will immediately know about it. The ping/pong message handshake happens every 30 seconds, and is used when the traffic to server is disrupted and or there is network contention. I think default is 3 missed ping messages = connection loss.
Are you connecting to a local ADS? The code that you show in question is getting data from RTO in the cloud!
0 -
@Gurpreet I am using the exact same code I showed in the first post, hence it is connecting to a AWS endpoint. I appreciate all the info on the underlying technology, however, I am still failing to make it crash when internet connection is broken. Do you have any tips for how I can reproduce your result? My process keeps on hanging for a very long time when connection is broken.
0 -
Hello @bartwozniak
Please enable debugging in the library and attach the logs. We can try to investigate what might be going on.
0 -
I disabled the network interface on my device after the last log message and killed the processes 15 minutes later.
0 -
Killing the connection or your router is a better way since it simulates the real world connection loss - giving the library a chance to try this or another server again.
However, when the networking is disabled on the machine, I see that no error is generated. I can raise this with library developer to see if there is a fix for this.
0 -
That will be most useful, thank you @Gurpreet . Let me know what they say and thanks for all the insights.
0 -
Hi @bartwozniak ,
To reply to your initial question "How to best handle connection errors in Python?", especially if you want to trace and manage disconnection, you can find how to implement session event and state notifications in LSEG Data Library for Python - Reference Guide : Callback functions section.
on_event callback will notify on following SessionEventCode:Event code
Description
SessionEventCode.StreamConnecting
The connection to the stream service within the session is pending.
SessionEventCode.StreamConnected
The connection to the stream service has been successfully established.
SessionEventCode.StreamDisconnected
The connection to the stream service is not established.
SessionEventCode.StreamAuthenticationSuccess
The session has successfully authenticated this client.
SessionEventCode.StreamAuthenticationFailed
The session has failed to authenticate this client.
SessionEventCode.SessionConnecting
The session is establishing the connections.
SessionEventCode.SessionConnected
The client successfully connected with the current session.
SessionEventCode.SessionDisconnected
The session disconnected the client.
SessionEventCode.SessionAuthenticationSuccess
The session has successfully authenticated this client.
SessionEventCode.SessionAuthenticationFailed
The session has failed to authenticate this client.
SessionEventCode.SessionReconnecting
The session is currently establishing the connection.
SessionEventCode.DataRequestOk
The request for content from the session data services has completed successfully.
SessionEventCode.DataRequestFailed
The request for content from the session data services has failed.
At PricingStream level, you can also manage notifications. Full example is described in TUT_2.2.03-Pricing-StreamingEvents.ipynb from tutorials.
Regarding ping/pong messages at websocket level, it's automatically manage by websocket-client lib as part of the protocol.
LD doesn't do anything except that logging the event with on_ping & on_pong callbacks:self._listener = WebSocketClient( url=self._config.url, header=headers, cookie=cookie, on_open=self._on_ws_open, on_message=self._on_message, on_error=self._on_ws_error, on_close=self._on_ws_close, on_ping=self._on_ws_ping, on_pong=self._on_ws_pong, subprotocols=subprotocols )
As @Gurpreet mentionned, TR_JSON WebSocket protocol has its own ping/pong messages (which are different than previous), and the timeout is driven by the server. LD is implemented to reply to received ping message to keep streaming connection up. The log you uploaded contains related received ping/sent pong messages.
2
Categories
- All Categories
- 3 Polls
- 6 AHS
- 36 Alpha
- 166 App Studio
- 6 Block Chain
- 4 Bot Platform
- 18 Connected Risk APIs
- 47 Data Fusion
- 34 Data Model Discovery
- 684 Datastream
- 1.4K DSS
- 615 Eikon COM
- 5.2K Eikon Data APIs
- 10 Electronic Trading
- Generic FIX
- 7 Local Bank Node API
- 3 Trading API
- 2.9K Elektron
- 1.4K EMA
- 249 ETA
- 554 WebSocket API
- 37 FX Venues
- 14 FX Market Data
- 1 FX Post Trade
- 1 FX Trading - Matching
- 12 FX Trading – RFQ Maker
- 5 Intelligent Tagging
- 2 Legal One
- 23 Messenger Bot
- 3 Messenger Side by Side
- 9 ONESOURCE
- 7 Indirect Tax
- 60 Open Calais
- 275 Open PermID
- 44 Entity Search
- 2 Org ID
- 1 PAM
- PAM - Logging
- 6 Product Insight
- Project Tracking
- ProView
- ProView Internal
- 22 RDMS
- 1.9K Refinitiv Data Platform
- 643 Refinitiv Data Platform Libraries
- 4 LSEG Due Diligence
- LSEG Due Diligence Portal API
- 4 Refinitiv Due Dilligence Centre
- Rose's Space
- 1.2K Screening
- 18 Qual-ID API
- 13 Screening Deployed
- 23 Screening Online
- 12 World-Check Customer Risk Screener
- 1K World-Check One
- 46 World-Check One Zero Footprint
- 45 Side by Side Integration API
- 2 Test Space
- 3 Thomson One Smart
- 10 TR Knowledge Graph
- 151 Transactions
- 143 REDI API
- 1.8K TREP APIs
- 4 CAT
- 26 DACS Station
- 121 Open DACS
- 1.1K RFA
- 104 UPA
- 192 TREP Infrastructure
- 228 TRKD
- 915 TRTH
- 5 Velocity Analytics
- 9 Wealth Management Web Services
- 90 Workspace SDK
- 11 Element Framework
- 5 Grid
- 18 World-Check Data File
- 1 Yield Book Analytics
- 46 中文论坛