How to best handle connection errors in Python?

Hi community!

I am connecting to pricing streams using the LSEG Python library (version lseg.data==2.0.1)

Here is a sample code. I am then able to register callbacks with the stream and listen to the data.

session = ld.session.platform.Definition(
      app_key=api_key,
      grant=ld.session.platform.ClientCredentials(
          client_id='...',
          client_secret='...'
      ),
      signon_control=False,
  ).get_session()
  session.open()

  streams = ld.content.pricing.Definition(
      universe=REQUESTED_RICS,
      fields=REQUESTED_FIELDS
  ).get_stream(session)

  streams.open()

My question is about handling connection problems. When I simulate internet connectivity going down on my box, the library does not throw any errors and the websocket connection does not seem to time out. How to best handle such issues?

Ideal for my use case would be to get an exception so that I can exit and launch a new process right away.

Thanks,

Bart

Welcome!

It looks like you're new here. Sign in or register to get started.

Answers

  • Hello @bartwozniak

    The websocket connection does maintain a ping/pong connectivity to server and after timeout period will throw and exception. At this point the library would try to recover the connection and resubscribe to the instruments.

    I used the streaming events example from the LD library and simulated a connection break and received the websockets error followed by auto recovery:

    Update : JPY=:{'BID': 148.82, 'ASK': 148.83}
    Update : JPY=:{'BID': 148.82, 'ASK': 148.83}
    Update : GBP=:{'BID': 1.2965, 'ASK': 1.2969}
    [OMMSTREAMING_PRICING_0.0] on_ws_error: Exception: [WinError 10054] An existing connection was forcibly closed by the remote host
    Refresh :   Instrument     BID     ASK
    0       EUR=  1.0848   1.085
    1       GBP=  1.2965  1.2969
    2       JPY=  148.82  148.83
    3     BADRIC    <NA>    <NA>
    Refresh :   Instrument     BID     ASK
    0       EUR=  1.0848   1.085
    1       GBP=  1.2965  1.2969
    2       JPY=  148.82  148.84
    3     BADRIC    <NA>    <NA>
    Refresh :   Instrument     BID     ASK
    0       EUR=  1.0848   1.085
    1       GBP=  1.2965  1.2969
    2       JPY=  148.82  148.84
    3     BADRIC    <NA>    <NA>
    Update : GBP=:{'BID': 1.2965, 'ASK': 1.2969}
    Status : BADRIC:NotFound:**The record could not be found
    Update : JPY=:{'BID': 148.82, 'ASK': 148.84}
    Update : EUR=:{'BID': 1.0849, 'ASK': 1.085}
    
  • Thank you for your answer @Gurpreet.

    What version of the library did you use? I am trying to reproduce this locally. Unfortunately, I observe that the websocket client used in the library is not configured with ping timeouts.

  • I am also using the latest version of lseg-data = 2.0.1.

    The ping/pong messages are the part of websockets protocol specification - which is used in the streaming events example. The library would not be able to work without implementing this functionality.

  • @Gurpreet thank you again for getting back to me on this issue.

    I looked into the library code and I noticed that for the platform session the ws client runs using run_forever method on the WebSocketApp. The source is located at lseg/data/delivery/_stream/stream_connection.py line 276. run_forever is invoked without any ping settings while the documentation say that if ping interval is set to 0 (default) then no pings will be sent. Please see here for details https://websocket-client.readthedocs.io/en/latest/app.html#websocket._app.WebSocketApp.run_forever

    Are you sure that the pings are exchanged? When I simulate my internet connection going down, I do not get any timeouts from the websocket for at least 30 minutes.

  • @bartwozniak,

    The TR_JSON2 websocket protocol defines its own ping/pong messages and not the ping as defined in the Websocket protocol specification.

    Here is a raw package of the above request which shows the streaming messages and the pings:

    image.png
  • I understand, thank you. What is the timeout on the ping set to and can I make it shorter than 30 minutes I observed? I would like the library to notify me faster when there is a problem with the connection.

  • Hello @bartwozniak

    The ping timeout can be adjustment on the server side (ADS on RTO) only.

    pingTimeout.png

    Alternatively, you may consider switch to the core https://developers.lseg.com/en/api-catalog/real-time-opnsrc/websocket-api which is a WebSocket transport layer of the https://developers.lseg.com/en/api-catalog/lseg-data-platform/lseg-data-library-for-python . However, you need to implement everything such as connection management, session management, authentication, subscription management, etc.) by yourself with the WebSocket API. It means you can get all status and handle them based on your preference.

    The Python WebSocket API for RTO connection examples are available at https://github.com/Refinitiv/websocket-api/tree/master/Applications/Examples/RTO/python page.

  • Thank you for extra info.

    Where exactly is this configuration for ADS? I can't find it.

    So the server will timeout, but what about my client application if I get disconnected from the internet? I don't think the client will know as I have verified experimentally. Is that expected?

  • @bartwozniak

    Websockets is based on TCP protocol and if your machine disconnects from internet, the application will immediately know about it. The ping/pong message handshake happens every 30 seconds, and is used when the traffic to server is disrupted and or there is network contention. I think default is 3 missed ping messages = connection loss.

    Are you connecting to a local ADS? The code that you show in question is getting data from RTO in the cloud!

  • @Gurpreet I am using the exact same code I showed in the first post, hence it is connecting to a AWS endpoint. I appreciate all the info on the underlying technology, however, I am still failing to make it crash when internet connection is broken. Do you have any tips for how I can reproduce your result? My process keeps on hanging for a very long time when connection is broken.

  • Hello @bartwozniak

    Please enable debugging in the library and attach the logs. We can try to investigate what might be going on.

  • I disabled the network interface on my device after the last log message and killed the processes 15 minutes later.

  • Killing the connection or your router is a better way since it simulates the real world connection loss - giving the library a chance to try this or another server again.

    However, when the networking is disabled on the machine, I see that no error is generated. I can raise this with library developer to see if there is a fix for this.

  • That will be most useful, thank you @Gurpreet . Let me know what they say and thanks for all the insights.

  • pf
    pf LSEG
    edited March 25

    Hi @bartwozniak ,

    To reply to your initial question "How to best handle connection errors in Python?", especially if you want to trace and manage disconnection, you can find how to implement session event and state notifications in LSEG Data Library for Python - Reference Guide : Callback functions section.
    on_event callback will notify on following SessionEventCode:

    Event code

    Description

    SessionEventCode.StreamConnecting

    The connection to the stream service within the session is pending.

    SessionEventCode.StreamConnected

    The connection to the stream service has been successfully established.

    SessionEventCode.StreamDisconnected

    The connection to the stream service is not established.

    SessionEventCode.StreamAuthenticationSuccess

    The session has successfully authenticated this client.

    SessionEventCode.StreamAuthenticationFailed

    The session has failed to authenticate this client.

    SessionEventCode.SessionConnecting

    The session is establishing the connections.

    SessionEventCode.SessionConnected

    The client successfully connected with the current session.

    SessionEventCode.SessionDisconnected

    The session disconnected the client.

    SessionEventCode.SessionAuthenticationSuccess

    The session has successfully authenticated this client.

    SessionEventCode.SessionAuthenticationFailed

    The session has failed to authenticate this client.

    SessionEventCode.SessionReconnecting

    The session is currently establishing the connection.

    SessionEventCode.DataRequestOk

    The request for content from the session data services has completed successfully.

    SessionEventCode.DataRequestFailed

    The request for content from the session data services has failed.

    At PricingStream level, you can also manage notifications. Full example is described in TUT_2.2.03-Pricing-StreamingEvents.ipynb from tutorials.

    Regarding ping/pong messages at websocket level, it's automatically manage by websocket-client lib as part of the protocol.
    LD doesn't do anything except that logging the event with on_ping & on_pong callbacks:

    self._listener = WebSocketClient(
        url=self._config.url,
        header=headers,
        cookie=cookie,
        on_open=self._on_ws_open,
        on_message=self._on_message,
        on_error=self._on_ws_error,
        on_close=self._on_ws_close,
        on_ping=self._on_ws_ping,
        on_pong=self._on_ws_pong,
        subprotocols=subprotocols
    )

    As @Gurpreet mentionned, TR_JSON WebSocket protocol has its own ping/pong messages (which are different than previous), and the timeout is driven by the server. LD is implemented to reply to received ping message to keep streaming connection up. The log you uploaded contains related received ping/sent pong messages.

Welcome!

It looks like you're new here. Sign in or register to get started.

Welcome!

It looks like you're new here. Sign in or register to get started.