Java RFA callback on disconnect

Hello @daniel.lipofsky ,

In addition to registering interest for item events, you may wish to register interest in connection and error events, for example:

 OMMErrorIntSpec ommErrorIntSpec = new OMMErrorIntSpec();
 OMMConnectionIntSpec ommConnectionIntSpec = new OMMConnectionIntSpec();

 Handle errHandle = _mainApp.getOMMConsumer().registerClient(_mainApp.getEventQueue(),
                ommErrorIntSpec, this, null);
 Handle connHandle = _mainApp.getOMMConsumer().registerClient(_mainApp.getEventQueue(),
                ommConnectionIntSpec, this, null);

Trigger a test disconnect, to verify if this is what you are looking for?

This doesn't appear to get me what I want. While I am getting OMMConnectionEventMsg that I never saw before, the timestamps in the log match the reconnects, not the disconnects. I am still not getting anything at all for the disconnects. The OMMConnectionEventMsg that I am getting look like:

OMM_CONNECTION_EVENT, connection name = SSLNamespace::pageReutersSSLConn, ConnectionType = RSSL, ConnectedHostName = 159.220.230.146, ConnectedHostPort = 14002, receive address = , receive port = , send address = , send port = , unicast port = , ConnectedComponentVersion = ads3.5.0.L1.linux.rrg 64-bit

Hi @daniel.lipofsky ,

It may be helpful to learn how do you test this failure on your side?

The way I test this on my side, is I break off my connection to infra (ADS) by forcing a disconnect (disconnecting VPN or WiFi). I see the status reported on the disconnect.

Thinking further of this, are you connecting to your infra directly, or via LPC? If by way of LPC, how do you trigger the issue for testing?

Hello @daniel.lipofsky ,

Thanks for this information. Something in this behavior in AWS is not exactly right, from what I understand.

Is there another way to force a disconnect in AWS during your test, so that we can both be sure that what you see is a disconnect, as well as fully verify the handling of it?

If what you detect is a loss of connectivity, I would expect the behavior of a custom app to be the same as that of a simple example StarterConsumer that comes with RFA Java SDK and that I use to do some quick testing. On connectivity loss:

...
ItemManager.processEvent: Received Item Event... MESSAGE Msg Type: MsgType.STATUS_RESP Msg Model Type: MARKET_PRICE Indication Flags: Hint Flags: HAS_ATTRIB_INFO | HAS_STATE State: OPEN, SUSPECT, NO_RESOURCES,  "An existing connection was forcibly close d by the remote host" AttribInfo ServiceName: ELEKTRON_DD ServiceId: 356 Name: MSFT.O NameType: 1 (RIC) Payload: None ItemManager.processEvent: Received Item Event... MESSAGE Msg Type: MsgType.STATUS_RESP Msg Model Type: MARKET_PRICE Indication Flags: Hint Flags: HAS_ATTRIB_INFO | HAS_STATE State: OPEN, SUSPECT, NONE,  "Waiting for service ELEKTRON_DD UP. Item recovery in progress..." AttribInfo ServiceName: ELEKTRON_DD Name: MSFT.O NameType: 1 (RIC) Payload: None LoginClient: Receive an OMM_CONNECTION_EVENT Name: myNS::connection28 Status: { state: DOWN, code: NONE, text: "An existing connection was forcibly cl osed by the remote host"}
...

- see status callback trigger, on connection loss, both on open item stream, and on open login stream.

I would try to verify the behavior with StarterConsumer running next to your custom app in the same AWS instance, you would need to be able to force a disconnect for a quick sanity check test like this.

The other approach is to sort out failure conditions by testing your application on a testbed where you can simulate connectivity loss. If the loss is not detected, test with StarterConsumer, make sure the loss is detected and move to the custom app.

Additionally, it may be helpful to enable RFA tracing, to learn more of . See this previous discussion thread -> answer by Pimchaya on how to enable RFA tracing.

Hope this information helps.

I don't know any way to force disconnects in the AWS environment, and I only get disconnects with a large watchlist. Turning on full tracing gives a rather huge file - I've done it an I have a 54 GB file to show for it, but really no idea what to do with that file. But in any case the goal is to be able to detect disconnect within the app, so an external log file is not useful.

But I do have a minimal log (attached: trace.txt) that shows the disconnects. You can see "Connection reset by peer" in this file line 49 and 97, e.g.

<record>
  <date>2021-10-27T19:34:07.235645Z</date>
  <millis>1635363247235</millis>
  <nanos>645000</nanos>
  <sequence>2</sequence>
  <logger>com.reuters.rsslc.0</logger>
  <level>FINER</level>
  <class>com.reuters.ipc.TraceLogger</class>
  <method>traceData</method>
  <thread>27</thread>
  <message>
Thread: SSLNamespace::pageReutersSSLSession Session EventQueueGroup
Connection 0
RSSL Connection failed for 159.220.230.146: Connection reset by peer
</message>
</record>

Hello @daniel.lipofsky ,

Please be assured that I am trying to help you to investigate the suspected issue, and to come up with the troubleshooting approach that will lead to the solution, within the framework of the discussion forums.

---

Let us start at the point, that disconnects should be detected by an RFA consumer, and reported as status, via callback. In this case, from your report, this is not happening with a custom consumer on a large watchlist.

Without the reliable way to reproduce and fully understand the cause of the issue, there is no way to the solution, as I am sure you know. Further, to support the production app, you will needs to have a way to test going forward, if and when needed, so this is a continuous requirement, rather then a once-off.

---

Please confirm, has the issue only started to manifest, as the watchlist grew (this is what I am guessing)?

Please confirm, was the application, prior to the deployment into AWS, fully tested for failures and error conditions on your dev testbed?

* If this is the case, going back to dev testbed, retesting, and understand when and why the status on disconnect does not manifest properly can be the best and most efficient course of action.

* An alternative approach would be to consider migrating to EMA API, as a strategic option, and investing time into designing a solution in EMA. A new article: Migrating from Refinitiv Legacy to Strategic APIs can be of help. To deliver a consumer application ready for production, we need a development environment where functionality, performance and failure conditions can be fully tested, where we design for robustness from the start.

If you think this could be helpful, please contact your account manager, and let's setup to discuss this offline.

@zoya faberov , to be clear, we very much want to go EMA but we have been waiting weeks for Refinitiv to provide us with IDs to access EMA, and it may be many more weeks (months?) before we get them. We can't even get an ETA on when we'll get them. In the mean time, we are having multiple production outages per day and are trying to find a solution.

We only see this with larger watch lists. We have dev and uat clusters we can test on. We are seeing the disconnects there too. What I need is suggestions for things to try.

Totally understand, and trying to help.

Are you in control of the connectivity for the purposes of testing, in dev/uat environment?

Would it make sense to you to test with your SDK version/flavor StarterConsumer, trigger disconnect, makes sure it is received and processed by your Starter? I feel that you should try to start with the working basics (working Starter, detecting disconnects) and move up to your requirements, would this make sense?

Otherwise, I would suggest to consider the same test, but starting by running a test with StarterConsumer, running it on your local developer machine, pointing it for testing to the same infra, is there a connectivity from your local developer machine to your infra, otherwise, to request the route with your local network group/admin. Would it make sense to use it as a starter environment to work on the connectivity issue from the ground up, as you are in control of the connectivity on it?

I am looking to start with something that is proven to work, and easily verifiable at every next step building up, with respect to this specific suspected issue.

I was able to use a mock server, then kill that server, to simulate that type of disconnect. There was no callback in that case either.

yes, that would probably be helpful. I am here for a few more hours today, then out on vacation Fri-Tue.

wasin.w

Could you please confirm if you see the "Contact premium support" on the RFA Java API (https://developers.refinitiv.com/en/api-catalog/refinitiv-real-time/robust-foundation-api-rfa-java) page? If so, you can click that button to submit a ticket to the RFA team directly.

Additionally, please give us the following details that can help us understand your environment better:

The Java SDK/JRE version
The OS/Platform version
The snippet code that registers and handles the OMMConnectionIntSpec

wasin.w

Please give us the following details that can help us understand your environment better:

The Java SDK/JRE version
The OS/Platform version
The snippet code that registers and handles the OMMConnectionIntSpec

How many EventQueue that the application is using? If you are using only EventQueue, all messages including the OMM connection event messages might not be dispatched to the application. You may separate the items and OMM connection events to different EventQueues and then use separate threads to dispatch them.

wasin.w