Our client is subscribing to 12k items from Refinitiv Real-Time Optimized (RTO) datafeed using EMA C++. They are running into the following issue:
We saw the following status messages for all the subscribed rics on 06/14 around 04:16 NYT and after these messages we did not get any more update/refresh messages till very later in the day.
2021-06-15T08:16:01.145Z INFO OmmClient::onStatusMsg grs_ommclient.cpp:110 Item Name : USDDKKSWFIXMP=WM Item State : Closed / Suspect / None / 'Login stream was closed.' 2021-06-15T08:16:01.145Z INFO OmmClient::onStatusMsg grs_ommclient.cpp:110 Item Name : USDDKKSWFIXP=WM Item State : Closed / Suspect / None / 'Login stream was closed.' 2021-06-15T08:16:01.145Z INFO OmmClient::onStatusMsg grs_ommclient.cpp:110 Item Name : USDDKKTNFIXMP=WM Item State : Closed / Suspect / None / 'Login stream was closed.' 2021-06-15T08:16:01.145Z INFO OmmClient::onStatusMsg grs_ommclient.cpp:110 Item Name : USDDKKTNFIXP=WM Item State : Closed / Suspect / None / 'Login stream was closed.' 2021-06-15T08:16:01.145Z INFO OmmClient::onStatusMsg grs_ommclient.cpp:110 Item Name : USDEEK1MFIXMP=WM Item State : Closed / Suspect / None / 'Login stream was closed.'
We had to restart our application to start processing messages from Refinitiv again.
The given log and status messages are mostly identical to this thread which indicates that the server-side disconnected your application.
Regarding the "we are bringing two services simultaneously that connect to refinitiv using the same credentials. ", do you mean you are using one Machine-ID credential for two sessions (or two instances)? If so, that might be the reason for the "TREP Authentication token has expired." error which causes the OMM Login stream to disconnected.
I suggest you use a unique Machine-ID for each instance (please contact your Refinitiv Representative/Account Manager on this).
If the same disconnection issue is still occurring (with one Machine-ID/one session or two Machine-IDs for two sessions), you may need to contact the RTO Support to investigate on the server-side by submitting a ticket via https://my.refinitiv.com/content/mytr/en/productsupport.html website, and select "Product" Refinitiv Real-Time - Optimized (please give the Machine-ID and time detail in the description too).
The "Login Stream was Closed" error message with "Closed / Suspect" status means the login steam is closed, so this item stream is closed too and the API will not recover the subscription for the application.
The issue can occur for various reasons such as the disconnection between API and server, the slow consumer, etc. Are there any error messages about the disconnection or login stream?
Since the application was logged out from the server, so all open item streams were also closed off.
Is the application using
Are they using ChannelSets to specify multiple endpoints or locations in their config?
See section 'Using ChannelSets with Refinitiv Real-Time Optimized Cloud-based Services' in the following article Enterprise Message API (EMA) - Configuration Overview | Refinitiv Developers
Thank you for looking into this issue, We saw this issue again yesterday evening 15:57 NYT.
Do you know why was the application logged off from the server ? Would be great if. you can put some scenarios in which the application can be logged off from the server, and how can the application best recover from such a state ? Also does the API automatically reconnect back in case of a TCP DISCONNECTIONs ?
I did not see any disconnect logs ,I saw the following log
Please find below the answer to your questions( answers start with a ->)
the session management feature to allow the API to select an endpoint in their region (example 113_MP_SessionMgmt),
-> We are closely following example 113 from the EMA examples, and use "Consumer_4" from the EMAConfig.xml as consumer type. My understanding based on EMA C++ guide section 6.7 is setting the consumerType to "Consumer_4" in OmmConsumerConfig enables the session management use case. I expect that only by setting consumer type in OmmConsmerConfig object to "Consumer_4" the API should take care of the following by default
• Obtains a token (according to the details in Section 6.3.2)
• Queries service discovery (according to the details in Section 6.4)
• Consumes market data (according to the details in Section 6.5)
• Manages login reissues when needed on a cyclical basis (according to the details in Section 6.3.3
Reconnects back on Disconnections
or specific RTO endpoint(s) in their config
-> We don't connect to any specific end point, but use "Channel_4" configuration from EMAConfig.xml
or using the Service Discovery option to select an endpoint (demonstrated in example 450_MP_QueryServiceDiscovery)
-> My understanding is setting OmmConsumerConfig.channelType to "Channel_4" enables session management which takes care of service discovery.
Please let us know if you need anything more from our side for carrying out your investigations.
Thanks again for helping out.
Thank you for the information. Based on the given error messages, the "TREP authentication token has expired." means the RDP authentication between the client and RTO Real-Time streaming server was expired, so the Login stream was closed.
Basically, the EMA automatically takes care of refreshing the Access Token with RDP Auth Service (via HTTP REST) before the session expired.
What is the version of RTSDK/EMA C+++ that you are using? I suggest you test with the latest version of the SDK. The latest version of the SDK is 2.0.1.L2 (EMA 6.3.1L2) You can download the SDK from the following links:
The other way is to enable the REST log messages between the API and RDP Auth service to verify the API behavior and RDP session. You can configure it via the following parameters in EmaConfig.xml file. Please remove your credentials before sharing the log file.
Thankyou @wasin.waeosri for the response,
Yes we are using the latest version of the API for linux ( which is Real-Time-SDK-2.0.1.L2.linux)
I have enabled the rest logs and would let you know when it happens again. Please note it has happened 3-4 times in the past week.
The OmmConsumer stopping to retry to connect to tokenservice after 2 timeouts is something concerning for us. We would want to recover from this state automatically if possible.
Trying to understand what are the options available for the client to recover automatically from this state.
I see the following three options, Please comment if they make sense, or if there is a better way of handling this.
-> Would increasing the RestRequestTimeout from 90 to a higher number help ?
-> Can the client resubscribe for the securities for which the stream was closed ? To be more specific invoke call OmmConsumer::registerClient() for expired securities on the existing OmmConsumer instance ?
-> Should the client create a new instance of OmmConsumer and resubscribe for all the securities ?
You should try to increase the value of RestRequestTimeout first. Please note that there is no "golden number", so you may need to adjust the value, test, and then repeat the adjust-test steps to find the number that matches your environment.
The other parameter that may help you is the TokenReissueRation.
Regarding the subscription, you need to re-initialize the connection with the EmaFactory.createOmmConsumer() statement before re-subscribe items with registerClient() function.
Thank you @wasin.waeosri I will try to tweak those parameters.
We again encountered some errors and this time we saw a different message ' Recoverable / Suspect / None / 'Service for this item was lost.' This looks like a different issue to me, and was resulted from a network issue.
Below is the log
Wanted to bring to your notice that we are bringing two services simultaneously that connect to refinitiv using the same credentials. Do you think that might be the reason for this behaviour ?
Also how are the disconnects handled in general by the API ?