Frequent Tunnel stream disconnection using EMA Java SDK 3.4.0.1 which is causing our service to p...

...ublish prices when the service starts and then break.

ced7e6a25126     ClientName: ChannelCallbackClient

ced7e6a25126     Severity: Info

ced7e6a25126     Text:    Received ChannelUp event on channel BNC_Channel_1

ced7e6a25126     Instance Name BNC_Consumer_1

ced7e6a25126     Component Version eta3.1.1.L1.linux.rrg 64-bit Static

ced7e6a25126 loggerMsgEnd

ced7e6a25126 2020-02-11 23:04:10.530  INFO 1 --- [pool-2-thread-1] c.bnc.refinitiv.client.RefinitivClient   : ----- Status message ----

ced7e6a25126 2020-02-11 23:04:10.546  INFO 1 --- [pool-2-thread-1] c.bnc.refinitiv.client.RefinitivClient   : StatusMsg received, preparing to perform client login... tunnelStreamHandle=1, streamId=5

ced7e6a25126 2020-02-11 23:04:13.187  INFO 1 --- [pool-2-thread-1] c.bnc.refinitiv.client.RefinitivClient   : ----- Refresh message ----

ced7e6a25126 2020-02-11 23:04:13.197  INFO 1 --- [pool-2-thread-1] c.bnc.refinitiv.client.RefinitivClient   : RefreshMsg received, starting to post data... streamId=5, streamState=Open / Ok / None / 'Login accepted by host c684yvpspcb11'

ced7e6a25126 2020-02-11 23:04:19.726  INFO 1 --- [           main] o.s.s.d.d.r.s.DynamoDBRepositoryFactory  : Spring Data DynamoDB Version: 5.0.4 (2.0)

ced7e6a25126 2020-02-11 23:04:19.734  INFO 1 --- [           main] o.s.s.d.d.r.s.DynamoDBRepositoryFactory  : Spring Data Version:          2.1.11.RELEASE

ced7e6a25126 2020-02-11 23:04:19.736  INFO 1 --- [           main] o.s.s.d.d.r.s.DynamoDBRepositoryFactory  : AWS SDK Version:              1.11.301

ced7e6a25126 2020-02-11 23:04:19.737  INFO 1 --- [           main] o.s.s.d.d.r.s.DynamoDBRepositoryFactory  : Java Version:                 11.0.6 - OpenJDK 64-Bit Server VM 11.0.6+10

ced7e6a25126 2020-02-11 23:04:19.737  INFO 1 --- [           main] o.s.s.d.d.r.s.DynamoDBRepositoryFactory  : Platform Details:             Linux 4.14.97-74.72.amzn1.x86_64

ced7e6a25126 2020-02-11 23:04:19.737  WARN 1 --- [           main] o.s.s.d.d.r.s.DynamoDBRepositoryFactory  : This Spring Data DynamoDB implementation might not be compatible with the available Spring Data classes on the classpath!

ced7e6a25126 NoDefClassFoundExceptions or similar might occur!

ced7e6a25126 2020-02-11 23:04:23.271  INFO 1 --- [           main] o.h.h.i.QueryTranslatorFactoryInitiator  : HHH000397: Using ASTQueryTranslatorFactory

ced7e6a25126 2020-02-11 23:04:25.297  INFO 1 --- [           main] o.h.h.i.QueryTranslatorFactoryInitiator  : HHH000397: Using ASTQueryTranslatorFactory

ced7e6a25126 2020-02-11 23:04:25.624  INFO 1 --- [           main] o.h.h.i.QueryTranslatorFactoryInitiator  : HHH000397: Using ASTQueryTranslatorFactory

ced7e6a25126 2020-02-11 23:04:31.240  INFO 1 --- [           main] o.s.s.concurrent.ThreadPoolTaskExecutor  : Initializing ExecutorService 'applicationTaskExecutor'

ced7e6a25126 2020-02-11 23:04:38.825  INFO 1 --- [           main] o.s.s.c.ThreadPoolTaskScheduler          : Initializing ExecutorService 'taskScheduler'

ced7e6a25126 2020-02-11 23:04:38.863  INFO 1 --- [           main] o.s.b.a.e.web.EndpointLinksResolver      : Exposing 15 endpoint(s) beneath base path '/actuator'

ced7e6a25126 2020-02-11 23:04:40.610  INFO 1 --- [           main] o.s.b.w.embedded.tomcat.TomcatWebServer  : Tomcat started on port(s): 8080 (http) with context path ''

ced7e6a25126 2020-02-11 23:04:40.658  INFO 1 --- [           main] com.bnc.refinitiv.Application            : Started Application in 184.101 seconds (JVM running for 198.668)

ced7e6a25126 2020-02-11 23:04:41.348  INFO 1 --- [nio-8080-exec-1] o.a.c.c.C.[Tomcat].[localhost].[/]       : Initializing Spring DispatcherServlet 'dispatcherServlet'

ced7e6a25126 2020-02-11 23:04:41.350  INFO 1 --- [nio-8080-exec-1] o.s.web.servlet.DispatcherServlet        : Initializing Servlet 'dispatcherServlet'

ced7e6a25126 2020-02-11 23:04:41.462  INFO 1 --- [nio-8080-exec-1] o.s.web.servlet.DispatcherServlet        : Completed initialization in 103 ms

ced7e6a25126 2020-02-11 23:06:00.104  INFO 1 --- [   scheduling-1] c.bnc.refinitiv.client.RefinitivClient   : Sent price ric="TXRPINX=BNC", symbol="XRPLX", price="0.2808085", timestamp="2020-02-11T23:00:00Z"

ced7e6a25126 2020-02-11 23:06:00.155  INFO 1 --- [   scheduling-1] c.bnc.refinitiv.client.RefinitivClient   : Sent price ric="TELXINX=BNC", symbol="ELX", price="237.4026572", timestamp="2020-02-11T23:00:00Z"

ced7e6a25126 2020-02-11 23:06:00.183  INFO 1 --- [   scheduling-1] c.bnc.refinitiv.client.RefinitivClient   : Sent price ric="TBTCINX=BNC", symbol="BLX", price="10230.7947016", timestamp="2020-02-11T23:00:00Z"

ced7e6a25126 2020-02-11 23:06:00.392  INFO 1 --- [pool-2-thread-1] c.bnc.refinitiv.client.RefinitivClient   : ----- Ack message ----

ced7e6a25126 2020-02-11 23:06:00.394  INFO 1 --- [pool-2-thread-1] c.bnc.refinitiv.client.RefinitivClient   : ----- Status message ----

ced7e6a25126 2020-02-11 23:06:00.394  WARN 1 --- [pool-2-thread-1] c.bnc.refinitiv.client.RefinitivClient   : Could not open tunnel stream expectedTunnelHandle=1, eventTunnelHandle=1, streamState=3

5f80c1e6f26a     ClientName: ItemCallbackClient

5f80c1e6f26a     Severity: Error

5f80c1e6f26a     Text:    Attempt to use invalid Handle on submit(PostMsg). Instance name='BNC_Consumer_1'.

5f80c1e6f26a loggerMsgEnd

Find more posts tagged with

elektron

OMM

elektron-sdk

refinitiv-realtime

rcc-api

tunnel-stream

Accepted answers

veerapath.rungruengrayubkul

Hi @joel30,

Normally, the Tunnel Stream will be closed once connection go down. There is no recovery on tunnel stream connection, so application has to create a new Tunnel Stream once the connection is recovered.

However, with current EMA version, 3.4.0, application may receive the 'Service not available', when it immediately re-create Tunnel Stream after the connection is recovered (Login Open/Ok received). This is because the service is not available immediately after the connection is recovered.

The current workaround is to register for Directory event to ensure that the service is immediately when the application tries to re-create Tunnel Stream.

Below are the summary instructions to recover the tunnel stream.
1. Application register for Login event to monitor connection and also register for Directory event to monitor when source directory become available.

LoginReq loginReq = EmaFactory.Domain.createLoginReq();consumer.registerClient(loginReq.message(), appClient);
ReqMsg reqMsg = EmaFactory.createReqMsg();consumer.registerClient(reqMsg.domainType(EmaRdm.MMT_DIRECTORY).serviceName("DDS_TRCE"), appClient);

2. Once Connection down, application will receive Login Status: Suspect/Open indicating that connection is down. Application will also receive System Status: Closed Recoverable/Suspect indicating that Tunnel Stream is closed.

----- Status message ----
StatusMsg
streamId="1"
domain="Login Domain"
state="Open / Suspect / None / ''"
StatusMsgEnd

----- Status message ----
StatusMsg
streamId="6"
domain="System Domain"
privateStream
state="Closed, Recoverable / Suspect / None / 'channel down.'"
name="TUNNEL1"
serviceId="10"
serviceName="DDS_TRCE"
StatusMsgEnd

3. EMA will automatically reconnect to the server. Once the connection is established, application will receive Login Refresh: Open/Ok indicating that connection is up.

----- Refresh message ----
RefreshMsg
streamId="1"
domain="Login Domain"
solicited
RefreshComplete
state="Open / Ok / None / ''"

    public void onRefreshMsg(RefreshMsg refreshMsg, OmmConsumerEvent event)    { 
        if(refreshMsg.domainType() == EmaRdm.MMT_LOGIN && refreshMsg.state().streamState() == OmmState.StreamState.OPEN && refreshMsg.state().dataState() == OmmState.DataState.OK)         {             
// Connect is up
connectionUp=true;         
}

4. After that, application needs to wait for Directory Update event received, and then re-create a new Tunnel Stream and continue posting data on the created Tunnel Stream.

    public void onUpdateMsg(UpdateMsg updateMsg, OmmConsumerEvent event)    {         if(updateMsg.domainType() == EmaRdm.MMT_DIRECTORY && connectionUp == true)         {             
           System.out.println("Starting tunnel stream...");             
           ClassOfService cos = EmaFactory.createClassOfService()....

All comments

pimchaya.wongrukun01

Hello @joel30

Could you try EMA Consumer - Posting data to TR Contribution Channel tutorial source code? This is to verify if the server, service, your account credential, RICs are correct and ready or not.

If you can post data using the tutorial source code, I suggest you reach out to the development team for further investigation for the application and EMA. You can submit your problem to the development team directly via Refinitiv/Elektron-SDK Github then click "New issue" button.
If you cannot post data using the tutorial source code, please contact Refinitv account team to verify the server, service, your account credential and RICs.

Jirapongse

@joel30

You may need to set logging to the FINEST level to see more information.

For example, create a new logging.properties file with this content.

#This file contains log configuration for java logging API.

# Level mapping between jdk and slf4j logging
# jdk.util.logging     SLF4J 
# FINEST              -> ALL
# FINEST               -> DEBUG
# FINEST               -> TRACE
# INFO                -> INFO
# WARNING             -> WARN
# SEVERE              -> ERROR

.level=FINEST
 
#handlers=java.util.logging.FileHandler, java.util.logging.ConsoleHandler
handlers=java.util.logging.FileHandler

#java.util.logging.ConsoleHandler.level=FINEST
#java.util.logging.ConsoleHandler.formatter=java.util.logging.SimpleFormatter
 
java.util.logging.FileHandler.level=FINEST
java.util.logging.FileHandler.pattern=./emaj.log
 
# Write 100000 bytes before rotating this file
java.util.logging.FileHandler.limit=50000000
 
# Number of rotating files to be used
java.util.logging.FileHandler.count=20
java.util.logging.FileHandler.formatter=java.util.logging.SimpleFormatter
# Format timestamp as date/time with millisecond
#java.util.logging.SimpleFormatter.format=%1$tY-%1$tm-%1$td %1$tH:%1$tM:%1$tS.%1$tL %4$-7s %2$s %n%5$s

Then, run the application with the following VM argument.

-Djava.util.logging.config.file=logging.properties

The emaj.log file will be created.

For more information regarding logging, please refer to section Troubleshooting in this tutorial.

joel30

I have created an issue on Refinitiv Github handle regarding this. I tried with the sample ema consumer for posting data to TRCC plugging in our configuration and it worked fine.

Jirapongse

@joel30

I found the following messages in the emaj.log file.

2020-02-19 11:21:02.153  INFO 317320 --- [pool-2-thread-1] c.bnc.refinitiv.client.RefinitivClient   : ----- Status message ----
2020-02-19 11:21:02.153  WARN 317320 --- [pool-2-thread-1] c.bnc.refinitiv.client.RefinitivClient   : Could not open tunnel stream expectedTunnelHandle=1, eventTunnelHandle=1, streamState=3
2020-02-19 11:21:02.153 TRACE 317320 --- [pool-2-thread-1] c.t.ema.access.OmmConsumerImpl           : loggerMsg
    ClientName: ItemCallbackClient
    Severity: Trace
    Text:    Removed Item 2 from item map
    Instance name BNC_Consumer_1
loggerMsgEnd


2020-02-19 11:21:02.153 TRACE 317320 --- [pool-2-thread-1] c.t.ema.access.OmmConsumerImpl           : loggerMsg
    ClientName: ItemCallbackClient
    Severity: Trace
    Text:    Removed Item 1 of StreamId 5 from item map
    Instance name BNC_Consumer_1
loggerMsgEnd

From the messages, the tunnel stream has been closed with streamState=3 (CLOSED_RECOVER). It may be closed due to disconnection. You can register for MMT_LOGIN to monitor the connection status or contact the server team to verify the server log for disconnections. To register to MMT_LOGIN, please refer to this thread.

Therefore, when the tunnel stream has been closed, the application needs to re-establish the tunnel stream before posting data.

joel30

I get the below error from the service logs:

StatusMsg

streamId="6"

domain="System Domain"

privateStream

state="Closed, Recoverable / Suspect / None / 'Service not available'"

name="BNC_Tunnel"

serviceId="10"

serviceName="DDS_TRCE"

StatusMsgEnd

pimchaya.wongrukun01

Hello @joel30

Service not available may cause from the service goes down. Please contact Refinitiv account team who should help you to contact the server team to verify the service.

veerapath.rungruengrayubkul

Hi @joel30,

The current workaround is to register for Directory event to ensure that the service is immediately when the application tries to re-create Tunnel Stream.

LoginReq loginReq = EmaFactory.Domain.createLoginReq();consumer.registerClient(loginReq.message(), appClient);
ReqMsg reqMsg = EmaFactory.createReqMsg();consumer.registerClient(reqMsg.domainType(EmaRdm.MMT_DIRECTORY).serviceName("DDS_TRCE"), appClient);

----- Status message ----
StatusMsg
streamId="1"
domain="Login Domain"
state="Open / Suspect / None / ''"
StatusMsgEnd

----- Status message ----
StatusMsg
streamId="6"
domain="System Domain"
privateStream
state="Closed, Recoverable / Suspect / None / 'channel down.'"
name="TUNNEL1"
serviceId="10"
serviceName="DDS_TRCE"
StatusMsgEnd

3. EMA will automatically reconnect to the server. Once the connection is established, application will receive Login Refresh: Open/Ok indicating that connection is up.

----- Refresh message ----
RefreshMsg
streamId="1"
domain="Login Domain"
solicited
RefreshComplete
state="Open / Ok / None / ''"

    public void onRefreshMsg(RefreshMsg refreshMsg, OmmConsumerEvent event)    { 
        if(refreshMsg.domainType() == EmaRdm.MMT_LOGIN && refreshMsg.state().streamState() == OmmState.StreamState.OPEN && refreshMsg.state().dataState() == OmmState.DataState.OK)         {             
// Connect is up
connectionUp=true;         
}

4. After that, application needs to wait for Directory Update event received, and then re-create a new Tunnel Stream and continue posting data on the created Tunnel Stream.

    public void onUpdateMsg(UpdateMsg updateMsg, OmmConsumerEvent event)    {         if(updateMsg.domainType() == EmaRdm.MMT_DIRECTORY && connectionUp == true)         {             
           System.out.println("Starting tunnel stream...");             
           ClassOfService cos = EmaFactory.createClassOfService()....

joel30

Thanks for the response. I have implemented this logic in my codebase.

EXPLORE OUR SITES