TREP resilient issue: 2 type of error, 1 successful, 1 failed to failover to next channel.
We are using Ema-3.3.1.0.jar in our TREP application to get live/delay feed. We setup 4 channels in our EmaConfig.xml. However, we notice that the resiliency failed on some scenario.
Scenario 1:
Error Message: A21: Access Denied. Waiting to connect to DACS server. Try request again.
Result: Resiliency works, the application able to failover to next channels and re-establish the connection
Scenario 2:
Error Message: A21: Access Denied. Timed out waiting for response from DACS server. Try request again.
Result: Resiliency failed, the application failed to failover to next channel and the application lost connection to TREP
May I know why the resiliency fail for second Scenario?
Hi @fuweng.ng
As explained in my earlier post, the ADS is the one that interacts with the DACS server - not your application. Your application only talks to the ADS.
Therefore, you need to speak to your Market Data team and/or DACS administrators to resolve this issue. If they are uncertain how to resolve they should raise a My.Refinitiv ticket for technical support for the ADS / DACS issue.
This is the log after 1st Scenario:
2020-11-28 21:12:30.005 WARN [pool-2-thread-1] OmmConsumerImpl - loggerMsg
ClientName: ChannelCallbackClient
Severity: Warning
Text: Received ChannelDownReconnecting event on channel Channel_3
RsslReactor @59dbf726
RsslChannel @571c126d
Error Id -1
Internal sysError 0
Error Location WlLoginHandler.readStatusMsg
Error text Received login response with Closed/Recover stream state. Disconnecting.
loggerMsgEnd
2020-11-28 21:12:30.822 INFO [Thread-0] ObjectStatReport - >> Program_statistics start:
2020-11-28 21:12:30.822 INFO [Thread-0] ObjectStatReport - >> UpdateSubMgr statistics: Subscriptions=0 Subscribers=0
UpdateSubMgr statiistics end. <<
2020-11-28 21:12:30.822 INFO [Thread-0] ObjectStatReport - Pub_i statistics :Pub_i statiistics end
2020-11-28 21:12:30.822 INFO [Thread-0] ObjectStatReport - Program_statistics end. <<
2020-11-28 21:12:34.023 INFO [pool-2-thread-1] OmmConsumerImpl - loggerMsg
ClientName: ChannelCallbackClient
Severity: Info
Text: Received ChannelUp event on channel Channel_4
Instance Name Consumer_1_1
Component Version ads3.3.1.L1.linux.tis.rrg 64-bit
loggerMsgEnd
This is the log after 2nd scenario:
2020-11-28 18:12:27.640 ERROR [pool-2-thread-1] RTROmmConsumer - [registerClient] Item: app_eqtrade / Service: <not set> [State: Closed / Suspect / Timeout / 'A21: Access Denied. Timed out waiting for response from DACS server. Try request again.']
2020-11-28 18:12:27.640 WARN [pool-2-thread-1] RTROmmConsumer - No. connection failed detected :2
2020-11-28 18:12:56.907 INFO [Thread-0] ObjectStatReport - >> Program_statistics start:
2020-11-28 18:12:56.907 INFO [Thread-0] ObjectStatReport - >> UpdateSubMgr statistics: Subscriptions=0 Subscribers=0
UpdateSubMgr statiistics end. <<
2020-11-28 18:12:56.907 INFO [Thread-0] ObjectStatReport - Pub_i statistics :Pub_i statiistics end
2020-11-28 18:12:56.907 INFO [Thread-0] ObjectStatReport - Program_statistics end. <<
2020-11-28 18:12:58.202 INFO [Thread-7] SL_SubMgr - proUnsub starting check
2020-11-28 18:12:58.202 INFO [Thread-7] SL_SubMgr - proUnsub check end. Count=0
2020-11-28 18:13:26.907 INFO [Thread-0] ObjectStatReport - >> Program_statistics start:
2020-11-28 18:13:26.907 INFO [Thread-0] ObjectStatReport - >> UpdateSubMgr statistics: Subscriptions=0 Subscribers=0
UpdateSubMgr statiistics end. <<
2020-11-28 18:13:26.907 INFO [Thread-0] ObjectStatReport - Pub_i statistics :Pub_i statiistics end
2020-11-28 18:13:26.907 INFO [Thread-0] ObjectStatReport - Program_statistics end. <<
Hi @fuweng.ng
Looking at the output you have provided, it would appear that there are problems with your DACS server - which is not a data provider - it is Data Access Control system.
In your consumer configuration, you have setup 4 channels - these alternate channels should be for different ADS instance. When your application connects to an ADS, the ADS must check your login credentials with the DACS server to ensure you are a valid user and then as you request data, it will again check with the DACS to ensure you are entitled to each of the data items you are requesting.
The problem is that If the DACS server(s) is slow to start or crashing - even if all your ADS are working fine they cannot proceed as they are unable to validate your credentials.
In the above examples you have posted, it is unlikely the failover is even being activated - you are most likely connecting to the first ADS. In scenario 1, the ADS then tries to validate your credentials with a DACS server, it waits a while and then it would appear the DACS server does respond before the timeout period, so the ADS can continue and process your login + data request.
In scenario 2, the ADS waits for the DACS server to respond which fails to do so in the timeout period so the ADS has no choice but to close the connection. There would be little point in failing over to a different ADS because the 1st ADS itself is running just fine.
I would speak to your Market Data team and identify why the DACS server is failing to respond to the ADS request to validate your credential. They need to resolve the DACS issues with a high priority. It is quite rare for a DACS server to fail to respond or crash - so there must some serious issue at play here. They can raise a support ticket at MyRefinitiv if they need assistance.
Thanks alot for your prompt reply. In this case, may i know where can i change the timeout for DACS server? is the below setting in EmaConfig.xml the timeout for DACS?
<ConnectionPingTimeout value="30000"/>
I am thinking if we can set the timeout longer, we can avoid the 2nd Scenario. Thanks.