Discover Refinitiv
MyRefinitiv Refinitiv Perspectives Careers
Created with Sketch.
All APIs Questions & Answers  Register |  Login
Ask a question
  • Questions
  • Tags
  • Badges
  • Unanswered
Search:
  • Home /
  • Elektron /
avatar image
Question by tbaker · May 09, 2018 at 05:34 AM · emajavaelektron sdk - java

OmmConsumer, ping timeout, channel recover and maximum load.

Hi,
We have an application using EMA Java to subscribe, process and then republish some data to our TREP. We are seeing some issues where the OmmConsumerImpl loses the connection (channel down) and then fails to recover. We have a ChannelSet configuration and you can see from the logs that the OmmConsumerImpl is bringing up a channel, getting a ping timeout (30seconds) and then trying the next channel. The ADS servers were both up at the time of this failure and other applications were working fine so this is an issue within EMA failing the connection locally. We have seen the connection recover from a channel down at other times but the example below shows that it never recovers.

The overall issue is also why we get the channel down events in the first place? The ChannelSet failing to recover is also a fairly big issue.

Is there a maximum capacity that we should use a single instance of a OmmConsumer for? We are subscribing to around 8000 tickers and we were receiving about 10k updates/second at the time of the initial Channel Down event.

Perhaps we need to tune some of the parameters, we have:

Consumer:

  • LoginRequestTimeOut=30000ms
  • User dispatch,
  • MaxDispatchCountUserThread= default (100), should we set this higher given we expect 1000s of messages a second?
  • DispatchTimeoutApiThread=0 (default), should we adjust this?

Channel:

  • NumInputBuffers=2048
  • GuaranteedOutputBuffers=5000
  • ConnectionPingTimeout= Default (looks like the 30second default below)
  • SysRecvBufSize - we're not setting this, should we?
  • SysSendBufSize - we're not setting this, should we?


Here are the logs from the channel down events:

2018-05-04 14:35:41.110 WARN  [ElektronConsumerDispatcher] 
access.OmmConsumerImpl - loggerMsg| ClientName: ChannelCallbackClient| 
Severity: Warning| Text:  Received ChannelDownReconnecting event on chan
nel
 2.rmds.| RsslReactor @1d63c996| RsslChannel @4f10502a| Error Id 0| 
Internal sysError 0| Error Location null| Error text SocketChannel.read 
returned -1 (end-of-stream)| loggerMsgEnd|
2018-05-04 14:35:41.205 
WARN  [ElektronConsumerDispatcher] access.OmmConsumerImpl - loggerMsg| 
ClientName: LoginCallbackClient| Severity: Warning| Text:  RDMLogin 
stream state was changed to suspect wit
h status message| username <not set>| usernameType <not set>| State: Open/Suspect/None - text: ""| loggerMsgEnd|
2018-05-04
 14:35:41.243 WARN  [ElektronConsumerDispatcher] access.OmmConsumerImpl -
 loggerMsg| ClientName: ChannelDictionary| Severity: Warning| Text:  
RDMDictionary stream state was changed to suspect with status message| 
streamId 3| Reason State: Open/Suspect/None - text: "channel down."| 
loggerMsgEnd|
2018-05-04 14:35:41.244 WARN  
[ElektronConsumerDispatcher] access.OmmConsumerImpl - loggerMsg| 
ClientName: ChannelDictionary| Severity: Warning| Text:  RDMDictionary 
stream state was changed to suspect with status message| streamId 4| 
Reason State: Open/Suspect/None - text: "channel down."| loggerMsgEnd|
2018-05-04
 14:35:41.245 WARN  [ElektronConsumerDispatcher] 
heartbeat.HeartbeatService - Error event for heartbeat message on HB. 
ErrorEvent{subject=HB, state=UNKNOWN, statusMsg=channel down.}
2018-05-04
 14:35:41.255 WARN  [ElektronConsumerDispatcher] 
subscribe.RmdsTickClient - Received: ErrorEvent{subject=.., 
state=UNKNOWN, statusMsg=channel down.}
...
2018-05-04 
14:35:42.877 WARN  [ElektronConsumerDispatcher] subscribe.RmdsTickClient
 - Received: ErrorEvent{subject=.., state=UNKNOWN, statusMsg=channel 
down.}
2018-05-04 14:35:42.877 WARN  [ElektronConsumerDispatcher] 
subscribe.RmdsTickClient - Received: ErrorEvent{subject=.., 
state=UNKNOWN, statusMsg=channel down.}
2018-05-04 14:35:42.878 INFO 
 [ElektronConsumerDispatcher] access.OmmConsumerImpl - loggerMsg| 
ClientName: ChannelCallbackClient| Severity: Info| Text:  Received 
ChannelUp event on channel 1.rmds.| Instance Name CONSUMER_B_PREPROD_1| 
Component Version ads2.6.9.L1.linux.tis.rrg 64-bit| loggerMsgEnd|
...
2018-05-04
 14:40:44.365 WARN  [ElektronConsumerDispatcher] access.OmmConsumerImpl -
 loggerMsg| ClientName: ChannelCallbackClient| Severity: Warning| Text: 
 Received ChannelDownReconnecting event on channel 1.rmds.| RsslReactor 
@1d63c996| RsslChannel @4f10502a| Error Id 0| Internal sysError 0| Error
 Location Reactor.processWorkerEvent| Error text Ping error for channel:
 Lost contact with connection...| loggerMsgEnd|
2018-05-04 
14:40:44.366 WARN  [ElektronConsumerDispatcher] access.OmmConsumerImpl -
 loggerMsg| ClientName: LoginCallbackClient| Severity: Warning| Text:  
RDMLogin stream state was changed to suspect with status message| 
username <not set>| usernameType <not set>| State: 
Open/Suspect/None - text: ""| loggerMsgEnd|
2018-05-04 14:40:44.367 
WARN  [ElektronConsumerDispatcher] subscribe.RmdsTickClient - Received: 
ErrorEvent{subject=.., state=UNKNOWN, statusMsg=channel down.}
...
2018-05-04
 14:40:45.391 INFO  [ElektronConsumerDispatcher] access.OmmConsumerImpl -
 loggerMsg| ClientName: ChannelCallbackClient| Severity: Info| Text:  
Received ChannelUp event on channel 2.rmds.| Instance Name 
CONSUMER_B_PREPROD_1| Component Version ads2.6.9.L1.linux.tis.rrg 
64-bit| loggerMsgEnd|


Later on, we had several ping timeouts:

2018-05-04 15:08:56.215 WARN  [ElektronConsumerDispatcher] 
access.OmmConsumerImpl - loggerMsg| ClientName: ChannelDictionary| 
Severity: Warning| Text:  RDMDictionary stream state was changed to 
suspect with status message| streamId 4| Reason State: Open/Suspect/None
 - text: "channel down."| loggerMsgEnd|
2018-05-04 15:08:56.197 WARN 
 [ElektronConsumerDispatcher] access.OmmConsumerImpl - loggerMsg| 
ClientName: ChannelCallbackClient| Severity: Warning| Text:  Received 
ChannelDownReconnecting event on channel 2.rmds.| RsslReactor @6091c183|
 RsslChannel @26d64e5a| Error Id 0| Internal sysError 0| Error Location 
Reactor.processWorkerEvent| Error text Ping error for channel: Lost 
contact with connection...| loggerMsgEnd|
2018-05-04 15:08:56.204 
WARN  [ElektronConsumerDispatcher] access.OmmConsumerImpl - loggerMsg| 
ClientName: LoginCallbackClient| Severity: Warning| Text:  RDMLogin 
stream state was changed to suspect with status message| username 
<not set>| usernameType <not set>| State: Open/Suspect/None -
 text: ""| loggerMsgEnd|
2018-05-04 15:08:56.213 WARN  
[ElektronConsumerDispatcher] access.OmmConsumerImpl - loggerMsg| 
ClientName: ChannelDictionary| Severity: Warning| Text:  RDMDictionary 
stream state was changed to suspect with status message| streamId 3| 
Reason State: Open/Suspect/None - text: "channel down."| loggerMsgEnd|
2018-05-04
 15:08:57.269 INFO  [ElektronConsumerDispatcher] access.OmmConsumerImpl -
 loggerMsg| ClientName: ChannelCallbackClient| Severity: Info| Text:  
Received ChannelUp event on channel 1.rmds.| Instance Name 
CONSUMER_B_PREPROD_1| Component Version ads2.6.9.L1.linux.tis.rrg 
64-bit| loggerMsgEnd|
2018-05-04 15:17:14.069 WARN  
[ElektronConsumerDispatcher] access.OmmConsumerImpl - loggerMsg| 
ClientName: ChannelCallbackClient| Severity: Warning| Text:  Received 
ChannelDownReconnecting event on channel 1.rmds.| RsslReactor @6091c183|
 RsslChannel @26d64e5a| Error Id 0| Internal sysError 0| Error Location 
Reactor.processWorkerEvent| Error text Ping error for channel: Lost 
contact with connection...| loggerMsgEnd|
2018-05-04 15:17:14.069 
WARN  [ElektronConsumerDispatcher] access.OmmConsumerImpl - loggerMsg| 
ClientName: LoginCallbackClient| Severity: Warning| Text:  RDMLogin 
stream state was changed to suspect with status message| username 
<not set>| usernameType <not set>| State: Open/Suspect/None -
 text: ""| loggerMsgEnd|
2018-05-04 15:17:15.116 INFO  
[ElektronConsumerDispatcher] access.OmmConsumerImpl - loggerMsg| 
ClientName: ChannelCallbackClient| Severity: Info| Text:  Received 
ChannelUp event on channel 2.rmds.| Instance Name CONSUMER_B_PREPROD_1| 
Component Version ads2.6.9.L1.linux.tis.rrg 64-bit| loggerMsgEnd|
2018-05-04
 15:19:44.217 WARN  [ElektronConsumerDispatcher] access.OmmConsumerImpl -
 loggerMsg| ClientName: ChannelCallbackClient| Severity: Warning| Text: 
 Received ChannelDownReconnecting event on channel 2.rmds.| RsslReactor 
@6091c183| RsslChannel @26d64e5a| Error Id 0| Internal sysError 0| Error
 Location Reactor.processWorkerEvent| Error text Ping error for channel:
 Lost contact with connection...| loggerMsgEnd|
2018-05-04 
15:19:44.218 WARN  [ElektronConsumerDispatcher] access.OmmConsumerImpl -
 loggerMsg| ClientName: LoginCallbackClient| Severity: Warning| Text:  
RDMLogin stream state was changed to suspect with status message| 
username <not set>| usernameType <not set>| State: 
Open/Suspect/None - text: ""| loggerMsgEnd|
2018-05-04 15:19:45.258 
INFO  [ElektronConsumerDispatcher] access.OmmConsumerImpl - loggerMsg| 
ClientName: ChannelCallbackClient| Severity: Info| Text:  Received 
ChannelUp event on channel 1.rmds.| Instance Name CONSUMER_B_PREPROD_1| 
Component Version ads2.6.9.L1.linux.tis.rrg 64-bit| loggerMsgEnd|
2018-05-04
 16:07:23.093 WARN  [ElektronConsumerDispatcher] access.OmmConsumerImpl -
 loggerMsg| ClientName: ChannelCallbackClient| Severity: Warning| Text: 
 Received ChannelDownReconnecting event on channel 1.rmds.| RsslReactor 
@6091c183| RsslChannel @26d64e5a| Error Id 0| Internal sysError 0| Error
 Location Reactor.processWorkerEvent| Error text Ping error for channel:
 Lost contact with connection...| loggerMsgEnd|
2018-05-04 
16:07:23.094 WARN  [ElektronConsumerDispatcher] access.OmmConsumerImpl -
 loggerMsg| ClientName: LoginCallbackClient| Severity: Warning| Text:  
RDMLogin stream state was changed to suspect with status message| 
username <not set>| usernameType <not set>| State: 
Open/Suspect/None - text: ""| loggerMsgEnd|
2018-05-04 16:07:24.663 
INFO  [ElektronConsumerDispatcher] access.OmmConsumerImpl - loggerMsg| 
ClientName: ChannelCallbackClient| Severity: Info| Text:  Received 
ChannelUp event on channel 2.rmds.| Instance Name CONSUMER_B_PREPROD_1| 
Component Version ads2.6.9.L1.linux.tis.rrg 64-bit| loggerMsgEnd|
2018-05-04
 16:22:31.342 WARN  [ElektronConsumerDispatcher] access.OmmConsumerImpl -
 loggerMsg| ClientName: ChannelCallbackClient| Severity: Warning| Text: 
 Received ChannelDownReconnecting event on channel 2.rmds.| RsslReactor 
@6091c183| RsslChannel @26d64e5a| Error Id 0| Internal sysError 0| Error
 Location Reactor.processWorkerEvent| Error text Ping error for channel:
 Lost contact with connection...| loggerMsgEnd|
2018-05-04 
16:22:31.343 WARN  [ElektronConsumerDispatcher] access.OmmConsumerImpl -
 loggerMsg| ClientName: LoginCallbackClient| Severity: Warning| Text:  
RDMLogin stream state was changed to suspect with status message| 
username <not set>| usernameType <not set>| State: 
Open/Suspect/None - text: ""| loggerMsgEnd|

People who like this

0 Show 0
Comment
10 |1500 characters needed characters left characters exceeded
▼
  • Viewable by all users
  • Viewable by moderators
  • Viewable by moderators and the original poster
  • Advanced visibility
Viewable by all users

Up to 2 attachments (including images) can be used with a maximum of 512.0 KiB each and 1.0 MiB total.

1 Reply

  • Sort: 
avatar image
REFINITIV
Best Answer
Answer by Nipat Kunvutipongsak · Jun 26, 2018 at 12:09 AM

After the investigation, somehow this “Error text Ping error for channel: Lost contact with connection...” error message doesn't occur with EMA version 3.2.0.1 (Elektron-SDK_1.2.0.1.L1). It could be used as a workaround/resolution for this issue.

Comment

People who like this

0 Show 1 · Share
10 |1500 characters needed characters left characters exceeded
▼
  • Viewable by all users
  • Viewable by moderators
  • Viewable by moderators and the original poster
  • Advanced visibility
Viewable by all users

Up to 2 attachments (including images) can be used with a maximum of 512.0 KiB each and 1.0 MiB total.

avatar image
tbaker · Jul 05, 2018 at 04:31 AM 0
Share

We haven't seen it again yet on the new version but it's fairly intermittent so we'll keep an eye out for any new occurences.

Watch this question

Add to watch list
Add to your watch list to receive emailed updates for this question. Too many emails? Change your settings >
12 People are following this question.

Related Questions

EMA API Login Issue (JAVA)

Can we have a batch request with EMA JAVA API (Version 3.2.0.1)? Also, Would be good to know how EMA handles batch internally and return response? Will it be a callback per RIC or single callback for whole batch.

What are the possible options to validate RICs using EMA JAVA API? Is it only Callback "Record Not Found" response or do we have any other API to validate multiple rics in single call?

What is the most appropriate FID Date/Time/DateTime combination to establish the message DateTime, TIMEACT seems to be only consistent field coming in both refresh and updates

Submit of CloseMsg failed

  • Feedback
  • Copyright
  • Cookie Policy
  • Privacy Statement
  • Terms of Use
  • Careers
  • Anonymous
  • Sign in
  • Create
  • Ask a question
  • Spaces
  • Alpha
  • App Studio
  • Block Chain
  • Bot Platform
  • Calais
  • Connected Risk APIs
  • DSS
  • Data Fusion
  • Data Model Discovery
  • Datastream
  • Eikon COM
  • Eikon Data APIs
  • Elektron
    • EMA
    • ETA
    • WebSocket API
  • Legal One
  • Messenger Bot
  • Messenger Side by Side
  • ONESOURCE
    • Indirect Tax
  • Open PermID
    • Entity Search
  • Org ID
  • PAM
    • PAM - Logging
  • ProView
  • ProView Internal
  • Product Insight
  • Project Tracking
  • Refinitiv Data Platform
    • Refinitiv Data Platform Libraries
  • Rose's Space
  • Screening
    • Qual-ID API
    • Screening Deployed
    • Screening Online
    • World-Check One
    • World-Check One Zero Footprint
  • Side by Side Integration API
  • TR Knowledge Graph
  • TREP APIs
    • CAT
    • DACS Station
    • Open DACS
    • RFA
    • UPA
  • TREP Infrastructure
  • TRIT
  • TRKD
  • TRTH
  • Thomson One Smart
  • Transactions
    • REDI API
  • Velocity Analytics
  • Wealth Management Web Services
  • World-Check Data File
  • Explore
  • Tags
  • Questions
  • Badges