RTO EMA C++ application triggered the server overflow condition
There is a RTO EMA C++ application always failed at the US Market Open time. And the US Market RICs (those suffixed with '.NB', ~ 12000 RICs) were subscribed hours ago. Then suddenly at the US Market Open time, it lost the connection.
The logs here, removed the username and position. The 'GuaranteedOutputBuffers' was set to 5000 as the example 450.And looks like it triggered the overflow condition in the backend. Finally EMA cannot recover it.
The question is the recipe to cope with this case?
loggerMsg
TimeStamp: 18:21:46.021
ClientName: ChannelCallbackClient
Severity: Success
Text: Received ChannelUp event on channel Channel_1
Instance Name Consumer_1_1
Connected component version: ads3.4.2.L1.linux.tis.rrg 64-bit
loggerMsgEnd
loggerMsg
TimeStamp: 22:30:30.517
ClientName: ChannelCallbackClient
Severity: Warning
Text: Received ChannelDownReconnecting event on channel Channel_1
Instance Name Consumer_1_1
RsslReactor 0x0x7f424c0d7980
RsslChannel 0x0x7f424c011f90
Error Id -1
Internal sysError 104
Error Location /local/jenkins/workspace/ESDKCore_RCDEV/OS/RH8-64/rcdev/source/rtsdk/Cpp-C/Eta/Impl/Reactor/rsslReactor.c:4884
Error Text </local/jenkins/workspace/ESDKCore_RCDEV/OS/RH8-64/rcdev/source/rtsdk/Cpp-C/Eta/Impl/Transport/rsslSocketTransportImpl.c:696> Error:1002 ipcRead() failure. System errno: (104)
loggerMsgEnd
loggerMsg
TimeStamp: 22:30:30.570
ClientName: LoginCallbackClient
Severity: Warning
Text: RDMLogin stream state was changed to suspect with status message
username ----
usernameType 1
position ----/net
appId 256
applicationName RTO
instanceId <not set>
singleOpen 1
allowSuspect 1
optimizedPauseResume 0
permissionExpressions 1
permissionProfile 0
supportBatchRequest 1
supportEnhancedSymbolList 1
supportPost 1
supportRtt 0
supportViewRequest 1
role 0
authenticationTTReissue 1645698687
authenticationErrorCode 0
State: Open / Suspect / None / 'Channel is down.'
loggerMsgEnd
loggerMsg
TimeStamp: 22:30:30.592
ClientName: ChannelDictionary
Severity: Warning
Text: RDMDictionary stream was closed with status message
streamId 3
Closed, Recoverable / Suspect / None / 'Service for this item was lost.'
loggerMsgEnd
loggerMsg
TimeStamp: 22:30:30.595
ClientName: ChannelDictionary
Severity: Warning
Text: RDMDictionary stream was closed with status message
streamId 4
Closed, Recoverable / Suspect / None / 'Service for this item was lost.'
loggerMsgEnd
loggerMsg
TimeStamp: 22:31:04.182
ClientName: ChannelCallbackClient
Severity: Warning
Text: Received ChannelDownReconnecting event on channel Channel_1
Instance Name Consumer_1_1
RsslReactor 0x0x7f424c0d7980
RsslChannel 0x(nil)
Error Id -1
Internal sysError 0
Error Location /local/jenkins/workspace/ESDKCore_RCDEV/OS/RH8-64/rcdev/source/rtsdk/Cpp-C/Eta/Impl/Reactor/rsslReactorWorker.c:1324
Error Text Initialization timed out.
loggerMsgEnd
......
loggerMsg
TimeStamp: 22:34:28.025
ClientName: ChannelCallbackClient
Severity: Warning
Text: Received ChannelDownReconnecting event on channel Channel_1
Instance Name Consumer_1_1
RsslReactor 0x0x7f424c0d7980
RsslChannel 0x(nil)
Error Id -1
Internal sysError 0
Error Location /local/jenkins/workspace/ESDKCore_RCDEV/OS/RH8-64/rcdev/source/rtsdk/Cpp-C/Eta/Impl/Reactor/rsslReactorWorker.c:1324
Error Text Initialization timed out.
loggerMsgEnd
<END, no more log>
Best Answer
-
Hello @Frederic ,
As you discuss that the disconnect is consistently reproducible, the application repeatedly fails at US market open, and the suspected cause if buffer overflow, I would increase the buffer to 10k or even 20k and see if this helps eliminate the issue. Example 450 just consumes single item, as it is intended to demo Service Disovery, not high performance.
However, a very common cause for this type of disconnect is the application not being able to process the events quick enough, a slow consumer, please see this previous discussion thread and by searching forums for "slow consumer" you will find many helpful posts and hints.
It may be helpful to review EMA examples -> PerTools -> PerfConsumer., there is also PerProvider that can be used for pinpointing the issue as well as verifying the changes.
If the client reporting the issue is a premium support customer and would like a more detailed review of their specific cause, they may submit all the details directly as premium support case.
0
Answers
-
Thanks, I give it a go to figure it out.
wordle0
Categories
- All Categories
- 6 AHS
- 36 Alpha
- 166 App Studio
- 6 Block Chain
- 4 Bot Platform
- 18 Connected Risk APIs
- 47 Data Fusion
- 34 Data Model Discovery
- 684 Datastream
- 1.4K DSS
- 613 Eikon COM
- 5.2K Eikon Data APIs
- 10 Electronic Trading
- Generic FIX
- 7 Local Bank Node API
- 3 Trading API
- 2.9K Elektron
- 1.4K EMA
- 248 ETA
- 552 WebSocket API
- 37 FX Venues
- 14 FX Market Data
- 1 FX Post Trade
- 1 FX Trading - Matching
- 12 FX Trading – RFQ Maker
- 5 Intelligent Tagging
- 2 Legal One
- 23 Messenger Bot
- 3 Messenger Side by Side
- 9 ONESOURCE
- 7 Indirect Tax
- 60 Open Calais
- 275 Open PermID
- 44 Entity Search
- 2 Org ID
- 1 PAM
- PAM - Logging
- 6 Product Insight
- Project Tracking
- ProView
- ProView Internal
- 22 RDMS
- 1.9K Refinitiv Data Platform
- 630 Refinitiv Data Platform Libraries
- 4 LSEG Due Diligence
- LSEG Due Diligence Portal API
- 4 Refinitiv Due Dilligence Centre
- Rose's Space
- 1.2K Screening
- 18 Qual-ID API
- 13 Screening Deployed
- 23 Screening Online
- 12 World-Check Customer Risk Screener
- 1K World-Check One
- 46 World-Check One Zero Footprint
- 45 Side by Side Integration API
- 2 Test Space
- 3 Thomson One Smart
- 10 TR Knowledge Graph
- 151 Transactions
- 143 REDI API
- 1.8K TREP APIs
- 4 CAT
- 26 DACS Station
- 121 Open DACS
- 1.1K RFA
- 104 UPA
- 191 TREP Infrastructure
- 228 TRKD
- 915 TRTH
- 5 Velocity Analytics
- 9 Wealth Management Web Services
- 86 Workspace SDK
- 11 Element Framework
- 5 Grid
- 18 World-Check Data File
- 1 Yield Book Analytics
- 46 中文论坛