Our EMA consumer periodically stops working due to an error regarding the reactor shutting down.
loggerMsg ClientName: ChannelCallbackClient Severity: Error Text: Received ChannelDown event on channel Channel Instance Name EmaConsumer_1 RsslReactor @4f363abd RsslChannel @21a2fda5 Error Id -1 Internal sysError 0 Error Location WlItemHandler.readUpdateMsg Error text ReactorCallbackReturnCodes.FAILURE was returned from defaultMsgCallback(). This caused the Reactor to shutdown. loggerMsgEnd
Once this occurs, all subsequent requests will fail with additional errors...
loggerMsg ClientName: SingleItem Severity: Error Text: Internal error: rsslChannel.submit() failed in SingleItem.submit(RequestMsg)RsslChannel 0 Error Id -1 Internal sysError 0 Error Location ReactorChannel.submit Error Text Reactor is shutdown, submit aborted. loggerMsgEnd com.thomsonreuters.ema.access.OmmInvalidUsageExceptionImpl: Failed to open or modify item request. Reason: ReactorReturnCodes.FAILURE. Error text: Reactor is shutdown, submit aborted.
Then our EMA client has to be restarted to get it to work again.
Can anyone explain why this occurs and how we can fix it? The errors do not provide any additional information as to where to start investigating.
There is a clue in the first EMA log output given:
Received ChannelDown event on channel Channel
In my test, when I tried to connect to a server, then I killed the server process or cut the connection, I got "Received ChannelDownReconnecting event on channel Channel_1" instead.
So, I searched for the meaning of the difference between them which are as follows:
CHANNEL_DOWN ReactorChannel has failed (e.g. the connection was lost or a ping timeout expired) and can no longer send or receive data. There is no connection recovery for this event.
CHANNEL_DOWN_RECONNECTING ReactorChannel has failed and connection recovery has started. This only occurs on client connections since there is no connection recovery for server connections.
Referring to the meaning of CHANNEL_DOWN definition, the problem should be from connectivity issue between EMA and the server.
Regarding the second EMA log output, it seems like the application tried to send a message to the server, but the reactor had been shut down already. I believe that it happened because of the connectivity problem earlier.
Therefore, I have several questions below:
If the application didn't perform activities in question 2 and 3, I suggest you test the application with the testing server such as IProvider example, to verify whether this problem still persists when connects to the local server or not.
For your questions...
Java application on Linux
Yes, the second part of the log output occurs when the consumer is posting
Not understanding the question.
If the Channel_Down error occurs, I thought the system automatically tries to reconnect, but we have no log entries to support that it is. Is there something we need to do to enable the retries? To get 'CHANNEL_DOWN_RECONNECTING' instead of just 'CHANNEL_DOWN' and stop working.