reactor is shutdown message
Our EMA consumer periodically stops working due to an error regarding the reactor shutting down.
loggerMsg
ClientName: ChannelCallbackClient
Severity: Error
Text: Received ChannelDown event on channel Channel
Instance Name EmaConsumer_1
RsslReactor @4f363abd
RsslChannel @21a2fda5
Error Id -1
Internal sysError 0
Error Location WlItemHandler.readUpdateMsg
Error text ReactorCallbackReturnCodes.FAILURE was returned from defaultMsgCallback(). This caused the Reactor to shutdown.
loggerMsgEnd
Once this occurs, all subsequent requests will fail with additional errors...
loggerMsg
ClientName: SingleItem
Severity: Error
Text: Internal error: rsslChannel.submit() failed in SingleItem.submit(RequestMsg)RsslChannel 0
Error Id -1
Internal sysError 0
Error Location ReactorChannel.submit
Error Text Reactor is shutdown, submit aborted.
loggerMsgEnd
com.thomsonreuters.ema.access.OmmInvalidUsageExceptionImpl: Failed to open or modify item request. Reason: ReactorReturnCodes.FAILURE. Error text: Reactor is shutdown, submit aborted.
Then our EMA client has to be restarted to get it to work again.
Can anyone explain why this occurs and how we can fix it? The errors do not provide any additional information as to where to start investigating.
There is a clue in the first EMA log output given:
Received ChannelDown event on channel Channel
In my test, when I tried to connect to a server, then I killed the server process or cut the connection, I got "Received ChannelDownReconnecting event on channel Channel_1" instead.
So, I searched for the meaning of the difference between them which are as follows:
CHANNEL_DOWN ReactorChannel has failed (e.g. the connection was lost or a ping timeout expired) and can no longer send or receive data. There is no connection recovery for this event.
CHANNEL_DOWN_RECONNECTING ReactorChannel has failed and connection recovery has started. This only occurs on client connections since there is no connection recovery for server connections.
Referring to the meaning of CHANNEL_DOWN definition, the problem should be from connectivity issue between EMA and the server.
Regarding the second EMA log output, it seems like the application tried to send a message to the server, but the reactor had been shut down already. I believe that it happened because of the connectivity problem earlier.
Therefore, I have several questions below:
- What's the server?
- Did the consumer application perform any posting
- Did the consumer application send item requests at the beginning only or it sent requests on the fly?
If the application didn't perform activities in question 2 and 3, I suggest you test the application with the testing server such as IProvider example, to verify whether this problem still persists when connects to the local server or not.
Thanks, Nipat.
For your questions...
- What's the server?
Java application on Linux
- Did the consumer application perform any posting
Yes, the second part of the log output occurs when the consumer is posting
- Did the consumer application send item requests at the beginning only or it sent requests on the fly?
Not understanding the question.
If the Channel_Down error occurs, I thought the system automatically tries to reconnect, but we have no log entries to support that it is. Is there something we need to do to enable the retries? To get 'CHANNEL_DOWN_RECONNECTING' instead of just 'CHANNEL_DOWN' and stop working.
@joe.hong, to elaborate my previous questions, I observed that the application logged the error message while it was trying to send outgoing messages to the server, which could be the scenarios that...
- The application performed posting messages
- The application sent requests. In the normal situation, the consumer application typically sends requests at the beginning rather than in the middle of running
These scenarios are my previous question 2 and 3 respectively.
We're trying to check which example can be the test server and run on the same machine as the consumer application. I'll be back soon.
joe.hong, With reference to the definition of 'CHANNEL_DOWN' from Eta ref manual (ETA is an underlying layer of EMA), it indicates that the problem is from either connection lost or ping timeout expired.
To perform an experiment whether the original problem relates to connectivity issue or not, I suggest you connect the application to the local test server, which is an ETA's Interactive Provider example. This example locates in <Elektron SDK Package>/Eta/Applications/Examples (package: com\thomsonreuters\upa\examples\provider). It can support both On-stream and Off-stream post.
To add on top of my previous answer, you can modify a couple of lines in Provider.java to make it is easy for tracking output result as follows:
1. Extend the default runtime ( default runtime is just 1200 seconds or 2 minutes )
2. Prevent the provider generates updates and sends to a consumer automatically to remove unnecessary information except posting data only.
Moreover, this Interactive Provider example accepts the following fields only.
Fid: 2 Name = RDNDISPLAY Fid: 4 Name = RDN_EXCHID Fid: 38 Name = DIVPAYDATE Fid: 6 Name = TRDPRC_1 Fid: 22 Name = BID Fid: 25 Name = ASK Fid: 32 Name = ACVOL_1 Fid: 11 Name = NETCHNG_1 Fid: 267 Name = ASK_TIME
You may need to consider changing your application to send out known fields such as BID or ASK.
Have you tried connecting the local provider? If so, does the application still encounter the connection down problem from Reactor?