EMA frequent channel down error

aastha.sharma
aastha.sharma Newcomer
in EMA

Hello,

Please find attached log new 2.txt. Unable to figure out the source of the error, I am subscribing to 3k RIC's using batch request and getting streaming updates. Once I get the updates, I retrieve the ask price and put it on a queue as follows -

@Override

public void onRefreshMsg(final RefreshMsg refreshMsg, final OmmConsumerEvent event) {

log.debug("RefreshMsg");

if (refreshMsg.hasName() && (DataType.DataTypes.FIELD_LIST == refreshMsg.payload()

.dataType())) {

Entity entity = new Entity();

entity.setRefreshMsg(refreshMsg);

this.jmsTemplate.convertAndSend(Queues.BATCH_UPDATE, entity);

}

}

@Override

public void onUpdateMsg(final UpdateMsg updateMsg, final OmmConsumerEvent event) {

log.debug("updateMsg");

if (updateMsg.hasName() && (DataType.DataTypes.FIELD_LIST == updateMsg.payload()

.dataType())) {

Entity entity = new Entity();

entity.setUpdateMsg(updateMsg);

this.jmsTemplate.convertAndSend(Queues.BATCH_UPDATE, entity);

}

}

I have set Output buffers as below -

innerElementList.add(EmaFactory.createElementEntry()
  .intValue("GuaranteedOutputBuffers", 50000));
Tagged:

Best Answer

  • umer.nalla
    umer.nalla LSEG
    Answer ✓

    Hi @aastha.sharma

    One of the most common causes for Consumer disconnects is a 'Slow Consumer' scenario. Increasing Guaranteed output buffers can only help so far - e.g. for short-lived periods of increased update rates.

    If the consumer is processing the updates too slowly, then eventually the buffers on the server you are connecting to will overflow and you would be disconnected.

    I cannot tell from the log file, if this is the case here - so you would need to speak to either your internal RTDS team (if you are connecting to an internal ADS) OR our Refinitiv Real-Time Optimized team (if you are connecting to RTO) - so that they can check the logs on the server you are connecting to, to get some clues as to why your consumer is being logged out so often.

    If they do confirm that they are seeing buffer overflow scenarios, then you need to modify your code so that less processing is done in the main API thread - the onUpdateMsg and onRefreshMsg callbacks are executed on the main API thread. And the longer you hold onto that thread, the less time the API can spend reading data from the server in a timely manner.

    The following article refers to one of our Legacy APIs - however, the same slow consumer scenario could be the case for you and the same solutions could also help you - e.g. worker threads or Horizontal scaling OR a combination of both :
    How to Diagnose and Solve Out-of-Memory Issues in RFA Java Applications | Refinitiv Developers

    The following example included with RT-SDK is a simple demonstration of Horizontal Scaling

    RTSDK-JAVA\Java\Ema\Examples\src\main\java\com\refinitiv\ema\examples\training\consumer\series400\ex410_MP_HorizontalScaling

Answers

  • zoya faberov
    zoya faberov ✭✭✭✭✭

    Hello @aastha.sharma ,

    I would suggest, for the purposes of testing, temporarily, to exclude, by commenting out, all Entity related code and to test if the disconnects continue to happen.

    If they do not, this code is likely holding the callback, resulting in slow consumer designation by the infra and the disconnect from the infra.

    If they continue to happen without any Entity processing code, I would suggest opening a ticket with Refinitiv infra support, including all the details of the disconnects (specific times), to investigate possible infrastructure causes as well with the local network admin/group, investigating any local network related causes, if you are on RTO connectivity or hosted, that may need to include internet provider originated causes.

    Let us know how this works on your side?

Categories