OmmProvider.submit seems dead locked on > 2000 RICs

The server in question registers 2000 RICs, and updates all of them at interval of 0.5s. The 2000 RefreshMsg on registration and 4000 UpdateMsg/s are queued, and OmmProvider.submit-ed in a single thread. It all works fine.

However, when the RIC size is raised to 5000, it gets hanged. The thread dump looks like this:

Any help would be greatly appreciated.

Jianping

Find more posts tagged with

java

multi-threading

#technology

Accepted answers

Gurpreet

Hello Jianping,

This seems to be a case of resource crunch - most likely network. I can recommend these things to try to identify and remedy the problem:

Try using the infrastructure tool like TestClient to subscribe to your list of instruments. You can specify the snapshot option in the command line.
Enable OMM logging in the SDK
Try to pace out the subscriptions

See more information on how to get the infrastructure tool and usage in this previous discussion.

All comments

Gurpreet

Hello @Jianping

I would recommend that you enable the OMM Logging for the EMA SDK and upload the log file when this deadlock happens. There might be a status message that your app might have missed. If this is connecting to your local RTMDS, then check the ADS logs for server side message as well.

You can also try to pace out the subscriptions in a batch size of 1000 each. The batch size is physically limited by the array size in the SDK as described in this previous discussion -

https://community.developers.refinitiv.com/discussion/15222/batch-size-limit-on-eta-api

Jirapongse

@Jianping

Deadlock situations typically involve two threads.

Could you please provide the full thread dump for analysis?

Jianping

https://community.developers.refinitiv.com/discussion/comment/127463#Comment_127463

Hi Gurpreet,

I simplify the use case where I register 5000 Rics without updating, and would expect to receive 5000 RefreshMsg. On registering, I loop through 5000 Rics one by one via registerClient continuously.

On the client side, I only receive 133 RefreshMsg, and the rest of (5000-133) comes with StatusMsg of Timeout, and then 5000 StatusMsg of Channel down.

On the server side, the log records 1062 Incoming reactor message of ReqMsg, which is fewer than 5000 expected. Furthermore, out of these 1062 Incoming reactor message, OmmProviderClient only receives 133 onReqMsg, which it replies back with RefreshMsg. The number of RefreshMsg do match with that in client.

I hope i made it clear.

Jianping

Gurpreet

Hello Jianping,

This seems to be a case of resource crunch - most likely network. I can recommend these things to try to identify and remedy the problem:

Try using the infrastructure tool like TestClient to subscribe to your list of instruments. You can specify the snapshot option in the command line.
Enable OMM logging in the SDK
Try to pace out the subscriptions

See more information on how to get the infrastructure tool and usage in this previous discussion.

Jianping

https://community.developers.refinitiv.com/discussion/comment/127608#Comment_127608

Hi Gurpreet,

Yes, it looks a case of resource crunch. I interleave 100 Ric registering with 100 ms sleep. All together 5000 Rics are registered successfully. I also want to have a try of batch mode.

Many thanks

Jianping

EXPLORE OUR SITES