When I run my application with its full universe of symbols (a few hundred instrument RICs and 30 or so chain RICs), often the default message dispatch thread gets stuck, and I stop getting updates.
I can usually run the application OK with fewer symbols, but even then, the message dispatch thread still occasionally gets stuck.
Here is a dump of the thread delivering messages while in a healthy state:
"pool-11-thread-1" #72 prio=5 os_prio=0 cpu=15.62ms elapsed=45.62s tid=0x0000022afd416260 nid=0xad30 runnable [0x000000be286fe000]
java.lang.Thread.State: RUNNABLE
at sun.nio.ch.WEPoll.wait(java.base@17 .0.8/Native Method)
at sun.nio.ch.WEPollSelectorImpl.doSelect(java.base@17 .0.8/WEPollSelectorImpl.java:111)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(java.base@17 .0.8/SelectorImpl.java:129)
- locked <0x00000000fecc8fe0> (a sun.nio.ch.Util$2)
- locked <0x00000000fecc8b48> (a sun.nio.ch.WEPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(java.base@17 .0.8/SelectorImpl.java:141)
at com.refinitiv.ema.access.OmmBaseImpl.rsslReactorDispatchLoop(OmmBaseImpl.java:1751)
at com.refinitiv.ema.access.OmmBaseImpl.run(OmmBaseImpl.java:1889)
at java.util.concurrent.ThreadPoolExecutor.runWorker(java.base@17 .0.8/ThreadPoolExecutor.java:1136)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(java.base@17 .0.8/ThreadPoolExecutor.java:635)
at java.lang.Thread.run(java.base@17 .0.8/Thread.java:833)
And this is what it looks like wedged, it seems to be the ItemCallbackClient#removeFromMap call that's causing the problem:
"pool-11-thread-1" #72 prio=5 os_prio=0 cpu=46.88ms elapsed=18.37s tid=0x000002067ef35ff0 nid=0x8a88 waiting on condition [0x00000032d3bfe000]
java.lang.Thread.State: WAITING (parking)
at jdk.internal.misc.Unsafe.park(java.base@17 .0.8/Native Method)
- parking to wait for <0x00000000ff033a50> (a java.util.concurrent.locks.ReentrantLock$NonfairSync)
at java.util.concurrent.locks.LockSupport.park(java.base@17 .0.8/LockSupport.java:211)
at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(java.base@17 .0.8/AbstractQueuedSynchronizer.java:715)
at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(java.base@17 .0.8/AbstractQueuedSynchronizer.java:938)
at java.util.concurrent.locks.ReentrantLock$Sync.lock(java.base@17 .0.8/ReentrantLock.java:153)
at java.util.concurrent.locks.ReentrantLock.lock(java.base@17 .0.8/ReentrantLock.java:322)
at com.refinitiv.ema.access.ItemCallbackClient.removeFromMap(ItemCallbackClient.java:2477)
at com.refinitiv.ema.access.SingleItem.remove(ItemCallbackClient.java:2932)
at com.refinitiv.ema.access.ItemCallbackClient.processStatusMsg(ItemCallbackClient.java:1893)
at com.refinitiv.ema.access.ItemCallbackClient.defaultMsgCallback(ItemCallbackClient.java:1630)
at com.refinitiv.eta.valueadd.reactor.Reactor.sendDefaultMsgCallback(Reactor.java:2787)
at com.refinitiv.eta.valueadd.reactor.Reactor.sendAndHandleDefaultMsgCallback(Reactor.java:2896)
at com.refinitiv.eta.valueadd.reactor.WlItemHandler.callbackUser(WlItemHandler.java:3029)
at com.refinitiv.eta.valueadd.reactor.WlItemHandler.readMsg(WlItemHandler.java:2002)
at com.refinitiv.eta.valueadd.reactor.Watchlist.readMsg(Watchlist.java:302)
at com.refinitiv.eta.valueadd.reactor.Reactor.processRwfMessage(Reactor.java:4662)
at com.refinitiv.eta.valueadd.reactor.Reactor.performChannelRead(Reactor.java:5001)
at com.refinitiv.eta.valueadd.reactor.Reactor.dispatchAll(Reactor.java:7674)
at com.refinitiv.ema.access.OmmBaseImpl.rsslReactorDispatchLoop(OmmBaseImpl.java:1759)
at com.refinitiv.ema.access.OmmBaseImpl.run(OmmBaseImpl.java:1889)
at java.util.concurrent.ThreadPoolExecutor.runWorker(java.base@17 .0.8/ThreadPoolExecutor.java:1136)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(java.base@17 .0.8/ThreadPoolExecutor.java:635)
at java.lang.Thread.run(java.base@17 .0.8/Thread.java:833)
