We're in the process of developing a C++ application that will utilize RTSDK to send a rather large amounts of messages (200k+) in periodic bursts.
We're basing this on Cpp EMA, and we're running multiple threads each with its own InteractiveProvider to spread business logic and encoding load between different CPU cores. This lead us to some issues with seemingly high lock contention in unexpected places.
We're reusing UpdateMsg and FieldList objects similarly to what's done here: https://github.com/Refinitiv/Real-Time-SDK/blob/master/Cpp-C/Ema/Examples/Training/IProvider/200_Series/280_MP_Perf/IProvider.cpp and see a lot of locking related to FieldList and UpdateMsg.
Clearing FieldList takes more time than actually encoding the data:
As far as I can see, the following scenario occurs (for FieldList.clear().addReal()):
- FieldList::clear() - does nothing of note and calls
- FieldListEncoder::clear() - clears some stuff and calls
- Encoder::releaseEncIterator() - clears _containerComplete an returns _pEncodeIter to global pool via
- void Pool< I, T >::returnItem( I* item ) - _pEncodeIter is returned to pool after lock is acquired
- FieldList::addReal() - calls
- FieldListEncoder::addReal() - which on first call after clearing will call
- Encoder::acquireEncIterator() - just to obtain probably the same _pEncodeIter we have just released (locking again)
The end result is lots of locks for no apparent gain. And theres a lot of contention because all of these resources are accessed via global pool.
Details from vtune (same as above, but expanded):
As you can see, around 65% of time spent on encoding is just releasing and acquiring EncIterator. I focused on FieldList here, but basically the same happens to UpdateMsg, with an interesting caveat - the clear() chains from UpdateMsg, to UpdateMsgEncoder, to MsgEncoder where it calls Encoder::releaseEncIterator; if instead it called Encoder::clear(), it would have cleared the iterator without the need to reacquire it later.
However, I don't think I could ever have a full picture after just glancing over the code for few hours, so it's very likely I'm missing some important parts, hence my questions:
- is this release/acquire cycle necessary? It's prevalent as far as I can see, but as I'm using almost exclusively UpdateMsg and FieldList those are the ones impacting me the most.
- can this be worked around somehow? Other than running two providers in separate processes that is.
- maybe there's a way to 'reset' FieldList/UpdateMsg after use, without releasing _pEncodeIter?
If you have any suggestion as to how to modify our usage (be it in code or through configuration), or could clarify what am I missing here, it would be very welcome.