[EMA/C++] FieldList/UpdateMsg performance with multithreading

maciej.sipior

Hi

We're in the process of developing a C++ application that will utilize RTSDK to send a rather large amounts of messages (200k+) in periodic bursts.

We're basing this on Cpp EMA, and we're running multiple threads each with its own InteractiveProvider to spread business logic and encoding load between different CPU cores. This lead us to some issues with seemingly high lock contention in unexpected places.
We're reusing UpdateMsg and FieldList objects similarly to what's done here: https://github.com/Refinitiv/Real-Time-SDK/blob/master/Cpp-C/Ema/Examples/Training/IProvider/200_Series/280_MP_Perf/IProvider.cpp and see a lot of locking related to FieldList and UpdateMsg.

Clearing FieldList takes more time than actually encoding the data:

As far as I can see, the following scenario occurs (for FieldList.clear().addReal()):

1. FieldList::clear() - does nothing of note and calls
2. FieldListEncoder::clear() - clears some stuff and calls
3. Encoder::releaseEncIterator() - clears _containerComplete an returns _pEncodeIter to global pool via
4. void Pool< I, T >::returnItem( I* item ) - _pEncodeIter is returned to pool after lock is acquired
5. FieldList::addReal() - calls
6. FieldListEncoder::addReal() - which on first call after clearing will call
7. Encoder::acquireEncIterator() - just to obtain probably the same _pEncodeIter we have just released (locking again)

The end result is lots of locks for no apparent gain. And theres a lot of contention because all of these resources are accessed via global pool.

Details from vtune (same as above, but expanded):

As you can see, around 65% of time spent on encoding is just releasing and acquiring EncIterator. I focused on FieldList here, but basically the same happens to UpdateMsg, with an interesting caveat - the clear() chains from UpdateMsg, to UpdateMsgEncoder, to MsgEncoder where it calls Encoder::releaseEncIterator; if instead it called Encoder::clear(), it would have cleared the iterator without the need to reacquire it later.

However, I don't think I could ever have a full picture after just glancing over the code for few hours, so it's very likely I'm missing some important parts, hence my questions:

- is this release/acquire cycle necessary? It's prevalent as far as I can see, but as I'm using almost exclusively UpdateMsg and FieldList those are the ones impacting me the most.

- can this be worked around somehow? Other than running two providers in separate processes that is.

- maybe there's a way to 'reset' FieldList/UpdateMsg after use, without releasing _pEncodeIter?

If you have any suggestion as to how to modify our usage (be it in code or through configuration), or could clarify what am I missing here, it would be very welcome.

Jirapongse

@maciej.sipior

Thanks for reaching out to us.

I think it is a design in EMA to make it easier to use. In EMA, developers can't access the Encoder directly. To change the design, if you are a Refinitiv Developer Connect named user, you can submit a ticket to Premium Support directly via Refinitiv Real-Time SDK - C/C++. Then, the support team will contact the developer team for further consideration. Otherwise, you can also raise this concern on GitHub.

To develop a high-performance provider application, you may consider using Enterprise Transport API in the RTSDK package.

The Enterprise Transport API (ETA) is the high-performance, low-latency, foundation of the Refinitiv Real-Time SDK. This product allows applications to achieve the highest throughput, lowest latency, low memory utilization, and low CPU utilization when publishing or consuming content.

With ETA, you can control almost everything. For example, you can have one Encoder iterator to encode all messages. There are provider examples in the package.