Increased memory usage during message rate bursts

leonardo.pupacic

Hello,

We are experiencing increased memory usage in our application built on Release Real-Time-SDK-2.2.3.L1 · Refinitiv/Real-Time-SDK. The memory usage grows during periods of message rate bursts and does not decrease thereafter. Over period of days this leads to memory usage reaching critical levels, requiring the application restart.
For additional context, we maintain a large watchlist of items (3 mil +) which is registered 24x7.

Through analysis thus far, the potential issue appears to be with EmaPools management, specifically in getItem() and returnItem() methods.

For example, looking at DecoderPool, a new item is created when all existing items are in use (Real-Time-SDK/Cpp-C/Ema/Src/Access/Impl/EmaPool.h at e9c377a28efb7aaa015da40382e00b200283c86f · Refinitiv/Real-Time-SDK). Once returned, pool grows if additional space is required (Real-Time-SDK/Cpp-C/Ema/Src/Access/Impl/EmaPool.h at e9c377a28efb7aaa015da40382e00b200283c86f · Refinitiv/Real-Time-SDK).

It does not appear that this growth is bounded, or pool size reduced if no longer required to be at current capacity.

We've tested our application against a build with modified EmaPool.h where EncoderPool, DecoderPool and Pool objects do not rely on _vector to keep items on the heap. Instead, a call to getItem() allocates an item, while call to returnItem() deletes it. This may have implications on CPU usage, or perhaps other unintended consequences, and was done strictly to try to isolate the memory growth issue.

With this modification in place, memory usage increases during message rate bursts, but it then decreases to previous level once the message rate returns to normal.

The ask for you is to confirm that EmaPools are indeed at play here, and if so to provide an official fix.

Thanks,

Leo

Jirapongse

@leonardo.pupacic

Thank you for detailed information.

First of all, EMA is designed with ease of use in mind, rather than performance. If your goal is to consume millions of items, ETA is a more suitable choice, as it offers significantly better performance compared to EMA. For further details, please refer to the article titled Real-Time SDKs Performance Test Summary.

Regarding this issue, it can occur particularly when the application uses a message copy constructor—such as UpdateMsg::UpdateMsg(const UpdateMsg& other)—to clone messages, and then processes a large number of active cloned messages. After a quick review of the code, I observed that the API retrieves a decoder from the pool when the copy constructor is invoked, and returns it to the pool when the message is deleted. Therefore, if many decoder objects accumulate in the pool, it may indicate that the application or EMA may not be able to process all retrieved messages during burst periods.

Were there any disconnections observed during high-traffic or burst periods?

However, EMA doesn't provide any configurations to control the size of the EMA Decoder Pool. This could be an enhancement request.

This forum is available to answer general questions about API usage. To verify and fix this issue, you need to:

• Contact the API support team directly via Contact Premium Support
• Or, raise this issue on GitHub