Filling EMA messages is not multi-thread-safe.

Hello

I have an interactive provider, that uses a dedicated thread per connected consumer to build up EMA messages and send them to the corresponding consumer.

Of course I use individual message objects per thread, of type RefreshMsg / UpdateMsg with a FieldList therein.

However, if the code that fills the message objects is not lock-protected, the data sent to to the consumers get corrupted (the fields in the field list get random values). If there is a lock protection arround the code, which fills the (thread-specific!) message objects, everything works fine. Obviously it is not safe, to build (different!) messages at the same time in multiple threads. Why ?!?

This is annoying, because the lock protection slows down the program.

Find more posts tagged with

elektron

fields

elektron-sdk

refinitiv-realtime

rrt

multi-threading

elektron-message-api

ema-api

update-message

Accepted answers

zoya faberov

Hello Michael @miczuc,

Off-topic, a sense of humor is not as common among developers, as one would think, and is appreciated .

On the topic:

The multi-threading in EMA gives you a lot, on consumer side of the equation, and less on the provider side of it.

Many improvements to our APIs originate with developer clients, such as yourself, and are prioritized, if several clients are interested, then the suggestion gains in priority. The other consideration is your Refinitiv account team may aid you in prioritization, for example, if your development is part of a strategic initiative, etc.

If you find something like this, we encourage you to let us know, so we can improve.

On how you'd prefer to proceed:

I can add this (in my opinion) useful suggestion, to Elektron SDK tracked issue list, as I understand it, on your behalf? If you would like your enhancement to be tracked as customer, please comment with your organization info, as you have registered on the portal with a private email address.

Or, alternatively, you can make the suggestion directly, via Elektron SDK on GitHub, including details, optionally a link to this forum, optionally adding me on the tag (zfarberov), to enable me to keep the community members interested, updated of any suggestions or progress?

All comments

zoya faberov

Hello @miczuc,

Sorry to hear you are finding this slow.

In my understanding, the thread-safe behavior per EMA design, is centered around safe dispatch by multiple threads. The dispatch mechanisms, the callback processing, both for consumers and providers, that implemented, are described in EMA Developer Guide.

I do not believe that the thread-safeness as documented was designed to cover your requirement by the library. I would venture a guess, that creating and populating messages for the purposes of publication is not considered as strictly time-sensitive.

Therefore, what you describe would be an enhancement request rather then an expected behavior.

If on your side you consider this very important, rather then nice to have, please submit the description of requirement directly to Elektron SDK Development on GitHub so the enhancement can be considered and if accepted, prioritized.

Would it make sense for your requirements, to create and populate messages from a single thread, perhaps to partially pre-populating the contents known ahead of time?

miczuc

I did't say, "I am finding this slow". I said, the need to protect the building of messages by a lock is slowing down the program. That means, the program could run faster (with higher troughput), if the building of messages was multi-thread-safe. One should expect, that building messages is multi-thread-safe ...

So maybe I have to register on the GitHub site and report my problems there ....

zoya faberov

Hello @miczuc,

I hear you and if you would like to report this to ESDK development team, you are entitled to do so.

However, I am also thinking as to what kind of numbers or ballpark rates on the provider side you are looking to achieve, and of the best way to achieving those numbers. If you are targeting very high numbers, configuration tuning, input and output buffers tuning is usually required. It may also be considered if EMA is the right API tier for you or if ETA is a better choice.

miczuc

Hello @zoya.farberov ,

you don't want to program yourself to death using the ETA C library directly.
You need a simplifying object-oriented layer on top of that like EMA.
There are many good things in the EMA library, I don't damn it.
Unforntunately EMA is not well designed concerning multi-threading.
E.g. the submit()-methods in class OmmIProviderImpl all lock the same mutex
in their interior, around all code therein. That means, sending data in multiple theads at
the same time is dramatically hindered, even if each thread sends data on a different connection
("channel"), because effectively only one thread can execute
the method at a time while the others are blocked.
Of course you need some locking to protect internal data structures,
but sending data on different connections should not interfere with each other.
So that could be done with less locking (smaller periods of time holding locks),
but that is more difficult to implement.
In my option this is another big problem, that should be reported to the development.

Regards, Michael Zuck

zoya faberov

Hello Michael @miczuc,

Off-topic, a sense of humor is not as common among developers, as one would think, and is appreciated .

On the topic:

The multi-threading in EMA gives you a lot, on consumer side of the equation, and less on the provider side of it.

If you find something like this, we encourage you to let us know, so we can improve.

On how you'd prefer to proceed:

EXPLORE OUR SITES