Parsing OMM Payload on different thread

anil.kalra

We are building Level2 market maker data processing application using RFA and OMM consumer. We want to do the parsing of different handles(Ric symbols) in parallel so we want to pass the RFA OMM payload to a different thread and not parse it in the consumer thread. Is it safe to pass the payload to different processing thread.

I know in EMA is it not recommended to do so because the field entries and field lists are reused.

Thanks

Anil Kalra

Gurpreet

Hi @anil.kalra I would recommend that you try out a barebone EMA sample like example210__MarketByOrder__Streaming for multiple instruments, and suppress printing to console and compare if there is still extraneous GC going on.

You can see the EMA source code at: https://github.com/Refinitiv/Elektron-SDK/tree/master/Java

Gurpreet

Hi @anil.kalra For any new application, I would strongly discourage using RFA API for any development. It is feature complete, and you will want to use an API which is supported for a long time to come - which is EMA API.

In both EMA and RFA, the API owns the OMM payload message which is delivered to the application layer and will get reused by the API. For any parallel processing, you can register multiple clients and event queue can be dispatched by multiple threads. All this requires careful execution, since many times, thread contention slows down rather than speed things up.

You will also have to account for race conditions - for e.g. a L2 delete order message with no payload, might get processed before a previous update order message with payload is processed.

anil.kalra

Gurpreet. for the reply. So we cannot handover the payload to a different thread unless we do a deep copy. As far as update and delete race condition is concerned, we were thinking to send the payload over a queue and same handle will always go to same queue so that they don't get process out of order.

Our idea was to not parse the message inline on a processEvent method on the consumer thread and not block it but if the payload is reused then we cannot achieve this.

Thanks

Anil

anil.kalra

Gurpreet. Also for EMA we tried it generated lot of garbage collection which increased our micro collection and pauses in the application. If there is any workaround for this we will be happy to change to EMA.

Gurpreet

@anil.kalra There is no simple answer here. There can be dozens of way an application might be architected and almost always interfaces with inhouse services. We have clients who used EMA for building their Level2 shadow books for whole venue, so it is definately doable.

anil.kalra

Thanks @Gurpreet I agree there are many ways to implement. We have a working model of Level2 using EMA and RFA. But our initial testing shows that EMA generates way lot more garbage compared to RFA and that was the only reason we decided to go with RFA.

Any suggestion on EMA garbage generation.

Thanks for your reply and suggestions.

Anil