question

Upvotes
Accepted
54 1 1 3

Is there a number of limitation for registerClient under one consumer ?

Hi Team,

Is there a number of limitation for registerClient under one consumer ? thx 1655273539741.png


ema-api
1655273539741.png (72.6 KiB)
icon clock
10 |1500

Up to 2 attachments (including images) can be used with a maximum of 512.0 KiB each and 1.0 MiB total.

Upvotes
Accepted
24.5k 86 10 22

Hi @gang.chen1

Thanks for sharing the log files etc.

A few things:

  1. In the LPC log file, I can see multiple types of errors - including an 'Exceeded maximum number of mounts per user' as well as many buffer-related errors. The maximum number of mounts errors I expect is related to too many consumers trying to connect to LPC. I am not an LPC expert so you will need to raise an LPC ticket so they can help interpret the buffer errors and advise what may be causing them.
  2. The buffer errors could be a variety of things - which the LPC team may be able to advise on. However, one common scenario with larger watchlists is the Slow Consumer scenario. This is where the consumer application is not processing the data quickly enough - which leads to a buffer overflow on the server and therefore the server disconnects the consumer.
    Typical ways of dealing with Slow Consumer are:
    1. to use Horizontal scaling - i.e. split the watchlist between more than one OmmConsumer.
    2. Minimise the amount of processing done in the OnRefresh and OnUpdate callbacks - e.g. spawn off the processing to a worker thread - so that the OnRefresh or OnUpdate callbacks return control back to the main API thread ASAP - so that the API can go and read some more data from the server.
    3. I also have to ask, how often the developer is creating OmmConsumer - during the lifetime of the application? OmmConsumer represents the session between the application and the server and is therefore a relatively heavy process - it connects to the server, sends a login request, downloads a data dictionary, downloads a service directory etc. If the developer is creating OmmConsumer every few minutes (which seems to be the case?) I would advise against this - they should create each OmmConsumer once and then re-use it until the application terminates. Unless off course the application loses the connection to the server - in which case it would require creating OmmConsumer again to reestablish the session,

The application log file is over 1 million lines long - which is not something I can realistically analyse in the context of a forum. If after speaking to the LPC team they confirm that you have a slow consumer, please try the above suggestions to handle the slow consumer scenario. If you continue to face issues, then it would require a much deeper investigation offline by the Premium Support RDC team.


icon clock
10 |1500

Up to 2 attachments (including images) can be used with a maximum of 512.0 KiB each and 1.0 MiB total.

hi Umer,


Horizontal scaling is a very good way but RTO charged by Machine ID. Client just have 1 prod id and 1 UAT id. in theory, both prod and UAT can build up 1 connection(1 consumer). Actually, prod id can build up 13 consumers at the same time. it works well but some times one of consumer will face token refresh issue(caused by RTO). Then I ask client only build up 1 consumer with 20 registerClients in UAT, and the above suspect issue happened.


can you advise how many RICs could support in 1 consumer ? and can multiple thread support to receive message under 1 consumer ?


Hi @gang.chen1

According to LPC documentation, up to five client applications can use the same Client ID credential concurrently - so you should not need to provide additional MachineID to the customer.
And you certainly should not need 13 OmmConsumer instances - just 2-3 are generally OK for Horizontal scaling implementations.


Upvote
24.5k 86 10 22

Hi @gang.chen1

I am not aware of any limit to the number of clients you can register for an instance of OmmConsumer.

If you are registering clients (subscribing) for several thousand instruments, then you can consider using Horizontal scaling i.e. create more than one OmmConsumer and split your instruments between the OmmConsumer instances. e.g. if you were subscribing to say 50,000 instruments you may choose to register 25,000 instruments per OmmConsumer.

The exact number of OmmConsumer and the split would be for you to determine based on testing with your particular instrument list and your own network and feed performance etc.

Horizontal scaling is demonstrated in the ema\examples\training\consumer\series400\ex410_MP_HorizontalScaling example. This particular example creates two OmmConsumers for a single instrument each - which is something you would clearly not do in real life - but it is just an example showing how Horizontal scaling could be implemented (albeit with a much larger instrument list).


icon clock
10 |1500

Up to 2 attachments (including images) can be used with a maximum of 512.0 KiB each and 1.0 MiB total.

Hi Umer,

Huge thanks.

One of client is using 1 EMA OmmConsumer with around 20 registerClients connect to LPC , but they are facing bellow issue first.

1655287119818.png

Then bellow issue occurred in further

1655287197154.png

Could you advise what caused it ?


thanks

Gang Chen

1655287119818.png (150.6 KiB)
1655287197154.png (75.5 KiB)
Upvotes
24.5k 86 10 22

Hi @gang.chen1

I cant see anything particularly helpful in the above output.

Are you saying the issue only starts if they call registerClient 20 times?

The original code snippet only shows 4 instruments split into two batches.

Are they really only requesting 4 instruments? How many total instruments are they requesting across the 20 registerClient calls?

Also, can they provide fuller console output (2nd screenshot above) - captured to a txt file and attached here - rather than a limited screenshot (remove any confidential info)?

If they can also enable XMLTrace as described in this post and share the output - that could also be useful How to enable tracing incoming/outgoing messages EMA Java receives/sends - Forum | Refinitiv Developer Community


icon clock
10 |1500

Up to 2 attachments (including images) can be used with a maximum of 512.0 KiB each and 1.0 MiB total.

Hi Umer,

Client subscribe around 12K RICs, pls ignore my original code. I will share client's code and log via OneDrive because of large files.

thanks

Upvotes
24.5k 86 10 22

Hi @gang.chen1

Also, another thing I noticed - is the customer trying to Snap the data or consume with update ticks as well?
If they want to just snap the data and are not interested in Updates, then they should set interestAfterRefresh to false e.g.

consumer.registerClient(reqMsg.serviceName("ELEKTRON_DD").name("IBM.N").interestAfterRefresh(false), appClient);

Setting the above flag to false will result in the stream being closed off automatically once the RefreshMsg has been received - saving the developer the effort of subscribing and also potentially having to process unwanted UpdateMsgs

icon clock
10 |1500

Up to 2 attachments (including images) can be used with a maximum of 512.0 KiB each and 1.0 MiB total.

Thanks. Client is subscribing NY stocks data and Updates is required.

Upvotes
24.5k 86 10 22

Hi @gang.chen1

The advice from our RTO product team is to allocate additional free MachineID to the customer until the concurrent login problem with token refresh is resolved by the newer Oauth2 system.

I can't advice on number of RICs etc, too many factors such as update rates, number of fields being processed, just how much processing is doing etc. We advise customer to test different scenarios to arrive at safe numbers.

Did you speak to LPC team to understand the errors in the LPC log?


icon clock
10 |1500

Up to 2 attachments (including images) can be used with a maximum of 512.0 KiB each and 1.0 MiB total.

Hi Umer,

thanks and i will discuss with sales team for aditional free MID.

Much appreciate if you could get support from LPC team . I am guessing the issue was caused by 1 consumer with too much RIC subscribed.


thanks


To get LPC support you will need to raise a ticket via My.Refinitiv