Thomson Reuters RFA EventQueue - Log Queue depth - How to log the current queue depth periodically

Can you please confirm how you are obtaining the Queue Depth - using EventQueueCurrentSize or the return value from the Dispatch() method call?

Also, can you confirm if you are enabling Statistics when you Create the Event Queue by providing a second parameter set to True?

public static EventQueue Create(RFA_String name,
     bool statistics
 )

Also, please confirm the version of RFA.NET you are using.

Also, if you refer to the RFA.Net Developers guide,you will note that it states "Using the existing registerNotificationClient() method is equivalent to enabling high threshold notification with a high threshold value of 1. It might not make sense to use both to call the dispatch method if the application is using both as notifications."

In other words, you will be notified each time the queue depth is 1 or higher. This could explain why you are always seeing a queue size of 1 (assuming you are using EventQueueCurrentSize to determine the queue depth).

If you are going to use the notification client then you should make use of the high/low threshold notifications are appropriate. Otherwise, you can use the return value from the Dispatch() call to get an approximate queue size instead.

moragodkrit.chumsri_1

@James.Kibirige

Not sure how many RIC you subscribe and how about the update rate of the RICs you are testing?

As Umer said, first of all, please make sure that you already turn on EventQueue statistic when you create the queue by setting the flag to true.

eventQueue = EventQueue.Create(new RFA_String("myEventQueue"),true);

I guess that you are monitoring the EventQueueCurrentSize so the number of events in the queue depending on the frequency of the update with events from RFA internal. Also, it depends on how fast your application can dispatch and process the event. If your application has a long-running task or app logic inside the process event it can block the event processing and the number of events in the queues may keep increasing. Anyway, if you want to test, you can use one of the RFA consumer examples such as StarterConsumer to subscribing to high update RIC such as currency RIC like JPY=, GBP=, and AUD=. I add those RICs to the itemList config in StarterConsumer.cfg and see the EventQueueCurrentSize more than 1 from the example.

Hi @moragodkrit, @Umer Nalla,

Thank you for the quick response to my original question.

The application is already set up to report statistics:

_context.EventQueue = EventQueue.Create(new RFA_String("rfaQueue"), true);

Where can I download the RFA.Net Developers guide?

In regards to the dispatch method do you thus suggest logging within the main control loop for the application where the call to Dispatch() is used. I don't want to prematurely Dispatch the EventQueue while the application runs.

Yes we are trying to ascertain why the EventQueue fills up causing the application to restart. I am not sure of the rate that update events arrive from RFA Internal, and would like to know how to measure this effectively. The main control loop manages the rate of EventQueue dispatch and we can use this to ascertain the rate that messages are processed on the queue.

Thanks for the advice I will apply this in the application as part of the testing I am currently doing.

Thanks

James Kibirige

Hi guys,

I am still having a problem with the Dispatch method the application is using the following version of the method:

_context.EventQueue.Dispatch(5000); --with a parameter timeOut of 5000

With this current set up when the application runs it logs the EventQueue size as 0 or -1 intermittently which isn't what I was expecting from the application.

Thanks

James Kibirige

zoya faberov

Hello @James.Kibirige

Please find RFA .Net Dev Guide in RFA->Documentation->Developer Guides

Hi @moragodkrit, @Umer-Nalla ,

I have been monitoring the logs after I made adjustments to the application to log the Current Queue size. The application has generated the following alert:

Queue threshold of '4000' breached

Unfortunately the logging I have to get the Current Queue Size is as follows:

DEBUG : NewsAnalytics.Service.RdmRunner: Current Queue Size: 0

So even when I capture the return value of the Dispatch() method the Queue size is always 0:

var dispatchReturn = _context.EventQueue.Dispatch(5000);

This is a problem because it means we cannot accurately measure or log the Current event queue depth, we need to understand why the Event queue is filling up. Is their anything that I am missing?

Thanks

James Kibirige

The reason your queue is filling up is because your application is unable to process all the events in a timely manner. It is taking too long to process each event it receives and therefore, you are building up a backlog in the event queue - a scenario described as "Slow consumer".

See a previous post on suggestions on how to deal with slow consumer scenarios.

You can also refer to this article - which although is based around RF Java - still provides some useful context.

As to why the dispatch method is not returning the event queue size correctly, you can raise a support ticket and investigate this issue in more detail offline.

I just tried a quick test here with RFA.Net v8.1.0.L1 example StarterConsumer subscribing to several highly traded RICs.

I added the following line of code in the Run() method

int dispatchReturn = eventQueue.Dispatch(10);
AppUtil.Log(AppUtil.LEVEL.INFO,string.Format("Dispatch : {0}\r\n ", dispatchReturn));

and I see positive values being output:

2020/01/13 12:26:36.134 INFO    Dispatch : 0
2020/01/13 12:26:36.160 INFO    Dispatch : 58
2020/01/13 12:26:36.181 INFO    Dispatch : 57
and so on.

Obviously the return value fluctuates between positive and -1, depending on how many events are in the queue.

Hi,

For our application the queue depth returned by the Dispatch() method is always low: {0,1,2} even when we get Max Queue depth breached messages. The notification client will log:

Max queue depth of '5000' breached. Application will be restarted.

But the queue depth logged was:

: DEBUG : NewsAnalytics.Service.RdmRunner: Current Queue Size: 1

What's even worse is that I we are seeing unexpected things with the applications memory usage:

: DEBUG : NewsAnalytics.Service.RdmRunner: Current Process Physical memory usage: 272908288

The memory usage has been steadily going up since I started running the application as a service last week. Even though a max queue depth breached error has occurred which states the application should restart the application memory usage is still going up. I expected the memory usage to drop following the application restart.

I was also expecting to see correlation between the queue depth & memory usage. The queue depth being logged by the application doesn't correlate with the other values being logged.

Right now I don't have any confidence in the queue depth values being logged by the application.

Thanks

James Kibirige

Hi,

As per my previous email please raise a support ticket to investigate why the dispatch method is not returning the queue size correctly.

In terms of memory usage, if the queue size is growing because the application is not processing the events quickly enough, then I would expect the memory size to increase inline with the queue size growth. Each event in the queue will consume some memory based on the payload of the event.

You have not mentioned the version of RFA.NET you are using - it could be that you are using an older version which had a bug in this area.

As per my previous suggestion please raise a support ticket so someone from our developer support team can work with you in more detail to diagnose this issue.

With regards the support ticket link, this may be controlled by your employer in terms of who can raise a ticket - I have reached out to your Refinitiv Account manager to try and identify who is the authorised user in your organisation. You may also be able to determine this by speaking to your internal Market Data team.

With regards your other points, I am somewhat confused. It is not up to the API to restart the application / process / service - that would be an application level implementation. All I can assume is that the restart is not being done as expected - perhaps the application code is not performing a proper cleanup and shutdown of the RFA API. Please refer to the Cleanup() method of the StarterConsumer example that comes with the RFA SDK or the Tutorials example for an idea of what RFA objects need to be deactivated, destroyed, released etc. for a clean RFA shutdown.

If, however, you are referring to the Event Queue reset as described in the developer guide, you will note this only occurs if you have set a EventQueueMaxDepth greater than 0. As explained in the documentation, this is different to the EventQueueHighThreshold and EventQueueLowThreshold values. Obviously, if you set do set EventQueueMaxDepth, then all the events in the queue will be discarded - as described in the developer guide. This may not be a desirable outcome - If an application needs to retain all events, the application should not enable the event queue maximum depth functionality.

For further details please refer to section Event Queue with Statistics of the RFA developer guide (and the EventQueue section of RFA Reference Manual). It provides further details on the whole mechanism including actions required on the part of the developer to reset the event queue when ready - otherwise it will not restart queuing events again.

Hi Umer,

We are setting the Event Queue Max Depth to 5000, and intermittently with an occurrence of about once a week the Max Depth Notification Client sends an e-mail with the following message:

Max queue depth of '5000' breached. Application will be restarted.

The logging I have added to the application and the logs we have captured haven't been enough to diagnose why the application performance drops in these instances causing the queue depth breach.

Even if we know that the issue is caused by the slow consumer scenario, getting to the bottom of why this occurs is still a pertinent question.

Thanks

James Kibirige

Based on my past experience, the most likely explanation are that the queue is gradually building over the week. This would suggest that your application is slightly too slow to process all the events it receives in a timely manner.

Just as an example, say it was receiving on average 1000 events an hour, perhaps it can only process around 970 - so the event queue is gradually increasing in size at a rate of 700 events per day on average.This would mean that over 7 days a backlog of approx 5000 events has build up in the queue.

As the message 'Max queue depth of '5000' breached. Application will be restarted.' is an application generated message, it is the applications responsibility to restart or reset RFA etc.

Did you read the article and post I linked previously, they talk about things you can try to improve the rate at which your application processes the events it receives.

I did reach out to your account manager but have not heard back in terms of your organisation's approved RDC contact. I will try an alternative source.

Regards,
Umer

Hi Umer,

Thanks for the feedback, based on what we are logging and the messages we are storing in the database I don't believe this explains the issue with the application memory usage. On average we are storing at least 5000 news feed records in the database per day, sometimes more than 10000 messages. The logs show that the memory usage does fluctuate in a minor way during each day which corresponds with events building up and being dispatched from the Event Queue as expected. But if the general trend of an increasing memory usage was because of events building up in the event queue then we would be getting Max Queue depth error alerts daily. Also When the queue depth is reached and the application restarts the memory usage should drop dramatically back to the rate it was when the application was first started.

The logs show that we process way more than 5000 messages in a week.

I think the memory/resource leak issue is related to unmanaged code in the RFA API.

Thanks

James Kibirige

Firstly, you should have received a Welcome email granting you permission to raise Premium Support tickets. If you have not already done so, I recommend you raise a ticket so this issue can be investigated in more depth off line.

In terms of your points above - I don't understand why you would expect to get Max Queue Depth alerts daily. You would only get this alert when the number of unprocessed events in the queue exceeds whatever value you have set for EventQueueMaxDepth. You say you are storing at least 5000-10000 news stories a day - which means you are processing i.e. removing 5000-10000 events from the queue every day. However, if more events are arriving from the server & being added to the queue than you are processing, then the queue size will build up. Or it may be that on most days you are processing all the events that arrive - but on a particular busy day more events are received than your application can process & so they build up in the queue until the max depth is breached. Once the number of unprocessed events exceeds 5000 then the Max Queue depth alert will be triggered.

Also, if an application is shutdown and restarted, all the memory used by the application should be released and reclaimed by the OS. I can't think of a how a process is fully shutdown and yet retains it memory footprint when it is restarted?
I am not a .NET expert so it is possible you have come across some bug - please raise a ticket to investigate further

Hi Umer,

I have been able to access and download the RFA API documentation for the latest version of the API 8.1.1.1. I wanted to know if there is anyway I can get a hold of the documentation effective for version 8.0.1.2?

Thanks

James Kibirige