question

Upvotes
Accepted
1 0 1 0

Thomson Reuters RFA EventQueue - Log Queue depth - How to log the current queue depth periodically

Hello,

We have a .NET client application that subscribes to Thomson Reuters RFA News Feed direct. We are trying to diagnose an issue with the application that features the Event Queues MaxQueueDepth being reached and reset. We need to understand the rate that the event queue fills over time to understand why the event queue is filling up passed the MaxQueueDepth threshold.

I have added logging in a class called RdmClient that inherits from ThomsonReuters.RFA.Common.Client. I am trying to log the current queue depth size and the currently executing process physical memory usage so that we can better understand how the application is performing over time. The application logs the queue depth size and the physical memory usage every-time it processes and event on the Event Queue.

So far the logs are showing the Current Event Queue depth as being either 0 or 1. I am not sure if I am logging the current queue depth accurately of correctly. I wanted to find if anybody has better ideas in regards to effectively logging the current queue depth as the application executes?

Thanks

James Kibirige

treprfarfa-apinewsloggingevent-queuequeue-depth
icon clock
10 |1500

Up to 2 attachments (including images) can be used with a maximum of 512.0 KiB each and 1.0 MiB total.

10 Answers

Upvotes
Accepted
52.4k 134 44 63

Refer to the RFA.NET reference guide, the EventQueue.Dispatch method returns an approximate number of events left in the queue after success. Therefore, this value is not accurate.

From the code, you are using the Event Queue with Statistics so you can use the EventQueue.EventQueueCurrentSize property. This property returns the current number of events stored in the queue if statistics are enabled and always zero (0) if not. It is a proper way to verify the number of events remaining in the statistic queue.

The application is a slow consumer which causes the event queue’s maximum depth is reached. To avoid this issue, the application must make sure that the client.processEvent() method returns as fast as possible. Running long processing tasks inside the client.processEvent() method must be avoided. Otherwise, you can use multiple event queues and each event queue will be dispatched by its own application thread in order to distribute workload among event queues and increase the application’s performance.

icon clock
10 |1500

Up to 2 attachments (including images) can be used with a maximum of 512.0 KiB each and 1.0 MiB total.

Upvotes
24.9k 86 10 22

Hi @James.Kibirige

Can you please confirm how you are obtaining the Queue Depth - using EventQueueCurrentSize or the return value from the Dispatch() method call?

Also, can you confirm if you are enabling Statistics when you Create the Event Queue by providing a second parameter set to True?

public static EventQueue Create(RFA_String name,
     bool statistics
 )

Also, please confirm the version of RFA.NET you are using.

Also, if you refer to the RFA.Net Developers guide,you will note that it states "Using the existing registerNotificationClient() method is equivalent to enabling high threshold notification with a high threshold value of 1. It might not make sense to use both to call the dispatch method if the application is using both as notifications."

In other words, you will be notified each time the queue depth is 1 or higher. This could explain why you are always seeing a queue size of 1 (assuming you are using EventQueueCurrentSize to determine the queue depth).

If you are going to use the notification client then you should make use of the high/low threshold notifications are appropriate. Otherwise, you can use the return value from the Dispatch() call to get an approximate queue size instead.

icon clock
10 |1500

Up to 2 attachments (including images) can be used with a maximum of 512.0 KiB each and 1.0 MiB total.

Upvote
7.6k 15 6 9

@James.Kibirige

Not sure how many RIC you subscribe and how about the update rate of the RICs you are testing?

As Umer said, first of all, please make sure that you already turn on EventQueue statistic when you create the queue by setting the flag to true.

eventQueue = EventQueue.Create(new RFA_String("myEventQueue"),true);


I guess that you are monitoring the EventQueueCurrentSize so the number of events in the queue depending on the frequency of the update with events from RFA internal. Also, it depends on how fast your application can dispatch and process the event. If your application has a long-running task or app logic inside the process event it can block the event processing and the number of events in the queues may keep increasing. Anyway, if you want to test, you can use one of the RFA consumer examples such as StarterConsumer to subscribing to high update RIC such as currency RIC like JPY=, GBP=, and AUD=. I add those RICs to the itemList config in StarterConsumer.cfg and see the EventQueueCurrentSize more than 1 from the example.


icon clock
10 |1500

Up to 2 attachments (including images) can be used with a maximum of 512.0 KiB each and 1.0 MiB total.

Upvotes
1 0 1 0

Hi @moragodkrit, @Umer Nalla,

Thank you for the quick response to my original question.

The application is already set up to report statistics:

  • _context.EventQueue = EventQueue.Create(new RFA_String("rfaQueue"), true);

Where can I download the RFA.Net Developers guide?

In regards to the dispatch method do you thus suggest logging within the main control loop for the application where the call to Dispatch() is used. I don't want to prematurely Dispatch the EventQueue while the application runs.

Yes we are trying to ascertain why the EventQueue fills up causing the application to restart. I am not sure of the rate that update events arrive from RFA Internal, and would like to know how to measure this effectively. The main control loop manages the rate of EventQueue dispatch and we can use this to ascertain the rate that messages are processed on the queue.

Thanks for the advice I will apply this in the application as part of the testing I am currently doing.

Thanks

James Kibirige

icon clock
10 |1500

Up to 2 attachments (including images) can be used with a maximum of 512.0 KiB each and 1.0 MiB total.

Hello @James.Kibirige,

Please find RFA .Net Dev Guide in RFA->Documentation->Developer Guides

Upvotes
1 0 1 0

Hi guys,


I am still having a problem with the Dispatch method the application is using the following version of the method:

  • _context.EventQueue.Dispatch(5000); --with a parameter timeOut of 5000

With this current set up when the application runs it logs the EventQueue size as 0 or -1 intermittently which isn't what I was expecting from the application.

Thanks

James Kibirige

icon clock
10 |1500

Up to 2 attachments (including images) can be used with a maximum of 512.0 KiB each and 1.0 MiB total.

Upvotes
1 0 1 0

Hi @moragodkrit, @Umer-Nalla ,

I have been monitoring the logs after I made adjustments to the application to log the Current Queue size. The application has generated the following alert:

  • Queue threshold of '4000' breached

Unfortunately the logging I have to get the Current Queue Size is as follows:

  • DEBUG : NewsAnalytics.Service.RdmRunner: Current Queue Size: 0

So even when I capture the return value of the Dispatch() method the Queue size is always 0:

  • var dispatchReturn = _context.EventQueue.Dispatch(5000);

This is a problem because it means we cannot accurately measure or log the Current event queue depth, we need to understand why the Event queue is filling up. Is their anything that I am missing?


Thanks

James Kibirige

icon clock
10 |1500

Up to 2 attachments (including images) can be used with a maximum of 512.0 KiB each and 1.0 MiB total.

Upvotes
24.9k 86 10 22

Hi @James.Kibirige

The reason your queue is filling up is because your application is unable to process all the events in a timely manner. It is taking too long to process each event it receives and therefore, you are building up a backlog in the event queue - a scenario described as "Slow consumer".

See a previous post on suggestions on how to deal with slow consumer scenarios.

You can also refer to this article - which although is based around RF Java - still provides some useful context.

As to why the dispatch method is not returning the event queue size correctly, you can raise a support ticket and investigate this issue in more detail offline.

icon clock
10 |1500

Up to 2 attachments (including images) can be used with a maximum of 512.0 KiB each and 1.0 MiB total.

Upvotes
24.9k 86 10 22

Hi @James.Kibirige

I just tried a quick test here with RFA.Net v8.1.0.L1 example StarterConsumer subscribing to several highly traded RICs.

I added the following line of code in the Run() method

int dispatchReturn = eventQueue.Dispatch(10);
AppUtil.Log(AppUtil.LEVEL.INFO,string.Format("Dispatch : {0}\r\n ", dispatchReturn));

and I see positive values being output:

2020/01/13 12:26:36.134 INFO    Dispatch : 0
2020/01/13 12:26:36.160 INFO    Dispatch : 58
2020/01/13 12:26:36.181 INFO    Dispatch : 57
and so on.

Obviously the return value fluctuates between positive and -1, depending on how many events are in the queue.

icon clock
10 |1500

Up to 2 attachments (including images) can be used with a maximum of 512.0 KiB each and 1.0 MiB total.

Hi,

For our application the queue depth returned by the Dispatch() method is always low: {0,1,2} even when we get Max Queue depth breached messages. The notification client will log:

  • Max queue depth of '5000' breached. Application will be restarted.

But the queue depth logged was:

  • : DEBUG : NewsAnalytics.Service.RdmRunner: Current Queue Size: 1

What's even worse is that I we are seeing unexpected things with the applications memory usage:

  • : DEBUG : NewsAnalytics.Service.RdmRunner: Current Process Physical memory usage: 272908288

The memory usage has been steadily going up since I started running the application as a service last week. Even though a max queue depth breached error has occurred which states the application should restart the application memory usage is still going up. I expected the memory usage to drop following the application restart.

I was also expecting to see correlation between the queue depth & memory usage. The queue depth being logged by the application doesn't correlate with the other values being logged.

Right now I don't have any confidence in the queue depth values being logged by the application.

Thanks

James Kibirige

Hi,

As per my previous email please raise a support ticket to investigate why the dispatch method is not returning the queue size correctly.

In terms of memory usage, if the queue size is growing because the application is not processing the events quickly enough, then I would expect the memory size to increase inline with the queue size growth. Each event in the queue will consume some memory based on the payload of the event.

You have not mentioned the version of RFA.NET you are using - it could be that you are using an older version which had a bug in this area.

As per my previous suggestion please raise a support ticket so someone from our developer support team can work with you in more detail to diagnose this issue.

Hi Umer,

Thanks again for your speedy response, we are currently using RFA.NET 8.0.1.2. Thus you suggest that we upgrade?

The link you provided for raising a support ticket is broken.

In regards to the memory usage I am noticing the following:

  • There is an overall memory usage value that steadily rises following the start or restart of the application as a service.
    • During the run-time of the application the memory usage goes up and down but this is a small fraction of the overall memory usage for example:
      • 2020-01-13 14:22:36,505 : DEBUG : NewsAnalytics.Service.RdmRunner: Current Process Physical memory usage: 272920576
      • 2020-01-13 14:22:47,581 : DEBUG : NewsAnalytics.Service.RdmRunner: Current Process Physical memory usage: 272875520

As you can see above the memory usage will fluctuate but the general trend is that the memory usage is increasing. So tomorrow the overall memory usage will be larger. Based on the Max Queue depth notification functionality I was expecting the memory usage to drop to 0 following application reset, the notification message states that this is the expected behavior.

But it's clear that the application doesn't restart since the memory usage hasn't dropped dramatically & suddenly as would be expected if I were to manually restart the service.

Thanks

James Kibirige

Upvote
24.9k 86 10 22

Hi @James.Kibirige

With regards the support ticket link, this may be controlled by your employer in terms of who can raise a ticket - I have reached out to your Refinitiv Account manager to try and identify who is the authorised user in your organisation. You may also be able to determine this by speaking to your internal Market Data team.

With regards your other points, I am somewhat confused. It is not up to the API to restart the application / process / service - that would be an application level implementation. All I can assume is that the restart is not being done as expected - perhaps the application code is not performing a proper cleanup and shutdown of the RFA API. Please refer to the Cleanup() method of the StarterConsumer example that comes with the RFA SDK or the Tutorials example for an idea of what RFA objects need to be deactivated, destroyed, released etc. for a clean RFA shutdown.

If, however, you are referring to the Event Queue reset as described in the developer guide, you will note this only occurs if you have set a EventQueueMaxDepth greater than 0. As explained in the documentation, this is different to the EventQueueHighThreshold and EventQueueLowThreshold values. Obviously, if you set do set EventQueueMaxDepth, then all the events in the queue will be discarded - as described in the developer guide. This may not be a desirable outcome - If an application needs to retain all events, the application should not enable the event queue maximum depth functionality.

For further details please refer to section Event Queue with Statistics of the RFA developer guide (and the EventQueue section of RFA Reference Manual). It provides further details on the whole mechanism including actions required on the part of the developer to reset the event queue when ready - otherwise it will not restart queuing events again.

icon clock
10 |1500

Up to 2 attachments (including images) can be used with a maximum of 512.0 KiB each and 1.0 MiB total.

Hi Umer,

We are setting the Event Queue Max Depth to 5000, and intermittently with an occurrence of about once a week the Max Depth Notification Client sends an e-mail with the following message:

  • Max queue depth of '5000' breached. Application will be restarted.

The logging I have added to the application and the logs we have captured haven't been enough to diagnose why the application performance drops in these instances causing the queue depth breach.

Even if we know that the issue is caused by the slow consumer scenario, getting to the bottom of why this occurs is still a pertinent question.

Thanks

James Kibirige

Hi Umer,

I have been able to access and download the RFA API documentation for the latest version of the API 8.1.1.1. I wanted to know if there is anyway I can get a hold of the documentation effective for version 8.0.1.2?

Thanks

James Kibirige

Hi @James.Kibirige

I have checked both the older versions sections of the Developer Portal and also our internal repository of official releases and I cannot see a version 8.0.1.2.

The 8.0.xx releases I can see are 8.0.1.E1.win , 8.0.1.L1 and 8.0.0.L1. You can download these from RFA - .NET - OLDER VERSIONS

Once you download a version and unzip the files you will find the documentation in the Docs subfolder.

Hi @James.Kibirige

Ok I see what you meant - the 8.0.1.E1.win version contains the 8.0.1.2 version DLLs

Upvotes
24.9k 86 10 22

Hi @James.Kibirige

Based on my past experience, the most likely explanation are that the queue is gradually building over the week. This would suggest that your application is slightly too slow to process all the events it receives in a timely manner.

Just as an example, say it was receiving on average 1000 events an hour, perhaps it can only process around 970 - so the event queue is gradually increasing in size at a rate of 700 events per day on average.This would mean that over 7 days a backlog of approx 5000 events has build up in the queue.

As the message 'Max queue depth of '5000' breached. Application will be restarted.' is an application generated message, it is the applications responsibility to restart or reset RFA etc.

Did you read the article and post I linked previously, they talk about things you can try to improve the rate at which your application processes the events it receives.

I did reach out to your account manager but have not heard back in terms of your organisation's approved RDC contact. I will try an alternative source.

Regards,
Umer

icon clock
10 |1500

Up to 2 attachments (including images) can be used with a maximum of 512.0 KiB each and 1.0 MiB total.

Hi Umer,


Thanks for the feedback, based on what we are logging and the messages we are storing in the database I don't believe this explains the issue with the application memory usage. On average we are storing at least 5000 news feed records in the database per day, sometimes more than 10000 messages. The logs show that the memory usage does fluctuate in a minor way during each day which corresponds with events building up and being dispatched from the Event Queue as expected. But if the general trend of an increasing memory usage was because of events building up in the event queue then we would be getting Max Queue depth error alerts daily. Also When the queue depth is reached and the application restarts the memory usage should drop dramatically back to the rate it was when the application was first started.

The logs show that we process way more than 5000 messages in a week.


I think the memory/resource leak issue is related to unmanaged code in the RFA API.

Thanks

James Kibirige

Hi @James.Kibirige

Firstly, you should have received a Welcome email granting you permission to raise Premium Support tickets. If you have not already done so, I recommend you raise a ticket so this issue can be investigated in more depth off line.

In terms of your points above - I don't understand why you would expect to get Max Queue Depth alerts daily. You would only get this alert when the number of unprocessed events in the queue exceeds whatever value you have set for EventQueueMaxDepth. You say you are storing at least 5000-10000 news stories a day - which means you are processing i.e. removing 5000-10000 events from the queue every day. However, if more events are arriving from the server & being added to the queue than you are processing, then the queue size will build up. Or it may be that on most days you are processing all the events that arrive - but on a particular busy day more events are received than your application can process & so they build up in the queue until the max depth is breached. Once the number of unprocessed events exceeds 5000 then the Max Queue depth alert will be triggered.

Also, if an application is shutdown and restarted, all the memory used by the application should be released and reclaimed by the OS. I can't think of a how a process is fully shutdown and yet retains it memory footprint when it is restarted?
I am not a .NET expert so it is possible you have come across some bug - please raise a ticket to investigate further


Hi Umer,

The point I was making about the queue depth is that I don't agree with your analysis of the memory usage issue. I don't believe that the Memory usage is going up and up day by day because of the Event Queue messages.

If the total memory usage was proportional to the number of messages on the queue why does the memory usage keep going up even after we get Max Queue depth breached errors? The Max queue depth error may occur on a particular day but 5 days later the memory usage is even higher than it was on the day that the Max queue depth was breached.

If the total memory usage was proportional to the number of messages on the queue I would expect it to drop and fluctuate significantly as messages are processed. This is not how it is behaving.

Memory/Resource leaks are rare in .NET applications because of the CLR's garbage collection functionality. They can happen when .NET apps use un-managed code like in this case with the RFA API.

Thanks

James Kibirige


Show more comments