question

Upvotes
Accepted
20 3 1 8

cleanup() halts on rfa::common::EventQueue::destroy()

After RFA C++ modified example StarterConsumer_BatchView has been running for a period of time, cleanup() halts on rfa::common::EventQueue::destroy()

How long does it take to destroy EventQueue? Is there any timeout?

Is there anything I can do to trace the destroy progress?

treprfarfa-apic++event-queue
icon clock
10 |1500

Up to 2 attachments (including images) can be used with a maximum of 512.0 KiB each and 1.0 MiB total.

Upvotes
Accepted
1.2k 23 28 42

EventQueue::destroy() must not be called until all active streams have been completed. This is why the cleanup() function is organised as follows:

// prevent new events from being added to the queue
_pEventQueue->deactivate();
// destroy all event consumers
...
// finally destroy the event queue
_pEventQueue->destroy();

If all the event consumers have not terminated, or events are being used in multiple threads without synchronisation then the EventQueue::destroy() will exhibit undefined behaviour.

It can be beneficial to wrap RFA objects with smart pointers and check the reference count before attempting to destroy.

assert (event_queue_.use_count() <= 1);
event_queue_.reset();

I also like to use a leak tracker around objects with RFA references to provide useful feedback on the shutdown sequence.

icon clock
10 |1500

Up to 2 attachments (including images) can be used with a maximum of 512.0 KiB each and 1.0 MiB total.

This situation is not happening all the time, my test case is the same but it sometimes unexpectedly halts. Will it take time to destroy if the streaming runs for a period of time? e.g. If the streaming has run for a day, it will take 15 minutes to destroy in order to fully exit the program.

@Haseuser destroy will only take time if the application has been a slow consumer and buffered many thousands of messages that each need to be inspected. By monitoring the application memory usage one should be able to determine whether the application has been building up a backlog due to insufficient processing performance.

Upvotes
45.2k 103 43 60

This may relate to the number of remaining events in the event queue and the size of memory used by the application.

From our test, we have modified StarterConsumer to be a slow consumer. Therefore, the number of remaining events in the event queue was around 79000 events and the size of memory was growing to around 6 GB.

When cleaning up, it took around 8 minutes to exit the process.

.....

Dispatch Return: 79878

Dispatch Return: 79877

Dispatch Return: 79876

Dispatch Return: 79875

********************** INFO ************************ INFO **********************

2016 Apr 13 14:44:44.934 ST GMT+07:00 2634 26FC 332 INFO

StarterConsumer Aborted (Control C) after attempting to dispatch msgs for 14

2 seconds

Exiting!.

2016 Apr 13 14:44:53.866 ST GMT+07:00 2634 26FC 629 TRACE ::cleanup().

_pEventQueue->deactivate() is called.....##### 7:44:54.198 #####

_pEventQueue->deactivate() return.....##### 7:44:54.216 #####

_pEventQueue->destroy(); is called.....##### 7:45:22.601 #####

_pEventQueue->destroy();return........##### 7:48:26.315 #####

Context::uninitialize() is called...##### 7:48:27.529 #####

Context::uninitialize() returns...##### 7:52: 8.271 #####

********************** INFO ************************ INFO **********************

2016 Apr 13 14:52:08.272 ST GMT+07:00 2634 26FC 138 INFO

Exiting StarterConsumer - Closing log file: StarterConsumer_9780.log.

Press any key to continue . . .

......

You may need to verify the total memory used by the application. If it is very high, the application may be a slow consumer and it will take time to clean up.


mem.png (43.8 KiB)
icon clock
10 |1500

Up to 2 attachments (including images) can be used with a maximum of 512.0 KiB each and 1.0 MiB total.