cleanup() halts on rfa::common::EventQueue::destroy()
After RFA C++ modified example StarterConsumer_BatchView has been running for a period of time, cleanup() halts on rfa::common::EventQueue::destroy()
How long does it take to destroy EventQueue? Is there any timeout?
Is there anything I can do to trace the destroy progress?
EventQueue::destroy() must not be called until all active streams have been completed. This is why the cleanup() function is organised as follows:
// prevent new events from being added to the queue _pEventQueue->deactivate(); // destroy all event consumers ... // finally destroy the event queue _pEventQueue->destroy();
If all the event consumers have not terminated, or events are being used in multiple threads without synchronisation then the EventQueue::destroy() will exhibit undefined behaviour.
It can be beneficial to wrap RFA objects with smart pointers and check the reference count before attempting to destroy.
assert (event_queue_.use_count() <= 1); event_queue_.reset();
I also like to use a leak tracker around objects with RFA references to provide useful feedback on the shutdown sequence.
This situation is not happening all the time, my test case is the same but it sometimes unexpectedly halts. Will it take time to destroy if the streaming runs for a period of time? e.g. If the streaming has run for a day, it will take 15 minutes to destroy in order to fully exit the program.
@Haseuser destroy will only take time if the application has been a slow consumer and buffered many thousands of messages that each need to be inspected. By monitoring the application memory usage one should be able to determine whether the application has been building up a backlog due to insufficient processing performance.
This may relate to the number of remaining events in the event queue and the size of memory used by the application.
From our test, we have modified StarterConsumer to be a slow consumer. Therefore, the number of remaining events in the event queue was around 79000 events and the size of memory was growing to around 6 GB.
When cleaning up, it took around 8 minutes to exit the process.
.....
Dispatch Return: 79878
Dispatch Return: 79877
Dispatch Return: 79876
Dispatch Return: 79875
********************** INFO ************************ INFO **********************
2016 Apr 13 14:44:44.934 ST GMT+07:00 2634 26FC 332 INFO
StarterConsumer Aborted (Control C) after attempting to dispatch msgs for 14
2 seconds
Exiting!.
2016 Apr 13 14:44:53.866 ST GMT+07:00 2634 26FC 629 TRACE ::cleanup().
_pEventQueue->deactivate() is called.....##### 7:44:54.198 #####
_pEventQueue->deactivate() return.....##### 7:44:54.216 #####
_pEventQueue->destroy(); is called.....##### 7:45:22.601 #####
_pEventQueue->destroy();return........##### 7:48:26.315 #####
Context::uninitialize() is called...##### 7:48:27.529 #####
Context::uninitialize() returns...##### 7:52: 8.271 #####
********************** INFO ************************ INFO **********************
2016 Apr 13 14:52:08.272 ST GMT+07:00 2634 26FC 138 INFO
Exiting StarterConsumer - Closing log file: StarterConsumer_9780.log.
Press any key to continue . . .
......
You may need to verify the total memory used by the application. If it is very high, the application may be a slow consumer and it will take time to clean up.