RFA Daily Reset

esawin

Hi,

As part of our process we have a nightly reset to clear caches and reset everything to "fresh". I have tried incorporating a call to Cleanup (included below) followed by init and run but am running into a whole host of issues that randomly pop up, usually during init, common exception is 'System.AccessViolationException'.

My goal here is just to stop the stream of data and clear out subscriptions for the start of a new day. Is this the right approach?

if (ommConsumer != null)
{
     if (loginHandle != 0)
     {
          ommConsumer.UnregisterClient(loginHandle);
          loginHandle = 0;
     }

     if (ommConnIntSpecHandle != 0)
     {
           ommConsumer.UnregisterClient(ommConnIntSpecHandle);
           ommConnIntSpecHandle = 0;
     }

      ommConsumer.Destroy();
      ommConsumer = null;
}


if (session != null)
{
     session.Release();
     session = null;
}


if (eventQueue != null)
{
      eventQueue.Deactivate();
      eventQueue.Destroy();
      eventQueue = null;
}


if ((!Context.Uninitialize()) && (Context.InitializedCount == 0))
{
}

zoya faberov

Hi @esawin,

You should be able to run Context.Uninitialize without a crush. The crush, typically, indicates that something was not cleaned up properly, as you have originally supposed.

Is it possible the cleanup is forced before all the streams are properly closed, and an update arrives when the queue is already destroyed? This is where I would check, would make sure that all the streams are closed prior. As already you have verified the order of the cleanup and it is correct.

The other indicator is that it's not reproducing consistently, it appears to be timing dependent.

---

I have modified the Consumer example code to do the rotation 40 times, as a simple test. Was not able to reproduce any cleanup issues as a result.

My modifications from the previous versions were as follows:

in Consumer.cfg, I limited runtime to 10 seconds time

   \ConsumerClient_10\runTimeInSeconds     = 10

In Program.cs, I increased the loop to 40 iterations, and added a distinctive print

for (int i = 0; i < 40; i++) {
     Application app = new Application();
...
 finally {
      app.Cleanup();
      Console.WriteLine("#Iteration #"+i+"#");
 }

In Application.cs I have modified the run loop to terminate based on time

public bool terminateOnTime;
public Application() {
...
	terminateOnTime = false;
}
...     
public void Run() {
    while (!CtrlBreakHandler.IsTerminated() && !terminateOnTime)     {
            eventQueue.Dispatch(10);
            //check to see if any Consumers wish to be removed
            CheckConsumers();
    }
}

In Consumer.cs, making sure we pass the flag to terminate based on time:

else {
	AppUtil.Log(AppUtil.LEVEL.INFO, string.Format("{0}: Exited (RunTime expired) after attempting to dispatch msgs for {1} seconds\r\n    Exiting!", consumerName, (System.DateTime.Now - startTime).TotalSeconds));
	theApp.terminateOnTime = true;
}

Thereafter, I run the loop 40 times and cleanup with no crush.

If you wish, I can send the complete code download link to your email address, so you could test it in your environment.

zoya faberov

Hello @esawin,

The order of cleanup looks slightly off. Not sure if all item requests get unregistered before the consumer is destroyed. Queue should be cleaned up before session.

To be on the safe side, would follow the order from starterConsumer example that came with SDK:

 if ( _connIntSpecHandle != null )
        _ommConsumer.unregisterClient(_connIntSpecHandle);

        // unregister all items
        if (_itemManager != null)
            _itemManager.closeRequest();

        // unregister login
        // Allow conditional login unregister.
        // loginCleanup flag is set to false if cleanup is called after login
        // failure.
        if (_loginClient != null && doLoginCleanup)
            _loginClient.closeRequest();

        if (_ommConsumer != null)
            _ommConsumer.destroy();

        _eventQueue.deactivate();
        _session.release();
        Context.uninitialize();  

esawin

hi @zoya faberov

I have made those changes but not seeing any difference. Most of my code related to init, start, and cleanup are all pretty much untouched from the tutorial. To take this 1 step farther I loaded up the tutorial (RFA .Net version, step 7) and with a few tweaks (moved logger/decoder out of init so it's only called once) been able to reproduce the same results.

To clarify, for testing the reset function I put a 1 min timer which calls cleanup, then init, and run. Not subscribing to anything in these examples.

I've got a hunch this is related to async calls, but I'm not sure which. When calling init right after cleanup it crashes usually on first reset, but if i add a wait after the cleanup it will successfully reset multiple times (varies based on wait time), but then it always does end up failing after some time.

zoya faberov

Hi @esawin,

I have tried a little experiment too. I have taken, from RFA.NET 8 SDK, example Consumer (simpler then tutorial, no applogger), and introduced a tight loop of 5 iterations:

public static void Main(string[] args)
        {
            for (int i = 0; i < 5; i++)
            {
                Application app = new Application();
                try
                {
                    bool fileConfig = !args.Contains("-r");
                    bool result = app.Init(fileConfig);
                    if (result)
                    {
                        app.Run();
                    }
                }
                catch (ThomsonReuters.RFA.Common.Exception exception)
                {
                    ExceptionHandler.CheckException(exception);
                }
                finally
                {
                    app.Cleanup();
                }
            }  
        }

I am able to Ctrl+C the example (triggering cleanup) 5 times, and after that it exists the loop gracefully. It does init and cleanup, repeatedly, 5 times.

But this is a truly bare bones example with a single subscription.

I suspect that some processing taking place in other parts of your app, may be preventing a "clean" cleanup.

Would it be possible to pare down the app to bare bones, start when you clean properly, and introduce back the functionality gradually, to see what does not clean?

esawin

Hi @zoya faberov

That was it! In my examples I was calling "Application app = new Application()" only once at the initial app startup. Using this approach does work in both the tutorial and my app, will use this going forward.

Thanks!

esawin

Hi @zoya faberov

I spoke too soon with my last post. Re-initializing only seemed to make the crash occur less often, sometimes taking 20-30 resets before it crashed.

I've continued to look into this and found not calling 'Cleanup' (but still reinitializing and calling init/run) resulted in no more crashes (tested with 100+ resets), but this didn't unsubscribe so i tried adding parts of 'Cleanup' back in. In the end it came down to 'Context.Uninitialize', with this in I would crash usually within 5 resets but taken out I've tested 200+ resets without a crash. So what does that call do? I have yet to find any side effect from removing it.