question

Joe.Ferraro avatar image
11 7 7 14

Segmentation fault in runReactorWorker

I've discovered a situation that causes my program to (sometimes) terminate with segment fault during the reconnection logic.

The code gets in the SEGV somewhere during the processing of runReactorWorker function, in the shared library: librsslVA.so

Since the library was shipped without debug symbols, I had to create my own version of the .so from the source included in the Impl directory in the distribution. (1.0.6)

The problem reproduces easily with the version compiled locally including debug symbols.

The problem shows up at line 614 of rsslReactorWorker.c

The line of source causing the problem is:

if (pReactorChannel->reactorChannel.pRsslChannel->socketId != REACTOR_INVALID_SOCKET)

When the problem occurs, pReactorChannel->reactorChannel.pRsslChannel is 0.

By looking at the logs in the Elektron Edge Device, I can see that the program was disconnected as a slow consumer (the output buffer had an overflow).

Elsewhere in the logic in RsslReactorWorker.c, when in the processing logic forRSSL_RC_CET_CHANNEL_DOWN_RECONNECTING, the following line appears:

pReactorChannel->reactorChannel.pRsslChannel = 0;

This is line 529, immediately following a call to close the channel.

That a SEGV does NOT occur every time a reconnect is processed indicates that there is some soft of timing issue here, that makes the error difficult to find.

The fix may be as simple as checking if pReactorChannel->reactorChannel.pRsslChannel is 0 before retrieving the socketId

(I've just had a look at the 1.0.7 version of the source to see if the problem is still present. In the newer version, the source the line of the SEGV is identical).

elektronrefinitiv-realtimeelektron-sdkrrteta-apielektron-transport-apirssl
10 |1500

Up to 2 attachments (including images) can be used with a maximum of 512.0 KiB each and 1.0 MiB total.

Jirapongse avatar image
45.2k 103 43 60

@Joe.Ferraro

I can replicate the issue in our environment. Next, I will refer the case to the development team to verify the problem.

10 |1500

Up to 2 attachments (including images) can be used with a maximum of 512.0 KiB each and 1.0 MiB total.

Joe.Ferraro avatar image
11 7 7 14

Any update here -- I see that version 1.0.7 is still the latest version.

10 |1500

Up to 2 attachments (including images) can be used with a maximum of 512.0 KiB each and 1.0 MiB total.

Joe.Ferraro avatar image
11 7 7 14

I've found a second bug causing a crash on line 498 of the same routine.

I've temporarily fixed this by adding changing the logic to:

case RSSL_RC_CET_CHANNEL_DOWN:
{
  /* Make sure descriptors are cleared */

  if (pReactorChannel && pReactorChannel->reactorChannel.pRsslChannel &&
      pReactorChannel->reactorChannel.pRsslChannel->socketId != 
      REACTOR_INVALID_SOCKET)
  {
    FD_CLR(pReactorChannel->reactorChannel.pRsslChannel->socketId, 
            pReactorWorker->readFds);
    FD_CLR(pReactorChannel->reactorChannel.pRsslChannel->socketId, 
           pReactorWorker->exceptFds);
    FD_CLR(pReactorChannel->reactorChannel.pRsslChannel->socketId,
           pReactorWorker->writeFds);
  }


  /* Remove channel from worker's list */
  _reactorWorkerMoveChannel(&pReactorWorker->inactiveChannels,
                            pReactorChannel);
  break;
}

We cannot deploy our code to production with a patched 3rd party library.
We are awaiting your fixed version of the reactor

10 |1500

Up to 2 attachments (including images) can be used with a maximum of 512.0 KiB each and 1.0 MiB total.

tina.murthy avatar image
16 0 0 1

Joe,

Can you please raise this case with TRDC

10 |1500

Up to 2 attachments (including images) can be used with a maximum of 512.0 KiB each and 1.0 MiB total.

Joe.Ferraro avatar image
11 7 7 14

I believe it already has been, Case: 04986557

@tina.murthy

10 |1500

Up to 2 attachments (including images) can be used with a maximum of 512.0 KiB each and 1.0 MiB total.

Jirapongse avatar image
45.2k 103 43 60

This issue has been fixed in ETA 3.1.0.L1 in the Elektron-SDK 1.1.0 package.


--------------------------------
eta3.1.0.L1.win.rrg
--------------------------------
…
[ETA-2613] - CaseId: 04986557 Reactor crashes when connection is down

You can download it from here.
10 |1500

Up to 2 attachments (including images) can be used with a maximum of 512.0 KiB each and 1.0 MiB total.