question

Upvotes
Accepted
ajish.thomas avatar image
1 ●1 ●1 ●2

IDLE Sockets and File descriptors on RFA 7.2

We are seeing high IDLE sockets and Open file descriptors in our application after the upgrade to JDK 8 and Solaris 11. It looks to be from below RFA library. Let us know a suggestion as workaround or a fix to address this issue.

We are using - rfa-7.2 version


Stack of RFA opening the channel but not closing it :


[4/18/22 12:57:08:046 EDT] 00000126 SystemErr R TS008852477: Mon Apr 18 12:57:08 EDT 2022: jRfaDacsLib Channel ID : 1645248514 FD : 11824: SocketChannelImpl(sp)::construtor() invoked :

at sun.nio.ch.SelectorProviderImpl.openSocketChannel(SelectorProviderImpl.java:60)

at java.nio.channels.SocketChannel.open(SocketChannel.java:145)

at com.reuters.rfa.dacs.jSipcLib.createSocketChannel(jSipcLib.java:591)

at com.reuters.rfa.dacs.jSipcLib.c_bind(jSipcLib.java:395)

at com.reuters.rfa.dacs.jDacsLib.DACS_ClientStart(jDacsLib.java:979)

at com.reuters.rfa.dacs.jDacsLib.DACS_ClientStart(jDacsLib.java:777)

at com.reuters.rfa.dacs.jDacsLibWrapper.DacsConnectRequest(jDacsLibWrapper.java:225)

at com.reuters.rfa.dacs.jDacsLibWrapper.FDacsCommand(jDacsLibWrapper.java:64)

at com.reuters.rfa.dacs.AuthorizationDacsLib.FDacsCommandEx(AuthorizationDacsLib.java:56)

at com.reuters.rfa.dacs.AuthorizationDacsLib.run(AuthorizationDacsLib.java:1441)

at java.lang.Thread.run(Thread.java:748)

------------------------------------------------


Thread Dump Stack trace:


"SSLNamespace::P2PSTest Session EventQueueGroup" #390 daemon prio=5 os_prio=64 tid=0x000000010bc02000 nid=0x187 runnable [0xffffffff39d8e000]

java.lang.Thread.State: RUNNABLE

at sun.nio.ch.PollArrayWrapper.poll0(Native Method)

at sun.nio.ch.PollArrayWrapper.poll(PollArrayWrapper.java:115)

at sun.nio.ch.PollSelectorImpl.doSelect(PollSelectorImpl.java:87)

at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)

- locked <0x000000072f40f958> (a sun.nio.ch.Util$3)

- locked <0x000000072f40f968> (a java.util.Collections$UnmodifiableSet)

- locked <0x000000072f40f900> (a sun.nio.ch.PollSelectorImpl)

at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)

at com.reuters.mainloop.channel.ChannelMainLoop.selectFor(Unknown Source)

at com.reuters.mainloop.channel.ChannelMainLoop.run(Unknown Source)

at com.reuters.rfa.internal.common.EventQueueMLThread.runImpl(Unknown Source)

at com.reuters.rfa.internal.common.InterruptibleThread.run(Unknown Source)

at java.lang.Thread.run(Thread.java:750)



rfajavaDACSsolarisend-of-life
icon clock
10 |1500

Up to 2 attachments (including images) can be used with a maximum of 512.0 KiB each and 1.0 MiB total.

wasin.w avatar image   wasin.w

<AHS>

The question originated from ticket 11540901. The TSG support team directed the client to post a question here.

The client's company already canceled all RDC members.

Problem statement:

Our Brokerage JVMs are holding these IDLE sockets and it’s adding up more number of open file descriptors. From the investigation these sockets were holding by Refinitiv library. (See Attached)

We would like to know from Refinitiv, if they ran into this situation before and if they have any solution.


IBM L3 Response:
Taking a leaked FD 11824 from pfiles output , we could observe from the below relevant stacks that this FD firstly assigned to socket channel ID 1316123199 which is opened by WAS at time 12:57:03:724 , later the WAS calling close on this socket channel ID at around same time , thus releasing this FD to be used by other process , in this case the FD is assigned to socket channel ID 1645248514 which is create by RFA library at time 12:57:08:046 , and this socket channel never gets closed . Similar pattern is observed for most of the leaked FDs. So rfa is the last process holding these FDs as seen from standard error log. We expect close on this socket channel to be called by rfa so that FD can be released .
wasin.w avatar image   wasin.w

Hello @ajish.thomas

Thank you for your participation in the forum. Is the reply below satisfactory in resolving your query?


If so please can you click the 'Accept' text next to the appropriate reply? This will guide all community members who have a similar question.

Thanks,


AHS

raksina.samasiri avatar image   raksina.samasiri

<AHS>

extend triage for a week base on client's comment that they will get back within next week

wasin.w avatar image   wasin.w

<AHS>

extend triage for a week base on client's comment that they will get back within next week

aramyan.h avatar image   aramyan.h

Hi,

Please be informed that a reply has been verified as correct in answering the question, and marked as such.

Thanks,

AHS

9 Answers

· Write an Answer
Upvotes
Accepted
wasin.w avatar image
24.7k ●54 ●17 ●14

Hello @ajish.thomas

Please be informed that the RFA Java API page requires RDC named user/Premium user account to access the API resources such as documents, tutorials, the API packages, etc.

Alternatively, you can download the RFA Java API package from the https://my.refinitiv.com/content/mytr/en/downloadcenter.html page, and get RFA Java documents from the package as mentioned by @Jirapongse. Please choose the following search condition for the RFA Java version 7.

  • Product family: MDS - API
  • Products: Robust Foundation API (RFA) - Java

rfa-java-download.png

You can verify if you are RDC named user/Premium user by logging in to the Developer Portal website, then go to the RFA Java API page and see if the "Contact premium support" button is presented.

rfa-java-premium.png

RDC named user/Premium user can download the API resources and can submit a support ticket to a dedicated Real-Time API support team to investigate the Real-Time API queries with SLA. If you are interested in the RDC named user/Premium user program, please contact your Refinitiv representative.

Hope this helps


rfa-java-download.png (55.6 KiB)
rfa-java-premium.png (99.4 KiB)
icon clock
10 |1500

Up to 2 attachments (including images) can be used with a maximum of 512.0 KiB each and 1.0 MiB total.

Upvotes
Jirapongse avatar image
79.2k ●251 ●52 ●74

@ajish.thomas

Please be informed that RFA Java 7.2 has been end-of-life since 2014. The migration path is RFA Java 7.7 or above.

You can download it from https://developers.refinitiv.com/en/api-catalog/refinitiv-real-time/robust-foundation-api-rfa-java/downloads or the Software Downloads. It is in Category: MDS - API and Products: Robust Foundation (RFA) - Java.

1661914594737.png

Moreover, please refer to the API Compatibility Matrix for the supported operating systems and compilers. It doesn't support Solaris 11 anymore.

1661914728881.png




1661914594737.png (11.7 KiB)
1661914728881.png (27.2 KiB)
icon clock
10 |1500

Up to 2 attachments (including images) can be used with a maximum of 512.0 KiB each and 1.0 MiB total.

Upvotes
ajish.thomas avatar image
1 ●1 ●1 ●2

Hello @Jirapongse

Thank you for the details.

Could you please share the compatibility matrix for JDK 1.8 + Solaris 11 OS as well?


icon clock
10 |1500

Up to 2 attachments (including images) can be used with a maximum of 512.0 KiB each and 1.0 MiB total.

Upvotes
ajish.thomas avatar image
1 ●1 ●1 ●2

I just got a response from the team as this same behaviour observed in RFA 7.7 as well. Could you please help to understand the socket open issue reported as a bug in any 7.X version and it's addressed in any latest versions of RFA?

icon clock
10 |1500

Up to 2 attachments (including images) can be used with a maximum of 512.0 KiB each and 1.0 MiB total.

Upvotes
Jirapongse avatar image
79.2k ●251 ●52 ●74

@ajish.thomas

From the call stack, it is a connection to a DACS daemon. RFA can directly connect to a DACS daemon in order to perform entitlement checks. However, when connecting to a server, such as ADS, the server can perform entitlement checks on behalf of the application. Therefore, you can disable entitlement checks in RFA Java by using the method mentioned in the README file.

5.5.1 Disabling Entitlements for MarketData Connections


    In some deployments, applications may be using data that is not 
    fee-liable (e.g. it is internal data or delayed). In other 
    circumstances, data may be entitled elsewhere, such as the 
    P2PS or using OpenDACS. To prevent RFA from attempting to connect
    to a DACS daemon, all DACS configuration variables must be set to 
    false, including:
      dacs_CbeEnabled
      dacs_SbePubEnabled
      dacs_SbeSubEnabled
      dacs_GenerateLocks
      dacs_NewsCbeEnabled
    Please refer to the RFA Java Configuration & Logging Guide for more
    details about these configuration variables. Also, the application 
    should not register for EntitlementsAuthenticationEvents using any
    of the following:
      MarketDataSubscriberInterestSpec.setEntitlementsInterest()
      MarketDataContributorInterestSpec.setEntitlementsInterest()
      MarketDataUnmanagedPublisherInterestSpec.setEntitlementsInterest()

If this still doesn't solve the issue, please contact the API support team (RDC) directly via Contact Premium Support. The API support team will investigate this issue or contact the development team to fix the problem.

RDC (Refinitiv Developer Connect) is a subscription-based service so only the RDC contacts can raise a query to the API support team.

If you would like to join the RDC service, please contact your Refinitiv account team.

icon clock
10 |1500

Up to 2 attachments (including images) can be used with a maximum of 512.0 KiB each and 1.0 MiB total.

ajish.thomas avatar image ajish.thomas
Could you please share the API reference guide Article link/PDF to review the changes we are making in the code.
Jirapongse avatar image Jirapongse ajish.thomas

@ajish.thomas

RFA Java documents are available in the RFA Java package.

1663138382194.png

1663138382194.png (36.8 KiB)
wasin.w avatar image wasin.w ajish.thomas

Hello @ajish.thomas

Additionally, you can download the RFA Java relate documents from https://developers.refinitiv.com/en/api-catalog/refinitiv-real-time/robust-foundation-api-rfa-java/documentation#rfa-java-development-guides page (you need to log in to our Developer Portal website with the RDC named user/premium user account).
rfa-java-doc.png

rfa-java-doc.png (88.2 KiB)
Upvotes
ajish.thomas avatar image
1 ●1 ●1 ●2

We are in the process of testing above suggestion on a lower environment. We will get back to you in 2 week with the results/further updates.

icon clock
10 |1500

Up to 2 attachments (including images) can be used with a maximum of 512.0 KiB each and 1.0 MiB total.

Upvotes
ajish.thomas avatar image
1 ●1 ●1 ●2

screen-shot-2022-09-14-at-35111-pm.png

Kindly provide a direct link or PDF zip to download. I couldn't find the documents in above link


screen-shot-2022-09-14-at-35111-pm.png (152.4 KiB)
icon clock
10 |1500

Up to 2 attachments (including images) can be used with a maximum of 512.0 KiB each and 1.0 MiB total.

Upvotes
ajish.thomas avatar image
1 ●1 ●1 ●2

Thank you for the above informations and directions. We delivered the change on a lower environment and not seeing the file descriptor issues now.

The testing cycles are still in progress so we are yet to conclude the resolution.


icon clock
10 |1500

Up to 2 attachments (including images) can be used with a maximum of 512.0 KiB each and 1.0 MiB total.

ajish.thomas avatar image ajish.thomas
We deployed the change on production last weekend and we see it's fixed the open file descriptor issue/FD leak. It is a good news. But we received a concern as there is delay in quote refresh(Symbol delete from the database is not reflecting immediately on the front end application, it reflected only on the next morning). Could you please check and confirm does the above change can cause this problem?


Upvotes
ajish.thomas avatar image
1 ●1 ●1 ●2

We are seeing the benefits on our production servers after applying the suggested change. No more open file descriptor leak now.

However we noticed a difference in quote refresh interval(Normally when I delete a symbol from the DB its effects are seen on Front end app immediately, but after the above change it reflects only on next day morning). Please check and let us know if this is expected after the above change ?

icon clock
10 |1500

Up to 2 attachments (including images) can be used with a maximum of 512.0 KiB each and 1.0 MiB total.

Write an Answer

Hint: Notify or tag a user in this post by typing @username.

Up to 2 attachments (including images) can be used with a maximum of 512.0 KiB each and 1.0 MiB total.