While using Java library ema v3.8.2.0, I'm getting java.lang.OutOfMemoryError in CryptoHelper.java

HemantKumar1
HemantKumar1 Newcomer
edited March 21 in EMA

While using Java library ema v3.8.2.0, I'm getting following error in CryptoHelper.java of corresponding eta jar v3.8.2.0 :

java.lang.OutOfMemoryError: Cannot reserve 66816 bytes of direct buffer memory (allocated: 524258534, limit: 524288000)at java.base/java.nio.Bits.reserveMemory(Bits.java:178)at java.base/java.nio.DirectByteBuffer.<init>(DirectByteBuffer.java:121)at java.base/java.nio.ByteBuffer.allocateDirect(ByteBuffer.java:332)at com.refinitiv.eta.transport.CryptoHelper.initializeEngine(CryptoHelper.java:321)at com.refinitiv.eta.transport.EncryptedSocketHelper.postProxyInit(EncryptedSocketHelper.java:90)at com.refinitiv.eta.transport.EncryptedSocketHelper.initialize(EncryptedSocketHelper.java:62)at com.refinitiv.eta.transport.RsslSocketChannel.setChannelAccept(RsslSocketChannel.java:3169)at com.refinitiv.eta.transport.SocketProtocol.channel(SocketProtocol.java:307)at com.refinitiv.eta.transport.ServerImpl.accept(ServerImpl.java:417)at com.refinitiv.eta.valueadd.reactor.Reactor.accept(Reactor.java:602)at com.refinitiv.ema.access.OmmServerBaseImpl.rsslReactorDispatchLoop(OmmServerBaseImpl.java:1247)at com.refinitiv.ema.access.OmmServerBaseImpl.dispatch(OmmServerBaseImpl.java:1717)at com.refinitiv.ema.access.OmmIProviderImpl.dispatch(OmmIProviderImpl.java:1308)at com.markit.n6platform.adapters.rfa.RFAServer.init(RFAServer.java:97)at com.markit.n6platform.adapters.rfa.RFAServer.lambda$start$0(RFAServer.java:127)at java.base/java.lang.Thread.run(Thread.java:840)

Can you please have a look and advise?

Tagged:

Answers

  • Gurpreet
    Gurpreet admin

    Hello @HemantKumar1

    Can you please tell us more about your application? How many instruments did it subscribed to and was it during the period of high market activity. Is this a recurring issue? If you are connected to a local RTMDS, what logs do you get in the ADS.

    Typically, out of memory exception can happen in the case of a slow consumer - i.e. a consumer client which is unable to process incoming messages fast enough causing the internal buffers in the consumer and server to fill up.

    From the exception logs it also appears that the application is running within a container. Sometimes the containers can temporarily restrict the resources available to the application - especially if the app is running in a server-less lambda function.

    Are you able to replicate this issue on a real machine, running as an application?

  • Jirapongse
    Jirapongse ✭✭✭✭✭

    @HemantKumar1

    According to the call stack, the application is an interactive provider.

    The exception is the out of memory when reserving the direct buffer memory which is limited to 524,288,000 bytes. I am not sure how many consumers and items are handled by this application.

    I am not sure why it is limited to around 500MB. The application may be running in a container as mentioned by my colleague. However, if you are running the application on a real machine, I found the way to increase the size of the direct buffer memory on StackOverflow and this page.

  • wasin.w
    wasin.w admin

    This issue has been tracked as Issue - #305 on the https://github.com/Refinitiv/Real-Time-SDK GitHub repository.

  • HemantKumar1
    HemantKumar1 Newcomer

    Hi Gurpreet & Jiraponge,

    This is a Java application that is running in a docker container. It has 8 GB of heap space. And the direct memory is set to 500 MB.

    There are multiple instances of the application that are running with same configuration but in different docker containers and with different URLS and Ports.

    Different instances of the application receive different count of Symbols (one example of Symbols is Bloomberg ISIN) as part of subscription requests. This ranges from around 100 for one instance of application to up to 8000 for some other instance of the application.

    The clients connect using Logon request. Once Logon is successful, the clients send in subscription request which has Symbols (Items) to our application. Our application works like a Server to those clients.

    This problem is observed in all the instances regardless of the count of Symbols subscribed for. It is occurring in QA, UAT and PROD env. The problem is that it can work fine for a few days, weeks or even months and then suddenly one day we have this issue. So, it is recurring issue, but there no definite steps to reproduce it. Repeated efforts to reproduce it in DEV env has not yielded any success.

    The docker container is Deployed on cloud dedicated machine and have a limit on how much max memory we can use. Moreover, it has been observed that the Direct Memory usage remain very low for most of the time when the clients connect to receive data for Symbols they subscribe for. See below image.

    image.png

    The number of Update Messages send in one instance of the application can be in double digit count per minute. And in another instance, it can be > 25000 updates per minute.

    As we can see from the graph above, even increasing the Direct Memory will not solve the problem as it seems it that once the problem starts, any amount of memory allocation to Direct Memory is insufficient.

    There are a lot of logs, but if you could guide to certain unique Keywords to look for, in the logs then I can search for them.

    If you need any other details, please let me know.

  • Jirapongse
    Jirapongse ✭✭✭✭✭

    First of all, as far as I know, the Real-Time SDK APIs (EMA) doesn't qualify on Docker.

    You can check the API Compatibility Matrix for the supported operating systems and compilers.

    Moreover, you have submitted this issue on GitHub so we need to wait for a response from the product team.

Categories