question

Upvotes
Accepted
3 0 0 1

OOM cause by com.refinitiv.eta.json.util.JsonFactory Ojbect costing large memory

Hi we met OutOfMemory issue in our service. we are using dependency below :

        <dependency>
            <groupId>com.refinitiv.eta</groupId>
            <artifactId>eta</artifactId>
            <version>3.6.2.2</version>
        </dependency>        
        <dependency>
            <groupId>com.refinitiv.ema</groupId>
            <artifactId>ema</artifactId>
            <version>3.6.2.2</version>
        </dependency>

we deployed service on our server. running sub rate from trep and posting message to trep too(every 5 mins).

after around 10 hours. OOM issue occur.

we have export the heapdump file and did some analysis.

we found that most memory used by a JsonFactory object.

1693212620622.png

for more information. big memory used by attribute in JsonFactory:

1693212704519.png

could you help provide more information about the #elementSetDefDbPool and #fieldSetDefDbPool ?

btw, we are using OmmConsumer.submit to post message. each post event we will create new OmmConsumer and release it after all AckId return backed.

PostMsg postMsg = EmaFactory.createPostMsg();
            UpdateMsg nestedUpdateMsg = EmaFactory.createUpdateMsg();
            FieldList fieldList = fieldListMap.get(ric).getKey();
            StringBuilder fieldListStringBuilder = fieldListMap.get(ric).getValue();
            nestedUpdateMsg.payload(fieldList);

            long currentPostId = postId.incrementAndGet();

            try {
                consumer
                        .submit(
                                postMsg
                                        .postId(currentPostId)
                                        .serviceName(serviceName)
                                        .name(ric)
                                        .solicitAck(true)
                                        .complete(true)
                                        .payload(nestedUpdateMsg)
                                , loginHandle
                        );



#technology#producteta-api
1693212620622.png (137.2 KiB)
1693212704519.png (55.7 KiB)
icon clock
10 |1500

Up to 2 attachments (including images) can be used with a maximum of 512.0 KiB each and 1.0 MiB total.

Upvotes
Accepted
79.2k 251 52 74

@charlven.tan

Thank you for reaching out to us.

I found the issue regarding memory leak that has been fixed in EMA Java 3.6.7.L1.

Both ETA Java and EMA Java 3.6.7.L1 Issues Resolved
---------------------------------------------------
- [RTSDK-6167] - Uncaught NPE in EMA due to race condition in cleanup after timeout [Case Number: 11147646]
- [RTSDK-6259] - Support JDK and Oracle OpenJDK 1.17
- [RTSDK-6347] - Update to Jackson jar due to vulnerabilities detected
- [RTSDK-6395] - Apache commons-configuration2 version 2.7 vulnerability [GitHub #201]
- [RTSDK-6411] - Memory leak upon repeated OMMConsumer initialize and un-intialize [Case Number: 11444081] 
- [RTSDK-6479] - Update to DACS 7.8
- [RTSDK-6531] - JSON Converter unexpected value with date of 2022-11 [GitHub #207]

Please upgrade the application to use the latest version of EMA to avoid all known issues.

The latest version is Real-Time-SDK-2.1.1.L1 (EMA 3.7.1.L1).


icon clock
10 |1500

Up to 2 attachments (including images) can be used with a maximum of 512.0 KiB each and 1.0 MiB total.

Upvotes
79.2k 251 52 74

@charlven.tan

Thank you for reaching out to us.

Please explain more about this statement.

we are using OmmConsumer.submit to post message. each post event we will create new OmmConsumer and release it after all AckId return backed.

Does it mean that the application regulary creates and releases OMMConsumer instances?

icon clock
10 |1500

Up to 2 attachments (including images) can be used with a maximum of 512.0 KiB each and 1.0 MiB total.

yes. we regulary creates and release.

I am not sure it will impact the growing of JsonFactory's elementSetDefDbPool and fieldSetDefDbPool.

Upvotes
3 0 0 1

@Jirapongse great thanks. I try to upgrade the SDK version. seems look good now, let me keep monitoring it for one week to verify if issue resolved.


Btw, I have an other question. when I restart my service, each publication event (post 500 page) and get all Ack Id back, it only spend around 2-3 seconds.

But when then service keep running about 1 days. the publication event time spend are came to 20+ seconds. each publication event also creating new OOMConsumer to trigger submit.


anything I should check with lib ? or I should re-raise a new ticked to ask QA ? many thinks.

icon clock
10 |1500

Up to 2 attachments (including images) can be used with a maximum of 512.0 KiB each and 1.0 MiB total.

Write an Answer

Hint: Notify or tag a user in this post by typing @username.

Up to 2 attachments (including images) can be used with a maximum of 512.0 KiB each and 1.0 MiB total.