TREP: Unexpected "Output threshold breached for <user>...."

philippe.mesmeur

Hi,

I developed a JAVA application based on Refinitiv Realtime SDK, that is consuming RICs on TREP.

Today, one of my customer reporting an issue that he was able to observe many time during the night:

• my application is disconnected from TREP
• ADS logs shows errors such as Output threshold breached for XXX" at position "12.23.45.78/abcdef" on host "thehost" and port number "12345" using application "256"

According to what I can read, it seems that my application does not consum RICs fast enough. However:

• at this time, the activity is not so huge (client think "only" around 1000 RICs per seconds)
• but during the day, the application is able to work easily with an activity (according to the client) around 4000 RICs per seconds

So there is apparently no reason why my application would not be able to consum the stream

According to your knowledge and your experience, what could be the reason of the error my client is having?

Is it possible to tune the "EmaConfig.xml" (or my java code) in order to avoid this error (or at least to reduce the risk it occurs)

Thank you

Regards,

Philippe

Jirapongse

@philippe.mesmeur

Thank you for reaching out to us.

Typically, this error indicates that the output buffers used by this connection breached. It could be a slow consumer or network issue. You may increase the number of output buffers via the ADS configurations. The current settings may be too low.

!++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
!
! *ads*"${port}"*guaranteedOutputBuffers : 200
!#help
!    guaranteedOutputBuffers: { 1 - N buffers }
!
!    The number of output buffers guaranteed per channel.
!
!    Default value: 24
!#end
!++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
!
! *ads*"${port}"*maxOutputBuffers : 400
!#help
!    maxOutputBuffers: { 1 - N buffers }
!
!    The maximum number of output buffers to allocate per channel.
!    The (maxOuputBuffers - guaranteedOutputBuffers) defines the number of
!    buffers to be dynamically created to be requested as needed from the pool
!    per channel. Mounts will initially be allocated the guaranteedOutputBuffers
!    and can take more from the buffer pool as needed until they reach the
!    maxOutputBuffers limit.
!
!    Default value: 48
!#end
!++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++

However, to investigate this issue on the application, you may need to contact the API support team directly via Contact Premium Support.

philippe.mesmeur

Hi

Our problem regarding the buffer overflow is not yet solved.

By adding logs in our application, we realized few things:

• the problem always occurs at midnight UTC
• the minute before the issue (i.e. before midnight UTC) the number of updates per seconds strangely starts to grow
• just before the overflow, many RICs are updated many times per seconds
  • client has between 14000 and 16000
  • I strongly suspect that many updates are identical
  • one minute before the throughput was around 1000 updates per seconds ⇒ it suddenly grows to more than 10.000 updates per seconds!!!

Can you please help me to understand what happens

Regards,

Philippe

Jirapongse

@philippe.mesmeur

According to the information, it could be a slow consumer that causes the buffer overflow on the server side.

EMA is used to subscribe to items and retrieve data from the server. You may need to contact the content support team directly via MyAccount to verify the retrieved content.

Did you subscribe to an item more than once? For example, if the application subscribes to IBM.N twice, EMA will fanout a retrieved update message so it means that the application will receive the same update message twice.

Can share the sample RICs, identical updates, and timestamps (including timezone)?

I can check the data from our Tick History database.

philippe.mesmeur

Hello

According to the information, it could be a slow consumer that causes the buffer overflow on the server side.

really? our feeling is that the consumer (i.e. my software) is flooded by many events. OK my software does not succeed to consume them fast enough but for me the real question is "why so many events and overall why exactly at midnight UTC?"

Did you subscribe to an item more than once? For example, if the application subscribes to IBM.N twice, EMA will fanout a retrieved update message so it means that the application will receive the same update message twice.

My application is not supposed to subscribe twice a market data but I will investigate in this direction; thank you for the advice

Can share the sample RICs, identical updates, and timestamps (including timezone)?

OK, when I'll have time I'll build a sample and we'll try to run it on client site

I can check the data from our Tick History database.

what to you mean? I don't understand?

Thank you for helping

Best regards,

Philippe

Jirapongse

https://community.developers.refinitiv.com/discussion/comment/125874#Comment_125874

Why so many events and overall why exactly at midnight UTC?

You need to contact the content support team to verify this behavior.

If I know RICs, data, and times, I can check if the data was really from the data feed. I can access the Tick History database to verify the data.

philippe.mesmeur

Hi

our issue is still not solved but fortunately we have new elements and we hope we are near to the solution 🤞

(this mail contains many questions for you and I hope you'll find time to answer all of them; thank you)

• Midnight UTC is in fact the opening of the Tokyo market: this explains why we have so many events at this time
  • although I'm surprised of the quantity (up to 1.2 million per minute for "only" around 16000 RICs subscribed)
• Yet, the conflation is enabled on TREP side
  • it seems that there are two kinds of RT updates: Quote-Updates and Trade-Updates, am I right?
  • we do not yet know the difference between both but our feeling is that we are only interested by quote updates
  • our understanding of the conflation done on TREP/ADS side is that Trade-Updates skip the conflation process and immediately publish the data ⇒ can you confirm?
  • using the Java RealTime SDK, is there a possibility to subscribe ONLY for Quote-Updates? if so, please give me or redirect me to a sample
  • I also saw that when subscribing by using interface ReqMsg, it is possible to specify the QOS via method
    public ReqMsg qos(int timeliness, int rate);
    for the moment I do not use this method: what are the default values used?
    do you think that using this method may help to solve my issue (i.e. reduce the amount of updates). If so, please give me or redirect me to a sample

Another candidate for my investigation is the Java Garbage collector: I have to investigate in this direction but I suspect that the garbage collector is extremely requested when receiving and processing so many updates

According to your experience, do you have recommendation for setting up the garbage collector (e.g. JVM parameters…)

Thank you for helping

Best regards,

Philippe

Jirapongse

It seems that there are two kinds of RT updates: Quote-Updates and Trade-Updates, am I right?

There are several update types, and their usage depends on the data feed, market, or specific changes. To confirm how these update types are applied in the Tokyo market, please contact the content support team.

our understanding of the conflation done on TREP/ADS side is that Trade-Updates skip the conflation process and immediately publish the data ⇒ can you confirm?

As far as I know, you can configure conflated services on TREP/ADS side. You need to confirm this behavior with the ADS support team.

using the Java RealTime SDK, is there a possibility to subscribe ONLY for Quote-Updates? if so, please give me or redirect me to a sample

No, you can't subscribe only to quote updates using RTSDK. Instead, you need to use a view request that subscribes to only quote fields, such as BID and ASK. Please refer to the ex360_MP_View example.

I also saw that when subscribing by using interface ReqMsg, it is possible to specify the QOS via method
public ReqMsg qos(int timeliness, int rate); for the moment I do not use this method: what are the default values used? do you think that using this method may help to solve my issue (i.e. reduce the amount of updates). If so, please give me or redirect me to a sample

QoS is a property of a real-time service, so the QoS specified in the request message must match the QoS of the subscribed service. If you have a conflated service on ADS, it can help reduce the number of updates. Additionally, using view requests can minimize the size of update messages

do you have recommendation for setting up the garbage collector?

Sorry, I don't have any recommendation for the settings of the garbage collector. You can check the Real-Time SDKs Performance Test Summary and Real-Time SDK (RTSDK) APIs Performance Test Results regarding the RTSDK performance.