TREP: Unexpected "Output threshold breached for <user>...."

philippe.mesmeur

Hi,

I developed a JAVA application based on Refinitiv Realtime SDK, that is consuming RICs on TREP.

Today, one of my customer reporting an issue that he was able to observe many time during the night:

• my application is disconnected from TREP
• ADS logs shows errors such as Output threshold breached for XXX" at position "12.23.45.78/abcdef" on host "thehost" and port number "12345" using application "256"

According to what I can read, it seems that my application does not consum RICs fast enough. However:

• at this time, the activity is not so huge (client think "only" around 1000 RICs per seconds)
• but during the day, the application is able to work easily with an activity (according to the client) around 4000 RICs per seconds

So there is apparently no reason why my application would not be able to consum the stream

According to your knowledge and your experience, what could be the reason of the error my client is having?

Is it possible to tune the "EmaConfig.xml" (or my java code) in order to avoid this error (or at least to reduce the risk it occurs)

Thank you

Regards,

Philippe

Jirapongse

@philippe.mesmeur

Thank you for reaching out to us.

Typically, this error indicates that the output buffers used by this connection breached. It could be a slow consumer or network issue. You may increase the number of output buffers via the ADS configurations. The current settings may be too low.

!++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
!
! *ads*"${port}"*guaranteedOutputBuffers : 200
!#help
!    guaranteedOutputBuffers: { 1 - N buffers }
!
!    The number of output buffers guaranteed per channel.
!
!    Default value: 24
!#end
!++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
!
! *ads*"${port}"*maxOutputBuffers : 400
!#help
!    maxOutputBuffers: { 1 - N buffers }
!
!    The maximum number of output buffers to allocate per channel.
!    The (maxOuputBuffers - guaranteedOutputBuffers) defines the number of
!    buffers to be dynamically created to be requested as needed from the pool
!    per channel. Mounts will initially be allocated the guaranteedOutputBuffers
!    and can take more from the buffer pool as needed until they reach the
!    maxOutputBuffers limit.
!
!    Default value: 48
!#end
!++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++

However, to investigate this issue on the application, you may need to contact the API support team directly via Contact Premium Support.

philippe.mesmeur

Hi

Our problem regarding the buffer overflow is not yet solved.

By adding logs in our application, we realized few things:

• the problem always occurs at midnight UTC
• the minute before the issue (i.e. before midnight UTC) the number of updates per seconds strangely starts to grow
• just before the overflow, many RICs are updated many times per seconds
  • client has between 14000 and 16000
  • I strongly suspect that many updates are identical
  • one minute before the throughput was around 1000 updates per seconds ⇒ it suddenly grows to more than 10.000 updates per seconds!!!

Can you please help me to understand what happens

Regards,

Philippe

Jirapongse

@philippe.mesmeur

According to the information, it could be a slow consumer that causes the buffer overflow on the server side.

EMA is used to subscribe to items and retrieve data from the server. You may need to contact the content support team directly via MyAccount to verify the retrieved content.

Did you subscribe to an item more than once? For example, if the application subscribes to IBM.N twice, EMA will fanout a retrieved update message so it means that the application will receive the same update message twice.

Can share the sample RICs, identical updates, and timestamps (including timezone)?

I can check the data from our Tick History database.

philippe.mesmeur

Hello

According to the information, it could be a slow consumer that causes the buffer overflow on the server side.

really? our feeling is that the consumer (i.e. my software) is flooded by many events. OK my software does not succeed to consume them fast enough but for me the real question is "why so many events and overall why exactly at midnight UTC?"

Did you subscribe to an item more than once? For example, if the application subscribes to IBM.N twice, EMA will fanout a retrieved update message so it means that the application will receive the same update message twice.

My application is not supposed to subscribe twice a market data but I will investigate in this direction; thank you for the advice

Can share the sample RICs, identical updates, and timestamps (including timezone)?

OK, when I'll have time I'll build a sample and we'll try to run it on client site

I can check the data from our Tick History database.

what to you mean? I don't understand?

Thank you for helping

Best regards,

Philippe

Jirapongse

https://community.developers.refinitiv.com/discussion/comment/125874#Comment_125874

Why so many events and overall why exactly at midnight UTC?

You need to contact the content support team to verify this behavior.

If I know RICs, data, and times, I can check if the data was really from the data feed. I can access the Tick History database to verify the data.