delay quote problem

iscas.wang

Hello, everyone, my application encountered a strange problem on July 2nd, feed data is stopped at about 9:26 EST and re-continued at about 9:39 EST, which makes quote data of the starting 8 minutes of that market day lost.

Let me explain my application logic first. At 4:00 EST, my application will start request chain records starting with 0#UNIVERSE.NB and batch subscribe all rics from the results of the above chain records. The app will continue receive UpdateMsg until at 9:29, when it will force unsubscribe all rics and re-subscribe them again. (The app does so because of one day of last year, we subscribed .IXIC at pre-market open time but receives no data when market open. So add such force unsubscribe and re-subscribe logic). I contact the feed data team about the proble, but they found no reason.

The difference betweeen that strange day and normal day I can notice is:

On normal days, if I re-subscribe at 9:29, seen from stdout log of OnStatusMsg callback, I can found such logs:

onStatusMsg: [2020-07-09 21:29:20 240495][23779:139987978176256]
Service Name: ELEKTRON_DD
Item State: Closed / Ok / None / 'Batch request acknowledged.'

onStatusMsg: [2020-07-09 21:29:22 258829][23779:139987978176256]
Service Name: ELEKTRON_DD
Item State: Closed / Ok / None / 'Batch request acknowledged.'

onStatusMsg: [2020-07-09 21:29:22 685333][23779:139987978176256]
Item Name: KLR_t.NB
Service Name: ELEKTRON_DD
Item State: Closed / Suspect / Not found / 'The record could not be found'

onStatusMsg: [2020-07-09 21:29:24 277349][23779:139987978176256]
Service Name: ELEKTRON_DD
Item State: Closed / Ok / None / 'Batch request acknowledged.'

onStatusMsg: [2020-07-09 21:29:26 289616][23779:139987978176256]
Service Name: ELEKTRON_DD
Item State: Closed / Ok / None / 'Batch request acknowledged.'

such 'Batch request acknowledged.' messages are printed in 21:29. But on that strange day, I found such logs continues from

onStatusMsg: [2020-07-02 21:29:18 676818][26493:140211614574336]
Service Name: ELEKTRON_DD
Item State: Closed / Ok / None / 'Batch request acknowledged.'

onStatusMsg: [2020-07-02 21:29:20 677285][26493:140211614574336]
Service Name: ELEKTRON_DD
Item State: Closed / Ok / None / 'Batch request acknowledged.'

onStatusMsg: [2020-07-02 21:29:22 677726][26493:140211614574336]
Service Name: ELEKTRON_DD
Item State: Closed / Ok / None / 'Batch request acknowledged.'

onStatusMsg: [2020-07-02 21:29:24 677971][26493:140211614574336]
Service Name: ELEKTRON_DD
Item State: Closed / Ok / None / 'Batch request acknowledged.'

onStatusMsg: [2020-07-02 21:29:26 678170][26493:140211614574336]
Service Name: ELEKTRON_DD
Item State: Closed / Ok / None / 'Batch request acknowledged.'

onStatusMsg: [2020-07-02 21:29:28 678555][26493:140211614574336]
Service Name: ELEKTRON_DD
Item State: Closed / Ok / None / 'Batch request acknowledged.'

...

onStatusMsg: [2020-07-02 21:33:28 735769][26493:140211614574336]
Service Name: ELEKTRON_DD
Item State: Closed / Ok / None / 'Batch request acknowledged.'

onStatusMsg: [2020-07-02 21:33:30 736099][26493:140211614574336]
Service Name: ELEKTRON_DD
Item State: Closed / Ok / None / 'Batch request acknowledged.'

onStatusMsg: [2020-07-02 21:33:32 736463][26493:140211614574336]
Service Name: ELEKTRON_DD
Item State: Closed / Ok / None / 'Batch request acknowledged.'

...

onStatusMsg: [2020-07-02 21:40:41 880045][26493:140211614574336]
Service Name: ELEKTRON_DD
Item State: Closed / Ok / None / 'Batch request acknowledged.'

onStatusMsg: [2020-07-02 21:40:43 898048][26493:140211614574336]
Service Name: ELEKTRON_DD
Item State: Closed / Ok / None / 'Batch request acknowledged.'

onStatusMsg: [2020-07-02 21:40:45 903138][26493:140211614574336]
Service Name: ELEKTRON_DD
Item State: Closed / Ok / None / 'Batch request acknowledged.'

Such acknowledge message is far more than normal days, 300 versus 5. So what's the possible reason for that quote delay ? From the acknowledge message, May I suspect the trep is too slow to respond to my subscription? Any suggestion is appreciated, thank you.

zoya faberov

Hello @wangfugen,

This informational log is not sufficient, in my opinion, to conclude what has happened. Was there any refresh messages coming in in parallel as the batches were requested? That could give additional insight into what was happening?

It looks like, there might have been an intermittent interruption in Elektron service, that did not allow requests to go through at that time; would expect in this suspected scenario, as soon as the suspected interruption was resolved, would expect the connection to be recovered by EMA fully, and item streams reestablished.

However, I would check with the operations stuff responsible for your infrastructure/connectivity, if there was perhaps a brief outage around that time.

Going forward, there may be a very long of time till any next interruption in service, however, if you would like to know more on the application side, may need to prepare ahead of time. To do so you may try:

• increasing application's custom logging
• adding EMA xml tracing, into your EmaConfig.xml, I seem to recall that you are in EMA C++, if so then XmlTraceToFile and XmlTraceFileName, see EMA C++ Config Guide for complete details (if you are in EMA Java, let me know and will share Java details)