question

Upvote
Accepted
87 10 13 14

Status updates sent through incorrect IPricingStreams registered actions

We are using Refinitiv Data Library for .NET (via nuget package Refinitiv.Data.Content).

Lately we have found an issue that some subscriptions transition into the "Closed" state and are not receiving further updates.

The strange thing is that we are apparently receiving status update messages that belong to other IPricingStreams.

We open stream subscriptions registering callback actions for refreshes, updates, and status changes:

    var pricingDefinition = Pricing.Definition(ticker).Closure(subscriptionId);
    var pricingStream = pricingDefinition.GetStream(activeSession)
        .OnRefresh((ric, msg, stream) =>
        {
            ...
        })
        .OnUpdate((ric, msg, stream) =>
        {
            ...
        })
        .OnStatus((ric, msg, stream) =>
        {
            ...
        });

We have registered logging via Refinitiv.Data.Log.Level (set to Trace).

This is an excerpt from the logs highlighting the issue:

1) Opening a stream for '.EVAS':

    2023-01-24T14:47:33    Debug    Refinitiv.Data.Delivery.Stream.OMMConnection    Stream request "SendMessageAsync": "{
      "ID": 2,
      "Streaming": true,
      "Key": {
        "Name": ".EVAS"
      },
      "View": [
        "ACTIV_DATE",
        "TIMACT"
      ]
    }"

From then one, we do receive .EVAS updates as expected.

2) Hours later, we try to open a non-streaming IPricingStream for '0#FDX' (note that this is a non-existent ric):

    2023-01-25T07:44:30    Debug    Refinitiv.Data.Delivery.Stream.OMMConnection    Stream request "SendMessageAsync": "{
      "ID": 9,
      "Streaming": false,
      "Key": {
        "Name": "0#FDX"
      }
    }"

3) We do get back an error status, HOWEVER this is sent via the actions registered for the .EVAS stream opened a day before. We would expect it to be sent through the actions registered in the '0#FDX' subscription!

  2023-01-25T07:44:30    Debug    Refinitiv.Data.Delivery.Stream.OMMConnection    Status response: "{
      "ID": 2,
      "Type": "Status",
      "Key": {
        "Service": "IDN_FD3",
        "Name": "0#FDX"
      },
      "State": {
        "Stream": "Closed",
        "Data": "Suspect",
        "Code": "AlreadyOpen",
        "Text": "Request Rejected: Cannot request new item on open stream."
      }
    }"

From this point onwards, no updates will be received for .EVAS anymore.

We also get this log callback:

    2023-01-25T07:44:30    Error    Refinitiv.Data.Delivery.Stream.OMMStreamCache    Internal error locating ID: 2 for duplicate requested item: [.EVAS]

"AlreadyOpen" and "Cannot request new item on open stream" sound suspicious. Can you please confirm if this is a server side or API error?

We are using Eikon 4.0.60 (4.0.60018) by the way.

#technologyrefinitiv-data-platformc#streaming-prices
icon clock
10 |1500

Up to 2 attachments (including images) can be used with a maximum of 512.0 KiB each and 1.0 MiB total.

Upvotes
Accepted
18.4k 84 39 63

Hi @GoGoGroundhog

As far as I can tell, the backend service appears to be sending incorrect results. For example, in your original post, the first request (.EVAS) is sent using ID: 2. When you send a snapshot request later using ID: 9 for item (0#FDX), the server somehow reported an issue with stream ID: 2. This is the issue. The library simply reacted to the status message, as reported by the server, to deal with a duplicate on ID: 2.

I did update my test to send a snapshot later on to match your original use case, but the correct response came back again.

As I believe the issue is in the backend, I will need to replicate what you have encountered. I don't know the nature of your application, nor the frequency of requests or the volume of items you are requesting. I can spend some time trying to replicate here, but so far no luck.

What I can do is see if there is a way to enable some kind of logs for the streaming service on the desktop - that way we may be able to provide some more insight to report to the backend development team. Let me look into seeing if this is possible.

thanks.

icon clock
10 |1500

Up to 2 attachments (including images) can be used with a maximum of 512.0 KiB each and 1.0 MiB total.

Upvote
18.4k 84 39 63

Hi @GoGoGroundhog

Thanks for the feedback.

I went through the same use case you provided above, except I did not try the 2nd request hours later, but rather 30 seconds - as a quick test. Does this issue occur randomly or are you able to replicate it every time? Have you tried a simple test where the 2nd request comes shortly after the 1st? I would like to try to replicate this so any additional tests you can do would be helpful.

This is what I did:

var stream1 = Pricing.Definition().Universe(".EVAS")
                                  .GetStream();

stream1.OnRefresh((item, refresh, s) => Console.WriteLine($"Stream 1 Refresh {item}"))
       .OnUpdate((item, update, s) => Console.WriteLine($"Stream 1 Update {item}"))
       .OnStatus((item, status, s) => Console.WriteLine($"Stream 1 Status {item}: {status}"))
       .OnComplete(s => Console.WriteLine("Stream 1 Request Completed."))
       .Open();

System.Threading.Thread.Sleep(30000);

var stream2 = Pricing.Definition().Universe("0#FDX")
                                  .GetStream();

stream2.OnRefresh((item, refresh, s) => Console.WriteLine($"Stream 2 {item}"))
       .OnUpdate((item, update, s) => Console.WriteLine($"Stream 2 Update {item}"))
       .OnStatus((item, status, s) => Console.WriteLine($"Stream 2 Status {item}: {status}"
       .OnComplete(s => Console.WriteLine("Stream 2 Request Completed."))
       .Open();

ahs.png

From the logs:

{
  "ID": 2,
  "Streaming": true,
  "Key": {
    "Name": ".EVAS"
  }
}
{
  "ID": 3,
  "Streaming": true,
  "Key": {
    "Name": "0#FDX"
  }
}
{
  "ID": 3,
  "Type": "Status",
  "Key": {
    "Service": "IDN_RDFNTS_CF",
    "Name": "0#FDX"
  },
  "State": {
    "Stream": "Closed",
    "Data": "Suspect",
    "Code": "NotFound",
    "Text": "The record could not be found"
  }
}

ahs.png (186.7 KiB)
icon clock
10 |1500

Up to 2 attachments (including images) can be used with a maximum of 512.0 KiB each and 1.0 MiB total.

Hi Nick,

thanks for checking this out.

Unfortunately, I would also like to have a clear reproduction case, but unfortunately this appears to happen rather randomly. I could not reproduce this yet on my development machine, only several times with the production app (on a vm).

The repro above was a bit simplified, actually we had several more RICs with an active (streaming) subscription running, happily providing updates until the next morning.Then, a number of streams (in this case with Streaming = False - not sure if this is relevant) where tried to be opened (all for unrecognized symbols). The closed status messages we received where then (imho falsely) distributed among the running streaming subscriptions, rendering them with closed status. Also note that the error message was not like "unknown ticker" but AlreadyOpen / "Cannot request new item on open stream". We have separate stream objects of type IPricingStream that are never reused for later requests.
In which circumstances can the API return such an error?
I am happy to provide as many further details, logs etc. as necessary!

Just for the record, we couldn't reproduce this on the vm today. We got the expected error status responses back through the expected IPricingStream registered action.

We also updated to Eikon 4.0.61017, just in case.

2023-01-26T08:46:06    Debug    Refinitiv.Data.Delivery.Stream.OMMConnection    Stream request "SendMessageAsync": "{
  "ID": 11,
  "Streaming": false,
  "Key": {
    "Name": "0#SDKFKLSDF"
  }
}"
2023-01-26T08:46:06    Debug    Refinitiv.Data.Delivery.Stream.OMMConnection    Status response: "{
  "ID": 11,
  "Type": "Status",
  "Key": {
    "Service": "IDN_FD3",
    "Name": "0#SDKFKLSDF"
  },
  "State": {
    "Stream": "Closed",
    "Data": "Suspect",
    "Code": "NotFound",
    "Text": "The record could not be found"
  }
}"
Upvotes
18.4k 84 39 63

Hi @GoGoGroundhog

I'm presently communicating with the backend team and have some follow up/details for you.

  1. Can you provide the API Proxy version as well?
  2. Do you have the ability to use Refinitiv Workspace instead of Eikon - they asked this question

They have found a similar problem in an old version of the API Proxy so they will need to determine if the version you have includes a fix. Also, they did provide instructions to enable logging, but these are instructions for Refinitiv Workspace, not Eikon. I'll wait to hear back from you and report back to the server team.

thanks,

Nick

icon clock
10 |1500

Up to 2 attachments (including images) can be used with a maximum of 512.0 KiB each and 1.0 MiB total.

Upvotes
87 10 13 14

Hi Nick,

Thank you for your update and thanks for talking to the server team.

Regarding API proxy Version, I'm not fully sure what you mean or where I have to look to find this.

In the Eikon About screen, under apps I found "Proxy (9.61.0.11)". As I mentioned, we updated Eikon the day before yesterday so the logs I reported may have been produced with an older version.

We're using the latest Refinitiv.Data nuget package (1.0.0-beta4).

Unfortunately we have no working Refinitiv Workspace accounts currently (our admin is trying to figure this out for some time now with Refinitiv support).

Our production app is also used by customers that only have Eikon available.

Thanks again, and have a nice weekend,

Ingo

icon clock
10 |1500

Up to 2 attachments (including images) can be used with a maximum of 512.0 KiB each and 1.0 MiB total.

Upvotes
18.4k 84 39 63

Hi @GoGoGroundhog

Can you use the following command within a browser to provide the version details the server team is after:

http://localhost:9060/api/status


icon clock
10 |1500

Up to 2 attachments (including images) can be used with a maximum of 512.0 KiB each and 1.0 MiB total.

{"statusCode":"ST_PROXY_READY","version":"3.3.11-eikon4"}

Hi @GoGoGroundhog

Thanks for these details - this is the latest version. The server team has been spending time trying to reproduce the issue. They want to confirm if you have been seeing this issue continue to occur recently. If so, any specific conditions you can help to narrow this down - how many instruments are you subscribing to? Is it just the 2? Does it seem to happen if / when you request hours later? Anything to help them during their tests.

thanks for your patience,

Nick

Hi Nick,

We think there were more subscriptions active than NGLNMc1 and .EVAS but unfortunately do not have a record which and how many.
I asked our QA team to set up an overnight test later today. I will follow up tomorrow on findings.

Thanks for coordinating with the server team!

Ingo


Upvotes
87 10 13 14

Our overnight test (with about 10 subscriptions) did not show issues.
This morning all subscribed symbols were still generating updates, so the problem could not be reproduced.

We'll keep an eye on this and will follow up if it comes up again.

icon clock
10 |1500

Up to 2 attachments (including images) can be used with a maximum of 512.0 KiB each and 1.0 MiB total.

Write an Answer

Hint: Notify or tag a user in this post by typing @username.

Up to 2 attachments (including images) can be used with a maximum of 512.0 KiB each and 1.0 MiB total.