Stream multiple chain request in RDP

The IChainStream only supports a single item specification. While there are "basic" chains, some do have well over 1000 constituents and the initial goal was to simplify the usage. It's unclear if you need to keep each chain open or whether you simply need to capture and stream their constituents. If so, I believe the IChainResponse using .GetData() may serve your purposes. However, you would still need to manage your own requests within a collection - but should not be significant. You could capture all 200 items and apply the entire list of constituents to a pricing stream.

Hello

For this case, I need to capture the chain constituents and their order DURING the entire exchange session.
Gainers and losers and their positions may change quickly during the day.
Chain.Definition(chainRIC).GetData() will do the job, but the preferred solution would be rather a subscription, since I would to handle only the updates. That's why I was surprised to find out that this is a single item API.
Again, is opening a large number of streams for each required chain is a good idea?
What are the limits on the number of streams?

The constituents are also streamed (pricing stream) separately for their last price, turnover etc., but here it is about the list of best/worst performers during the trading day.

Sorry, but for each chain you WILL need to monitor the order? If so, then .GetData() will not do the job. You will need to use .GetStream(). Also, the Chain definition does not open a stream for each constituent - I want to make sure we're on the same page here. If you need to maintain a logical order for each chain, then having a single streaming chain for each item is sufficient. Perhaps I don't fully understand your use case, but having a single open streaming chain may be easier to manage if you have to ensure the order is maintained throughout the life of the chain.

Hi @nick.zincone
For the "basic" chains like top gainers/losers (like .PG.xx or .PL.xx, I need to maintain the order in my application, therefore having the streaming (subscription) would be the best.
I think though I can still use .GetData() by doing the scheduled requests (say every 2 minutes) and maintain the order with it, I suppose, unless the list of IChainResponse.Data.Constituents is unordered, but this would be my second choice.
That's why I insist on streaming (subscription): how many streams I can open? What is the max supported?

For the large chains (like index members, issued from "0#.IXIC" for example), order does not matter because their purpose is different then obtaining gainers and losers in order of the chain predicate.

So given "basic" chains, for example .PG.QA has a small number of constituents (10)

And you need to make sure the order is retained, you can do this:

// Create streaming chain
var chain = Chain.Definition(".PG.QA").GetStream().OnStatus((item, status, stream) => 
                                           Console.WriteLine($"Status for item: {item} {status}"))
                                                  .OnError((item, error, stream) => 
                                           Console.WriteLine($"Error for item: {item} {error}");
chain.Open()

// Once opened, grab the ordered constituents
DisplayConstituents(chain.Constituents);   // Iterate through IList and display

// Open streaming subscriptions for the constituents - using IPricingStream
// Note: you can pass in the entire list of constituents
OpenSubscription(chain.Constituents);      // Create IPricingStream for IList

Because you need to keep the above chain ordered, you can completely rely on the streaming chain to ensure the Constituents list will automatically order it for you - you do not need to schedule a timer and call out to get the chain. At anytime, you can call: chain.Constituents and the list will be up-to-date because the chain is streaming and will manage all the add/delete/update events it receives automatically.

Now, you also have the option to react to the add/delete/update events so you can manage the order automatically - for example, if you have the chain in some kind of display and you want to shuffle around the display based on chain events. You can capture these events like this:

var chain = Chain.Definition(".PG.QA").GetStream().OnAdd((index, newv, stream) => 
                  Console.WriteLine($"\tNew constituent {newv} added at index: {index}"))
                                                  .OnRemove((index, oldv, stream) => 
                  Console.WriteLine($"\tRemoved constituent {oldv} added at index: {index}"))
                                                  .OnUpdate((index, oldv, newv, stream) => 
                  Console.WriteLine($"Index {index} within our Chain changed from {oldv} => {newv}"))
                                                  .OnStatus((item, status, stream) => 
                  Console.WriteLine($"Status for item: {item} {status}"))
                                                  .OnError((item, error, stream) => 
                  Console.WriteLine($"Error for item: {item} {error}");

Refer to the Streaming Chains Example within Git.

For the larger chains where you don't need to worry about order, you don't stream. The call would be something like this:

var chain = Chain.Definition("0#.IXIC").GetStream().Streaming(false)
                                                   .OnStatus((item, status, stream) => 
                                           Console.WriteLine($"Status for item: {item} {status}"))
                                                   .OnError((item, error, stream) => 
                                           Console.WriteLine($"Error for item: {item} {error}");
chain.Open();

*Note*: The above is not using the .GetData() call. The reason is because the .GetData() call actually is using a RDP chain service - which you may not have a license. The above code, which sets .Streaming(false) then a call to open() is equivalent. Refer to the Streaming Snapshot Example within Git.

From here, like the above example, you can create an IPricingStream and pass in the entire .Constituents list.

Refer to the Streaming Pricing Example within Git.

The max number of open streams depends on your streaming license - you will have to confirm with your Account Manager who set up the details of your license package.

Hello @nick.zincone
Thanks for the explanations. I did see the example and I tried the same schema in my application. Still have some comments though.
1.Working with the list of constituents is a good idea, except that to call it, at any time, as you write, I need an event. At the initial chain.Open() I can get call the chain.Constituents and build my initial list of chain members, but then I need to keep it up to date, so I need the event(s). I need to explore this, but what I found out in the test is that in the event handlers the call to chain.Constituents gives empty IList and the chain.Constituents will only be populated once the event callbacks are completed. This gives kind of the egg/chicken problem… So probably the best way is either rely only the events (Add/Remove/Update) or forget the events entirely and simply periodically call chain.Constituents and rebuild the list every time…
2.For the larger chains, definitely using Streaming(false) is the interesting solution.
3.Still, as I mentioned, because of the one RIC per chain limitation, I will finish with the large number of streams open. Are there any limitations on the number of streams? How does it affect the performance. To tell the truth, this is surprising, If in Pricing the limit of RICS per stream seems to be 50K, why in the case of the chains the limit is 1?

When working with ordered constituents, you have a choice. Within my answer above, the first 2 code samples explains this. The first one is there for you to refer to the order constituents when ever you need to - when you do, the list will be ordered. Alternatively, the 2nd example shows how to utilize the events callbacks so you can react to changes in the order. If you have your constituents in a display, for example, the 2nd example demonstrates how to keep it up-to-date. If you are having trouble with the event mechanism or see unexpected behavior (empty list for example), post a separate question and we can address it there.

Regarding the "one RIC per chain limitation", I don't see it the same as you. The design of the chain is to keep it simple. If we allow multiple chains per request, then we would significantly complicate the interface and the implementation. If I need to manage multiple chains, I can easily keep the multiple chain definitions and references with a collection and manage it cleanly from there. Also, I don't understand this statement: "...I will finish with the large number of streams open". When you open a streaming chain, only 1 stream is open. it is up to your application to open multiple streams based on the constituents returned. I also provided you an answer above regarding the limitation on the number of streams. The more streams you open, the more work your application will need to go through. For example, the initial request for fetch a chain of over 3000 constituents takes a few seconds given how chain requesting works with it's underlying LINK fields. Also, there is not a limitation in the API, but rather the backend as I mentioned above.

Sorry, I don't understand: "why in the case of chains the limit is 1"? The limit is not 1 - the backend can open many chains. As I mentioned in the previous paragraph, the design of the chain API is to keep things simple for the developer to manage and simplifies the implementation.

Hello @nick.zincone

Putting aside the discussion regarding the "one RIC per chain limitation" which is rather academic, I tried to follow the guidelines that you included. In general, it does not look good, partially because of my needs but also because the chain API seems to miss the essential functionality.
1. Regarding my particular needs, for the “big” chains using the pattern
“Chain.Definition("0#.IXIC").GetStream().Streaming(false).OnStatus((item, status, stream) => cb“

is fine in the sense that I can easily call the chain.Constituents and populate from it my internal structures. The twist is that while I do not care for the order here, I need to refresh this chain at least once a day to detect new securities in the market index. So, theoretically, I should use rather .Streaming(true) and open the stream only when I need it. Seems excessive though to keep the stream for 1 call a day. Seems better just to get the stream when I need it and then discard it. What is your opinion?

2. MAJOR problem is with the “small” chains (.PG.XX, .PL.XX). Here the better pattern are the callbacks, where one can update the content of the list by handling the individual Add/Remove/Update. The problem is that as a client I am unable to know when the series of changes is complete so I cannot publish the chain. It is easily visible in the fields in the update message: say there are 5 constituents that changed index. The callback will be called individually for every updated index, but as a client I need something that will indicate that the updates are completed, the chain is coherent and can published. I cannot publish after every individual change because the chain would be internally incoherent. And there is nothing in the API that the client application can use.
There should be additional arguments in the callback signature, indicating the sequential number of the change in the set of changes included in the underlying update/refresh message. Ex. 5^th of 8 (total). Or OnChangeComplete() callback.

This leaves only the possibility to use the chain.Constituents but as you write yourself, I have to call it instead of reacting to an event. This defies the whole idea of streaming and even driven programming. What is you opinion?

Regarding your question around the big chain where you only need to refresh this chain once, your preference to turn streaming on makes sense to me. This way, you can simply get the constituents whenever you desire and the collection will be up-to-date.

Regarding the ordered chains, I understand your concerns. After looking deeper into messages coming from the backend, the update is coming in as a single message but the library is sending out individual events to help manage the combined collection. Based on what I see, I think your suggestion to provide some sort of event to indicate the transaction/update is complete makes sense to me and would be extremely useful.

I’m going to reach out to the Development team to look into it.

Thanks.

Hello @nick.zincone

Thanks for your message.
Regarding the MAJOR question, it is not only this that the indication of the transaction complete would be useful. It is a simply required if the chain interface is to be working properly as the streaming/subscription interface.
Also an indicator whether the transaction originates from refresh or from update, as it is in the pricing streaming with OnRefresh and OnUpdate callbacks.
I write about it, because I got disconnected during my test and on reconnection I got
“Application Exception in Refresh: An item with the same key has already been added. Key: .PG.QA” from the Refinitiv.Data.Log.Output that I activated.
If I have a collection of a number open IChainStream anI get disconenctd, how to avoid this kind of exception?
See the attached file chain_disconenct_20240719.txt at line 378. I included only the first one, but I get an exception for all the “reconnected” items.
Thanks