For a deeper look into our Elektron API, look into:

Overview |  Quickstart |  Documentation |  Downloads |  Tutorials |  Articles

question

Upvotes
Accepted
7 3 2 3

Websocket API - Dictionary handling, best requests implementation

We have a working test application in .net capable of streaming a single RIC and displaying all available fields for such RIC.

We are going to use it among others to investigate the fields and whether a RIC is available in the stream etc.


We have also started to modify the true feed server application in order to use the new technology, while remaining compatible with the older one.

So that we can switch between them if necessary.

This app is much more advanced than the simple test one and I have a few questions / remarks, that I am going to enumerate, without a specific order:


  1. Strings into enumerators


The DataState field comes as a string but we used to have it as an enumerator defined in RFA dll, which is much more solid.

I want to be sure the string can be parsed into such an enumerator, so can you confirm that all and only possible values are the following:

  • NoChange
  • Ok
  • Suspect
  • Unspecified


Same question for the StreamState enum:

  • Closed
  • ClosedRecover
  • NonStreaming
  • Open
  • Redirected
  • UnspecifiedStreamState



  1. The initial imaging of records


In the old technology, after the first login message we would receive a full image of all RICs (i.e. all available fields).

Then a second login message.

Finally partial updates.


The second login message was very useful.

We would indeed perform a check on available RICs and a lot of business logic deriving from it.

For example, we would distinguish between three states, Disconnected / Subscribing / Streaming (where Streaming only takes place after the initial imaging of all RICs)


This second login message seems not to take place anymore in the new technology, so I am looking to find a way to tell.

Can I safely assume that there is always a ‘Refresh’ (full) message for every RIC and only after that an ‘Update’ message (partial) for existing RICs ?

For non-existing RICs, can I assume that there is always at least a Status message coming back, telling the record in Closed or Suspicious, etc. ?

On the basis of this behavior I could reconstruct our business logic.


  1. The Post request of multiple RICs


I tried in two different ways:


  • With n requests of 1 RIC
  • With 1 batch request of n RICs


For former seems not to work or at least I am missing something.

Almost like it subscribes only to one RIC anyway, probably the last one sent.


The latter works fine, but so far only tested for about 40 RICs.

Will it work for few thousands too ??



  1. Buffer size



The WebSocket object is defined with 8K buffer size in your examples.

Do you think we shall use a larger size for real-world applications with thousands of RICs being streamed ?

What would happen and how can we inspect a buffer overflow ?

websocketsrrtorefinitiv-realtime-optimised
icon clock
10 |1500

Up to 2 attachments (including images) can be used with a maximum of 512.0 KiB each and 1.0 MiB total.

Upvotes
Accepted
25.3k 89 12 25

Hi @erol.komac

Is the client connecting to an ADS or RTO?

There is a theoretical limit of 64K for batch request message size, and as my colleague just posted above, the MaxMsgSize value return in the Login response shows the actual value. So, the client should reduce the batch request to say 4000 RICs and confirm it works. If reducing the batch size works, then they should try making splitting the 5K RICS into two batch requests instead.

In terms of your questions about RTO feed limits etc:

I have certainly run Websocket API tests with 3k RICs without issue, however, I was not doing any particular processing with the response payload (other than dumping to console / file).

There is a possibility the client may struggle with 10K instruments from a single .Net example. The RTO or ADS ability to support 10K RICs should not be an issue, rather the ability of the .NET application to process 10K RICs worth of Streaming JSON updates, how many fields they are processing, how CPU intensive the payload processing is etc.

If the customer does not need every field in the Refresh and Update, then they should consider using a View - to minimize the bandwidth usage and processing of the payload required. A Batch and View can be combined into a single request - see View Tutorial or demonstrated in WebSocketAPI.Python.TestClient




icon clock
10 |1500

Up to 2 attachments (including images) can be used with a maximum of 512.0 KiB each and 1.0 MiB total.

Upvote
18.6k 85 39 63

Hi @erol.komac,

There are a number of questions in there, some of which I may not be able to provide - I will also reach out to colleagues to help.

String into enumerators

DataState and StreamState fields have been very consistent historically and we do not foresee any changes that will break existing applications.

The initial imaging of records

I don't recall a 2nd login response after you receive the initial image. Reading your description, it appears this 2nd login response contains State information, for example, for a streaming request:

  "State": {
    "Stream": "Open",
    "Data": "Ok"
  },

however, I may be misreading your explanation. You are correct, there is only 1 login response.

Can I safely assume that there is always a ‘Refresh’ (full) message for every RIC and only after that an ‘Update’ message (partial) for existing RICs?

Correct. The one exception is if you included an invalid RIC in the request, you will receive a Status in lieu of a Refresh.

For non-existing RICs, can I assume that there is always at least a Status message coming back, telling the record in Closed or Suspicious, etc.?

Correct.

Buffer size

Do you think we shall use a larger size for real-world applications with thousands of RICs being streamed?

I would suggest you start with the default and increase to measure the effects if any.

What would happen and how can we inspect a buffer overflow?

If it is a situation where the buffer overflows as a result of a slow consumer, the server will disconnect the WebSocket connection.

icon clock
10 |1500

Up to 2 attachments (including images) can be used with a maximum of 512.0 KiB each and 1.0 MiB total.

Upvotes
26.3k 61 17 14

Hello @erol.komac

The Post request of multiple RICs

Could you please clarify this question? Do you want to send multiple RICs data in a single Post message?

The Post message can contain only 1 RIC data (Refresh or Update message) in a message. If the application needs to contribute multiple RICs data, the application needs to send multiple Post request messages.

There are 2 types of Post Message:

  • On-Stream Post: Before sending an on-stream post, the client must first request data for an item. After opening the data stream, the client application can then send a post. Example: The application needs to subscribe for EUR= RIC before contributing EUR= additional data in the same stream id
  • Off-Stream Post: In an off-stream post, the client application can send a post for an item via a Login stream, regardless of whether a data stream first exists. It means the application can contribute multiple RICs via multiple Post messages via the Login stream id.

Hope this helps.

icon clock
10 |1500

Up to 2 attachments (including images) can be used with a maximum of 512.0 KiB each and 1.0 MiB total.

Upvotes
7 3 2 3

Hi @wasin.waeosri,


The question was basically, how to request for multiple instruments in the post message.

Fyi, the trial access is for 10K instruments.

See below questions from the developer for more context:


I moved from the test of a subset of 40 RICs (where all seems to work well) into the full ~ 5000 RICs, but websocket is failing.


In particular, I get the message: The remote party closed the WebSocket connection without completing the close handshake.


How scalable is it the feed??


‘ ----------


One thing I noticed is that the POST message, where I request all RICs at once, is huge.

Because it’s a unique post of all RICs as a batch.

So, it’s about 63K of size.

I set the buffers of the websocket to 65K (that’s its max) but still the same error. So the size of buffer does not seem to make a difference.


Can you advise ?

Are there limitations on total number of RICs to subscribe to ?

Are there limitations in the initial POST message to request the RICs ?

icon clock
10 |1500

Up to 2 attachments (including images) can be used with a maximum of 512.0 KiB each and 1.0 MiB total.

Upvotes
26.3k 61 17 14

Hello @erol.komac

Thank you for clarification, the client's "POST" means "send item request message" (not Refinitiv Real-Time POST/Contribution message).

What is the infra that the client is connecting to? Is it deployed Refinitiv Real-Time Distribution System or Refinitiv Real-Time Optimized (Cloud)?

Can the client try to throttle the request by reducing the number of RICs in each batch request message to be smaller and send multiple batch request messages instead?

icon clock
10 |1500

Up to 2 attachments (including images) can be used with a maximum of 512.0 KiB each and 1.0 MiB total.

Upvotes
18.6k 85 39 63

@erol.komac,

The WebSocket server limits requests based on the size of the message request, not on the number of items in the request. The login response will contain a field called: MaxMsgSize Which represents the maximum size, in bytes, a request can be. If you exceed this limit, the WebSocket service will disconnect.


icon clock
10 |1500

Up to 2 attachments (including images) can be used with a maximum of 512.0 KiB each and 1.0 MiB total.

Upvotes
7 3 2 3

The Websocket connection it directly into the RTO feed.

icon clock
10 |1500

Up to 2 attachments (including images) can be used with a maximum of 512.0 KiB each and 1.0 MiB total.

Upvotes
7 3 2 3

Hi @umer.nalla, the client is connecting to RTO directly.

Thank you all for the suggestions and guidance.

icon clock
10 |1500

Up to 2 attachments (including images) can be used with a maximum of 512.0 KiB each and 1.0 MiB total.

Write an Answer

Hint: Notify or tag a user in this post by typing @username.

Up to 2 attachments (including images) can be used with a maximum of 512.0 KiB each and 1.0 MiB total.