Refinitiv RDP: Websocket API: The remote party closed the WebSocket connection without completing...

ops01 · October 2022

... the close handshake

Hi Team-

We are seeing a disconnect at a specific time period while receiving updates. We see a disconnect at around 7:00:00AM EST time on our logs on daily basis.

Error message on our application log:

2022-09-28 11:01:01 , 172.17.15.15 , 172.17.15.15 , news , err , RefinitiveFeedHandler ,  09/28/22 07:01:01.397 Thrd(3880)  Web-Socket::Run() AggregateException InnerException: The remote party closed the WebSocket connection without completing the close handshake.

Error message on Refinitiv AWS log:

2022-09-28 11:00:14.635,"<ads-fanout-lrg-az1-use1-prd.1.ads: Info: Wed Sep 28  11:00:13.790515 2022>
WS JSON2 disconnect from ""GE-3345"" at position ""172.17.15.15"" on host """" using application ""256"" on channel 55.
Reason: User ""GE-3345"" at position ""172.17.15.15"" on host """" using application ""256"" on channel 55 has been disconnected due to an overflow condition.
<END>

We are using Refinitiv RDP, and Websocket API (C#) on the windows server environment.

API: https://github.com/Refinitiv/websocket-api/tree/master/Applications/Examples/RDP/CSharp

User Name:GE-A-XXXXXXX-X-XXXXXX
Ticker:TRACEBTDS.FINR

Application error log:

2022-09-28 11:01:01 , 172.17.15.15 , 172.17.15.15 , news , err , RefinitiveFeedHandler ,  09/28/22 07:01:01.390 Thrd(3880)  Web-Socket::Run() AggregateException: One or more errors occurred.
2022-09-28 11:01:01 , 172.17.15.15 , 172.17.15.15 , news , err , RefinitiveFeedHandler ,  09/28/22 07:01:01.397 Thrd(3880)  Web-Socket::Run() AggregateException StackTrace:    at System.Threading.Tasks.Task.ThrowIfExceptional(Boolean includeTaskCanceledExceptions)
2022-09-28 11:01:01 , 172.17.15.15 , 172.17.15.15 , news , err , RefinitiveFeedHandler ,  09/28/22 07:01:01.397 Thrd(3880)  Web-Socket::Run() AggregateException InnerException: The remote party closed the WebSocket connection without completing the close handshake.
2022-09-28 11:01:01 , 172.17.15.15 , 172.17.15.15 , news , err , RefinitiveFeedHandler ,  09/28/22 07:01:01.398 Thrd(3880)  Reconnect to the endpoint for Session1 after 1 seconds...
2022-09-28 11:01:02 , 172.17.15.15 , 172.17.15.15 , news , info , RefinitiveFeedHandler ,  09/28/22 07:01:02.419 Thrd(3880)  *** WebSocket::Connect() SendLogin(false) 
2022-09-28 11:01:02 , 172.17.15.15 , 172.17.15.15 , news , info , RefinitiveFeedHandler ,  09/28/22 07:01:02.419 Thrd(3880)  ***  Send Login() ***

raksina.samasiri · November 2022

Hi @ops01 ,

Regarding case number 11718559 that's raised to answer this question, please refer to the following Dev’s response below

Can we please let us know how many updates and the size of updates the WebSocket connection can handle?

> We provide a protocol specification and sample code for customer to use to build their own Websocket applications. The performance of customer’s application will depend on how well the code is written.

It is noted that this customer is seeing spikes up to 85,000 updates in 50 seconds. With Websocket JSON encrypted connections using our EMA perf tool, we see a ceiling of 20k updates/sec, to provide a ballpark estimate. We recommend that customer do some performance evaluations using RTSDK Performance tools: run their consuming Websocket application connecting to an RTSDK ETA ProvPerf in direct connection to understand their ceiling rate. We have a PerfTools guide (PDF doc) and perf tool code that can be obtained from GitHub (java or c/c++).

Please let me know if you require any additional information or assistance.

Jirapongse · October 2022

@ops01

Thanks for reaching out to us.

According to the answer in this thread, the error message on AWS is "... disconnected due to an overflow condition." Therefore, the problem could be:

The buffer overflow on the server-side. It could be from the network bandwidth or the application is a slow consumer. If the application subscribes to a lot of items and the update rate is higher than the rate that the application can handle, the server can cut the connection due to the overflow condition. To avoid this issue, the application must process the retrieved message as fast as possible. For example, the application can read the data from the WebSocket, and then put it into an internal queue to be processed by another thread. With this method, the thread that reads data from the WebSocket will not be blocked.

Please feel free to reach out if you have any further questions.

ops01 · October 2022

Hi @Jirapongse -

Thanks for the quick response.

Yes, based on the error "buffer overflow on the server-side", we made a change of processing or messages on the other thread. We made a change to storing messages on the queue

and another thread processing the updates. but, even after making the change we still see the

same disconnect issue.

Please suggest!!

Please find below a screenshot of the source code for reference.

Jirapongse · October 2022

@ops01

You can remove the data processing logic from the code and count the update messages per second that the application can receive from the WebSocket connection. If the connection still disconnects with the buffer overflow condition, it could be the limitation of the WebSocket, or network.

However, if the code (without the data processing logic) runs fine without disconnections, then you can count the update messages per second that the application (with the data processing logic) can receive from the WebSocket connection. Next, you can compare those values to verify the bottleneck.

ops01 · October 2022

@Jirapongse -

Thanks for the suggestion!

We are not doing much data processing, we simply parse the JSON messages to identify their type

, and if they are update type or refresh type messages we send them to Queue instead of processing them in the same thread.

Please let me know if above thing looks ok to you or you want me to make any change

Jirapongse · October 2022

https://community.developers.refinitiv.com/discussion/comment/98264#Comment_98264

@ops01

Sorry, I am not familiar with SQS messages.

Please measure the performance of the application by counting update messages that the application can process per second.

With this method, we will know how fast the application can read and process the retrieved data.

Gurpreet · October 2022

Hi @ops01,

In addition to not-decoding the received JSON messages, you can also limit the amount of data requested from server by using the VIEW feature. This will save on the network bandwidth and cut the processing time.

If your application only needs say BID and ASK fields, you can construct the request message like this -

{
  'ID': 2,
  'Key': {
    'Name': ['comma seperated list of instruments'],
    'Service': serviceName
  },
  'View': ['BID', 'ASK']
}

You might also need to suppress the debug logging, dynamic memory allocation, array resizing etc in code, since that will eat away the processing time.

The easiest way, is to try one of the ready made samples, to see if you are still experience these issues.

ops01 · October 2022

Thanks for the suggestion, we required all the fields so viewing only selected fields will not work in our case.

We tried processing updates on another thread using AWS SQS, we queued updates on the queue and process updates on the other thread, but, we see we see the same issue of disconnect.

Jirapongse · October 2022

https://community.developers.refinitiv.com/discussion/comment/98422#Comment_98422

@ops01

Please share a list of RICs that the application was subscribing to.

I would like to check the update rate of those RICs.

ops01 · October 2022

https://community.developers.refinitiv.com/discussion/comment/98433#Comment_98433

We subscribe only one RIC "TRACEBTDS.FINR".

Gurpreet · October 2022

https://community.developers.refinitiv.com/discussion/comment/98422#Comment_98422

Hi @ops01,

Underlying problem seems to be slow consumer - either due to network bandwidth or CPU bottlenecks. Can you disable the SQS queue and tell us if this is still an issue. Are you using high throughput queues? Can you try to bundle the SQS in the batch updates.

Jirapongse · October 2022

@ops01

I tested it yesterday by subscribing to TRACEBTDS.FINR.

There were a lot of update messages around 11:00 GMT.

2022-10-11 18:00:00.699588  Update Per Sec: 0 max:  1
2022-10-11 18:00:01.707050  Update Per Sec: 567 max:  567
2022-10-11 18:00:02.727342  Update Per Sec: 1262 max:  1262
2022-10-11 18:00:03.737329  Update Per Sec: 1251 max:  1262
2022-10-11 18:00:04.759695  Update Per Sec: 1288 max:  1288
2022-10-11 18:00:05.768519  Update Per Sec: 1214 max:  1288
2022-10-11 18:00:06.773765  Update Per Sec: 1256 max:  1288
2022-10-11 18:00:07.795474  Update Per Sec: 1305 max:  1305
2022-10-11 18:00:08.818008  Update Per Sec: 1290 max:  1305
2022-10-11 18:00:09.835881  Update Per Sec: 1296 max:  1305
2022-10-11 18:00:10.854778  Update Per Sec: 1231 max:  1305
2022-10-11 18:00:11.867766  Update Per Sec: 1262 max:  1305
2022-10-11 18:00:12.907905  Update Per Sec: 1300 max:  1305
2022-10-11 18:00:13.951785  Update Per Sec: 1311 max:  1311
2022-10-11 18:00:14.975552  Update Per Sec: 1294 max:  1311
2022-10-11 18:00:15.989549  Update Per Sec: 1268 max:  1311
2022-10-11 18:00:17.018954  Update Per Sec: 1265 max:  1311
2022-10-11 18:00:18.050105  Update Per Sec: 1273 max:  1311
2022-10-11 18:00:19.092790  Update Per Sec: 1332 max:  1332
2022-10-11 18:00:20.106447  Update Per Sec: 1308 max:  1332
2022-10-11 18:00:21.112374  Update Per Sec: 1257 max:  1332
2022-10-11 18:00:22.141999  Update Per Sec: 1277 max:  1332
2022-10-11 18:00:23.156945  Update Per Sec: 1324 max:  1332
2022-10-11 18:00:24.187142  Update Per Sec: 1344 max:  1344
2022-10-11 18:00:25.197338  Update Per Sec: 1289 max:  1344
2022-10-11 18:00:26.226197  Update Per Sec: 1285 max:  1344
2022-10-11 18:00:27.239789  Update Per Sec: 1267 max:  1344
2022-10-11 18:00:28.262927  Update Per Sec: 1303 max:  1344
2022-10-11 18:00:29.301034  Update Per Sec: 1277 max:  1344
2022-10-11 18:00:30.316078  Update Per Sec: 1279 max:  1344
2022-10-11 18:00:31.328291  Update Per Sec: 1295 max:  1344
2022-10-11 18:00:32.343672  Update Per Sec: 1298 max:  1344
2022-10-11 18:00:33.367752  Update Per Sec: 1386 max:  1386
2022-10-11 18:00:34.389024  Update Per Sec: 1367 max:  1386
2022-10-11 18:00:35.416875  Update Per Sec: 1266 max:  1386
2022-10-11 18:00:36.436777  Update Per Sec: 1290 max:  1386
2022-10-11 18:00:37.475291  Update Per Sec: 1365 max:  1386
2022-10-11 18:00:38.498353  Update Per Sec: 1343 max:  1386
2022-10-11 18:00:39.519245  Update Per Sec: 1308 max:  1386
2022-10-11 18:00:40.529847  Update Per Sec: 1248 max:  1386
2022-10-11 18:00:41.540243  Update Per Sec: 1290 max:  1386
2022-10-11 18:00:42.577044  Update Per Sec: 1305 max:  1386
2022-10-11 18:00:43.602227  Update Per Sec: 1346 max:  1386
2022-10-11 18:00:44.626414  Update Per Sec: 1293 max:  1386
2022-10-11 18:00:45.637904  Update Per Sec: 1247 max:  1386
2022-10-11 18:00:46.656105  Update Per Sec: 1311 max:  1386
2022-10-11 18:00:47.676083  Update Per Sec: 1260 max:  1386
2022-10-11 18:00:48.717259  Update Per Sec: 1351 max:  1386
2022-10-11 18:00:49.741336  Update Per Sec: 1334 max:  1386
2022-10-11 18:00:50.784531  Update Per Sec: 1275 max:  1386
2022-10-11 18:00:51.800804  Update Per Sec: 1258 max:  1386
2022-10-11 18:00:52.822604  Update Per Sec: 1326 max:  1386
2022-10-11 18:00:53.845771  Update Per Sec: 1298 max:  1386
2022-10-11 18:00:54.868046  Update Per Sec: 1322 max:  1386
2022-10-11 18:00:55.908339  Update Per Sec: 1294 max:  1386
2022-10-11 18:00:56.934278  Update Per Sec: 1314 max:  1386
2022-10-11 18:00:57.977892  Update Per Sec: 1335 max:  1386
Session: session1;  Error: Connection to remote host was lost.
WebSocket Closed for session1
Reconnect to the endpoint for session1 after 3 seconds...
2022-10-11 18:00:58.984908  Update Per Sec: 1001 max:  1386
2022-10-11 18:01:00.005427  Update Per Sec: 0 max:  1386
2022-10-11 18:01:01.022307  Update Per Sec: 0 max:  1386
Connecting to WebSocket wss://ap-northeast-1-aws-3-lrg.optimized-pricing-api.refinitiv.net:443/WebSocket for session1...
2022-10-11 18:01:02.040908  Update Per Sec: 0 max:  1386
2022-10-11 18:01:03.044312  Update Per Sec: 0 max:  1386
WebSocket successfully connected for session1!

The timestamps are in GMT+7. The connection was disconnected around 11:00:57 (GMT).

Does the problem always happen around 11:00 GMT?

ops01 · October 2022

https://community.developers.refinitiv.com/discussion/comment/98456#Comment_98456

Yes, we see a disconnect every day around 11:00 GMT always.

ops01 · October 2022

Yes, we see a disconnect every day around 11:00 GMT always

Jirapongse · October 2022

@ops01

Thank you for confirmation.

I have contacted the RTO support team to verify the bandwidth usage and why there are a lot of update messages sent around 11:00 GMT which can cause a buffer overflow condition on RTO.

ops01 · October 2022

https://community.developers.refinitiv.com/discussion/comment/98458#Comment_98458

Hi @Jirapongse - Did you get any update from the RTO support team?

ops01 · October 2022

@Jirapongse -

Did you get an update from the RTO support team?

okatz · October 2022

I had a problem that might imply on your situation... If more them one developer on your company use the same api key and credentials to open session on the same time, last session grabs the refresh token for timeout expiration refresh process, so when the first developer process reaches the point of Refreshing Expiration token - his token was already overwritten on the server by the second developer process.

Make sure only one process is active with same credentials (one account = one connection)

ops01 · October 2022

From the error, we can see that it is related to a buffer overflow, it has nothing to do with one account with one connection...

okatz · October 2022

https://community.developers.refinitiv.com/discussion/comment/98481#Comment_98481

look at my question with the python 1.0.0b20 rd pack. There might be an error with disconnection related to timeout. the error message response is a Malformed Jason message were the json decoder on your framework might find problematic for parsing/decoding because of multiple ' " ' (double quats) - the decoder may fail with buffer overflow because of bad text processing state.

As I stated before it might be related and might not....

Jirapongse · October 2022

@ops01

I got the response from the RTO team.

< TRACEBTDS.FINR> is a FINRA Master RIC that updates every second. The updates from thousands of RICs flow to this one Master RIC due to which when the market activity is higher there will be an increase in the output. We have a closing run scheduled at 11:00 GMT for all the FINRA RICs, at this time we clear out all the FINRA RICs. Since the universe size of FINRA RIC is large it is possible that the client will get a large number of output during this time.

They ran a test using an RSSL connection into RTO and we received >85k updates within 50 seconds of 11:00 GMTfor this RIC. It would seem that this is too much for Websocket/JSON connections.

To retrieve data from this RIC via RTO, you need to use the RSSL protocol instead. The RSSL protocol supports C/C++ and Java programming languages. Otherwise, you need to contact the content team to find any alternative instruments that can provide equivalent data.

ops01 · October 2022

@Jirapongse - Thanks for the update.

As per your suggestion, if we use WebSocket/JSON connections, even if we do not do much processing on updates. we will have a disconnect issue(due to the no. of updates and size of updates) is it correct?

Also, is there any way, if we can handle these updates using WebSocket?

Jirapongse · October 2022

https://community.developers.refinitiv.com/discussion/comment/98581#Comment_98581

@ops01

Yes, you are correct. According to the update from the RTO team, during 11:00 GMT, there are a lot of update messages sent to the application. The websocket/json connection is unable to handle those messages at 11:00 GMT.

You may try to use a view instead, as mentioned by my colleague.

ops01 · October 2022

@Jirapongse -

Thanks for the details..

We are also looking for the answers to the below questions.

Can we please let us know how many updates and the size of updates the WebSocket connection can handle?
If we disconnect due to not being able to handle the update size when we reconnect, do we lose any data?
Need more information regarding RSSL protocol implementation using c++,
If we look to implement RSSL protocol using c++, do we need a separate account/login to access data( can we use the same WebSocket login/account)?

Refinitiv RDP: Websocket API: The remote party closed the WebSocket connection without completing...

Best Answer

Answers

Categories

EXPLORE OUR SITES

Refinitiv RDP: Websocket API: The remote party closed the WebSocket connection without completing...

Best Answer

Answers

Categories