Discover Refinitiv
MyRefinitiv Refinitiv Perspectives Careers
Created with Sketch.
All APIs Questions & Answers  Register |  Login
Ask a question
  • Questions
  • Tags
  • Badges
  • Unanswered
Search:
  • Home /
  • Eikon Data APIs /
avatar image
Question by bokun-a.wei · Apr 29, 2020 at 03:22 PM · mrnert in cloudmrn_story

Interrupted streaming service from ERT in cloud with MRN

I am trying to get real time news data using ERT in cloud and the MRN service. I am using the template code shown here https://github.com/Refinitiv-API-Samples/Example.WebSocketAPI.Python.TestClient in pywstestclient.py, but I added changes so that in the process_message function I added the processMRNUpdate function from https://github.com/Refinitiv-API-Samples/Example.WebSocketAPI.Python.MRN/blob/master/mrn_console_app.py, which is unique to MRN and process the data coming through. I also added code to put the streamed data into a mongo DB. My problem is that after a random amount of time, I would get one of the several errors:

[ E r r n o 1 0 4 ] C o n n e c t i o n r e s e t b y p e e r

W e b S o c k e t C l o s e d


or


C o n n e c t i o n i s a l r e a d y c l o s e d .

W e b S o c k e t C l o s e d


or


Traceback (most recent call last):

File "MRNclient.py", line 348, in <module>

market_data.reissue_token(ws_app,sts_token)

File "/mnt/batch/tasks/shared/LS_root/mounts/clusters/bokun/code/Users/Bokun/Refinitive_real_time_data/market_data.py", line 246, in reissue_token

send_login_request(ws, True)

File "/mnt/batch/tasks/shared/LS_root/mounts/clusters/bokun/code/Users/Bokun/Refinitive_real_time_data/market_data.py", line 277, in send_login_request

ws.send(json.dumps(login_json))

File "/anaconda/envs/azureml_py36/lib/python3.6/site-packages/websocket/_app.py", line 154, in send

if not self.sock or self.sock.send(data, opcode) == 0:

File "/anaconda/envs/azureml_py36/lib/python3.6/site-packages/websocket/_core.py", line 253, in send

return self.send_frame(frame)

File "/anaconda/envs/azureml_py36/lib/python3.6/site-packages/websocket/_core.py", line 279, in send_frame

l = self._send(data)

File "/anaconda/envs/azureml_py36/lib/python3.6/site-packages/websocket/_core.py", line 449, in _send

return send(self.sock, data)

File "/anaconda/envs/azureml_py36/lib/python3.6/site-packages/websocket/_socket.py", line 157, in send

return _send()

File "/anaconda/envs/azureml_py36/lib/python3.6/site-packages/websocket/_socket.py", line 139, in _send

return sock.send(data)

File "/anaconda/envs/azureml_py36/lib/python3.6/ssl.py", line 944, in send

return self._sslobj.write(data)

File "/anaconda/envs/azureml_py36/lib/python3.6/ssl.py", line 642, in write

return self._sslobj.write(data)

BrokenPipeError: [Errno 32] Broken pipe


And the stream would stop when the error occur. I understand that this problem would not occur if I do not process the MRN messages and put it on mongoDB but I am wondering why is it causing my streaming to stop? Has anyone else ran into similar problems when trying to process the streamed data (in my case doing the MRN processing and put in to mongoDB)? Also it is weird that one of the 3 errors would occur randomly.


Thank you!

People who like this

0 Show 0
Comment
10 |1500 characters needed characters left characters exceeded
▼
  • Viewable by all users
  • Viewable by moderators
  • Viewable by moderators and the original poster
  • Advanced visibility
Viewable by all users

Up to 2 attachments (including images) can be used with a maximum of 512.0 KiB each and 1.0 MiB total.

5 Replies

  • Sort: 
avatar image
REFINITIV
Best Answer
Answer by nick.zincone.1 · Apr 30, 2020 at 05:06 PM

Hi @bokun-a.wei,

Thanks for confirming.

You are correct that you are not the first one trying to use MRN over WebSockets. There are examples that retrieve MRN data from ERT in Cloud:

MRN example viewer using JavaScript

MRN example using RDP Library for .NET

And a number of other examples getting data via deployed ADS that are available within Refinitiv API Samples

When using MRN vs the raw code (which I assume subscribes to Market Price data) should be quite similar. That is, they both subscribe to data and capture updates within a callback. If you find that each individual example (WebSocket ERT in Cloud example and MRN example) works as expected independently, then there is likely an issue with how you merged the code. I would personally start simple - replace the subscription from MarketPrice data to MRN news and just echo the fact that you received data - no processing. This should work perfectly given the raw WebSocket ERT in Cloud code is likely doing something similar.

@Wasin Waeosri, you may be able to provide some troubleshooting steps regarding how to merge MRN code within your example.

Comment

People who like this

0 Show 0 · Share
10 |1500 characters needed characters left characters exceeded
▼
  • Viewable by all users
  • Viewable by moderators
  • Viewable by moderators and the original poster
  • Advanced visibility
Viewable by all users

Up to 2 attachments (including images) can be used with a maximum of 512.0 KiB each and 1.0 MiB total.

avatar image
REFINITIV
Answer by nick.zincone.1 · Apr 29, 2020 at 04:30 PM

Hi @bokun-a.wei,

Can you confirm how you are connecting into your streaming service? Via Elektron? Refinitiv Data Platform (ERT in Cloud)? Eikon? This is important to understand. You categorized your question under Eikon Data APIs but at this moment, MRN is not available via Eikon.

I'll assume you want to get MRN data from the Refinitiv Data Platform (RDP) given I can see from your stack trace you are trying to refresh tokens.

Given your application is rich with a combination of 2 examples (you linked above) plus the integration of Mongo DB, it would be difficult to narrow down at first glance. One area I would first look into is whether the access token expired, thus causing the WebSocket channel to close. I do see above that the code does refresh the access token so here are some suggestions.

Can you clarify random? Every few minutes? Is it consistent?

I would first run the basic streaming example (without MRN subscriptions/processing and Mongo DB) and observe the behavior. If you find the example runs perfectly for a long time, then the algorithm to get streaming data from RDP and refreshing access tokens is operating as expected. I would then include the code to process/manage MRN streaming data and continue from there.

Comment

People who like this

0 Show 0 · Share
10 |1500 characters needed characters left characters exceeded
▼
  • Viewable by all users
  • Viewable by moderators
  • Viewable by moderators and the original poster
  • Advanced visibility
Viewable by all users

Up to 2 attachments (including images) can be used with a maximum of 512.0 KiB each and 1.0 MiB total.

avatar image
Answer by bokun-a.wei · Apr 30, 2020 at 01:35 PM

Hi! Apologise for not making the post clear. I am using Refinitiv Data Platform (ERT in Cloud) as shown in the examples here (https://github.com/Refinitiv-API-Samples/Example.WebSocketAPI.Python.TestClient ). With some changes regarding process MRN messages.


The access token are being refreshed as seen from the log. And by random I meant sometimes it would run for a few min and stops itself, and sometimes it would run for half an hour to an hour before it stops itself. I have tried running the raw code without me processing and it works fine...but the problem occur when I added the MRN processing code to the pure ERT in cloud as shown in github. This is why I am posting on this forum to see if anyone has tried what I have done, which is using the ERT in cloud Websocket API code to get MRN message and process it. I assume I am not the first one trying to do this.

Comment

People who like this

0 Show 0 · Share
10 |1500 characters needed characters left characters exceeded
▼
  • Viewable by all users
  • Viewable by moderators
  • Viewable by moderators and the original poster
  • Advanced visibility
Viewable by all users

Up to 2 attachments (including images) can be used with a maximum of 512.0 KiB each and 1.0 MiB total.

avatar image
REFINITIV
Answer by zoya.farberov · May 15, 2020 at 02:06 PM

Hello @bokun-a.wei,

In addition to the suggestions by @nick.zincone.1 and @Umer Nalla,

Depending on your use case, your organization may be interested in the news already stored, please see full details on News Archive, you will need to login with your customer account credentials, if yes, please contact your Refinitiv account manager to discuss.

You may be interested in NewsAnalytics Database Store Example (MRN) example, it it uses EMA Java and stores MRN into MySQL db, demonstrates analogous concepts, so you may find useful hints.

Comment

People who like this

0 Show 0 · Share
10 |1500 characters needed characters left characters exceeded
▼
  • Viewable by all users
  • Viewable by moderators
  • Viewable by moderators and the original poster
  • Advanced visibility
Viewable by all users

Up to 2 attachments (including images) can be used with a maximum of 512.0 KiB each and 1.0 MiB total.

avatar image
REFINITIV
Answer by umer.nalla · May 18, 2020 at 09:20 AM

Hi @bokun-a.wei

Based on the information provided, both myself and colleague think this could be a case of Slow Consumer scenario.

i.e. the writes to the Mongo database are taking too long, your python script is not reading the data quickly enough from the server and eventually, the buffer (that the server reserves for your connection) overflows and the server disconnects your application.

With our other (Java/C++) APIs we usually recommend farming the database writes (or any other slower activity) to a separate thread.

Whilst not being a Python expert, I understand that Python is not truly multi-threaded– so not sure just how much that will help in this situation.


Comment

People who like this

0 Show 0 · Share
10 |1500 characters needed characters left characters exceeded
▼
  • Viewable by all users
  • Viewable by moderators
  • Viewable by moderators and the original poster
  • Advanced visibility
Viewable by all users

Up to 2 attachments (including images) can be used with a maximum of 512.0 KiB each and 1.0 MiB total.

Watch this question

Add to watch list
Add to your watch list to receive emailed updates for this question. Too many emails? Change your settings >
14 People are following this question.

Related Questions

Looking to get subjects and sentiment for a news article using Eikon python api

  • Feedback
  • Copyright
  • Cookie Policy
  • Privacy Statement
  • Terms of Use
  • Careers
  • Anonymous
  • Sign in
  • Create
  • Ask a question
  • Spaces
  • Alpha
  • App Studio
  • Block Chain
  • Bot Platform
  • Calais
  • Connected Risk APIs
  • DSS
  • Data Fusion
  • Data Model Discovery
  • Datastream
  • Eikon COM
  • Eikon Data APIs
  • Electronic Trading
    • Generic FIX
    • Local Bank Node API
    • Trading API
  • Elektron
    • EMA
    • ETA
    • WebSocket API
  • Legal One
  • Messenger Bot
  • Messenger Side by Side
  • ONESOURCE
    • Indirect Tax
  • Open PermID
    • Entity Search
  • Org ID
  • PAM
    • PAM - Logging
  • ProView
  • ProView Internal
  • Product Insight
  • Project Tracking
  • Refinitiv Data Platform
    • Refinitiv Data Platform Libraries
  • Rose's Space
  • Screening
    • Qual-ID API
    • Screening Deployed
    • Screening Online
    • World-Check One
    • World-Check One Zero Footprint
  • Side by Side Integration API
  • TR Knowledge Graph
  • TREP APIs
    • CAT
    • DACS Station
    • Open DACS
    • RFA
    • UPA
  • TREP Infrastructure
  • TRIT
  • TRKD
  • TRTH
  • Thomson One Smart
  • Transactions
    • REDI API
  • Velocity Analytics
  • Wealth Management Web Services
  • World-Check Data File
  • Explore
  • Tags
  • Questions
  • Badges