I'm connecting using the Java API
Hello @vuk.magdelinic,
I would differentiate two types of possible issues with the described scenario:
1. Code related:
If you run a standard example that came with SDK, with the same connection parameters as your custom application and it does not drop connection, then the issue is likely related to custom application code and needs to be investigated from the code perspective. One of the possible approach is divide and conquer - separate and remove pieces of custom code till you find a point when the connection drop does not occur, and that should allow to at least narrow down the possible causes and consequently come up with the solution.
2. Connectivity and infrastructure related:
If the standard example drops connection at the same or very close pattern, you are looking at the connectivity or infrastructure related issues. If you are connecting to Refinitiv Realtime Optimized or Refinitiv hosted, the first step is to check with Refinitiv Helpdesk Online -> Notifications and Alerts -> Service Alerts and if you don't find a relevant alert, open support ticket with Refinitiv Helpdesk Online to help you investigate the issue. If you are connecting to your local deployed infra, contact your market data/network admin/group to investigate the issue.
I hope this information helps to proceed toward the resolution of the connection issue
Here's the error I get:
com.refinitiv.ema.access.OmmInvalidUsageExceptionImpl: RDMLogin stream was closed with status message
username XXXXXX
usernameType 1
State: Closed/Suspect/Internal error from sender - text: "Login Denied:Failed to validate sso token"
In order for us to try to help further, please provide the complete details:
Versions: of RTSDK or Elektron SDK, EMA or ETA, language, OS?
Infra connectivity: is it Refinitiv Realtime Optimized in cloud, Refinitiv Realtime Hosted? a deployed infrustructure?
Is the issue reproducible with one of the standard examples that came with your SDK, or only in a custom app?
Is my understanding correct that you establish connectivity and subscription, you observe streaming data for 15-20 mins and only then your connection drops, and you observe this error message?
Any other relevant information or details.
Thanks
We use Java EMAfactory to connect to ELECKTRON_DD service.
Pls point us to the standard examples as we couldn't find them.
We establish connection, it lasts for 15-20mins and then dies without any error. Sometimes when we try to reestablish connection, we get the error pasted earlier. Other times we get the following error:
java.lang.NullPointerException: null
at com.thomsonreuters.ema.access.SingleItem.rsslSubmit(ItemCallbackClient.java:3047) ~[MarketPriceChain_KPL.jar:?]
at com.thomsonreuters.ema.access.SingleItem.close(ItemCallbackClient.java:2902) ~[MarketPriceChain_KPL.jar:?]
at com.thomsonreuters.ema.access.ItemCallbackClient.unregister(ItemCallbackClient.java:2301) ~[MarketPriceChain_KPL.jar:?]
at com.thomsonreuters.ema.access.OmmBaseImpl.unregister(OmmBaseImpl.java:509) ~[MarketPriceChain_KPL.jar:?]
at com.thomsonreuters.ema.access.OmmConsumerImpl.unregister(OmmConsumerImpl.java:167) ~[MarketPriceChain_KPL.jar:?]
at com.thomsonreuters.platformservices.elektron.objects.marketprice.MarketPriceImpl.close(MarketPriceImpl.java:156) ~[MarketPriceChain_KPL.jar:?]
at MarketPriceASKBID.SwapRates.fetchSwapData(SwapRates.java:201)
Are there any timeouts/timelimits before a connection expires and needs to be restarted?
Last day, we had uninterrupted service for 4hrs and then it dropped twice
I believe you are connecting to Refinitiv Real Time Optimized ( RRTO, also formerly Elektron Real in cloud).
I do not know what SDK you are working with, you appear to be using EMA Java.
In order to determine if the disconnects are related to connectivity interrupts, infra outages or are code-related, would like to suggest to run a simple standard EMA Java consumer that came with your SDK.
For instance, in RTSDK 2.0 examples folder:
...\Ema\Examples\src\main\java\com\refinitiv\ema\examples\training\consumer\series400\ex430_MP_Authentication
Example 430 = MarketPrice Authentication,
And see if it faces the same connection issues as your custom code.
Does this help, are you able to proceed?
Hi I hope you have read my responses to your questions.
We are using EMA (ema-3.6.0.0.jar) in JAVA. On Unix
We are connecting to RSSL_Encrypted
Host: amer-3-t2.streaming-pricing-api.refinitiv.com Port 14002
Does this give you some clarity??
As explained multiple times, sometimes we see null pointer exception Error message, sometimes no error message
In the Java jar file that I'm using I dont see \Ema\Examples\src\main\java\com\refinitiv\ema\examples\training\consumer\series400\ex430_MP_Authentication
I see the following:
Thank you for this confirmation. Appreciate your patience. You are on the latest RTSDK 2.x.
As you mention you run on Unix, please refer to the latest API Compatibility Matrix, tab Operating System. I do not believe any UNIX/Solaris are certified for RTSDK/EMA. This is very likely the root cause of the problem.
However, let me try to learn more. Both pre-built versions of RTSDK from Downloads and the complete source version available from GitHub RTSDK Repository includes complete set of our examples. Let's try to step back, do you see Examples folder that came with your version of RTSDK? This directory would not be part of the library itself, within EMA jar, this would be part of SDK, next to Docs and Libs?
Will you able to step back, and test with our starter example, I would suggest following EMA Quickstart - Connecting to RRTO for complete sequence of steps, on one of the certified platforms, to confirm if the issue persists?
Hi @vuk.magdelinic
Are you by any chance using the same Machine ID / password for multiple applications making multiple connections to RRTO ?
@chavalit-jintamalit Error I'm getting is this: (are you saying this is because of multiple connections?)
java.lang.IllegalMonitorStateException
at java.util.concurrent.locks.ReentrantLock$Sync.tryRelease(ReentrantLock.java:151)
at java.util.concurrent.locks.AbstractQueuedSynchronizer.release(AbstractQueuedSynchronizer.java:1261)
at java.util.concurrent.locks.ReentrantLock.unlock(ReentrantLock.java:457)
at com.refinitiv.eta.transport.ReentrantLock.unlock(ReentrantLock.java:28)
at com.refinitiv.eta.transport.RsslHttpSocketChannel.read(RsslHttpSocketChannel.java:1031)
at com.refinitiv.eta.valueadd.reactor.Reactor.performChannelRead(Reactor.java:2816)
at com.refinitiv.eta.valueadd.reactor.Reactor.dispatchChannel(Reactor.java:2481)
at com.refinitiv.eta.valueadd.reactor.ReactorChannel.dispatch(ReactorChannel.java:574)
at com.refinitiv.ema.access.OmmBaseImpl.rsslReactorDispatchLoop(OmmBaseImpl.java:1487)
at com.refinitiv.ema.access.OmmBaseImpl.dispatch(OmmBaseImpl.java:598)
at com.refinitiv.ema.access.OmmConsumerImpl.dispatch(OmmConsumerImpl.java:185)
at com.refinitiv.platformservices.elektron.objects.common.DispatcherImpl.dispatchEventsUntilComplete(DispatcherImpl.java:49)
at com.refinitiv.platformservices.elektron.objects.marketprice.MarketPriceImpl.waitForCompletion(MarketPriceImpl.java:346)
at com.refinitiv.platformservices.elektron.objects.marketprice.MarketPriceImpl.open(MarketPriceImpl.java:146)
at MarketPriceASKBID.SwapRates.fetchSwapData(SwapRates.java:142)
I am looking at this error.
Do you use the same Machine ID on more than one connection?
Or did you by any chance happen to login with the same machine ID before around the time you got this error?
Are you using the latest EMA API?
Do you enableSessionManagement on EMA API?
When you log in to get token, did you pass in takeExclusiveSignOnControl as True or False?
@chavalit-jintamalit pls look at the below error instead: (because Login denied error was probably when we were trying to reconnect the session every 15-20mins automatically)
Java.lang.IllegalMonitorStateException
can you. pls advise why I see this error:
I'll look into the links you have provided. thanks!
Hi @vuk.magdelinic,
Can you please confirm, if you observe this error on one of the supported OSs, according to API Compatibility Matrix and with one of the simple RTTO examples that came with EMA Java SDK, I would suggest either EMA Java MarketPrice 430 (service discovery) or if you prefer to test with the specific endpoint, please try EM Java Market price 113.
This Quickstart Guide will take you through the quick steps on how to get started with example.
Once the minimal example is running for you successfully, you can build up to your custom requirements.
Does this make sense?
@zoya faberov @chavalit-jintamalit did you guys look at this error? any update?
Looking for the confirmation from you, that you are running on a supported platform according to the latest API Compatibility Matrix?
Were you able to proceed as suggested, and reproduce the issue that you describe with connection being dropped after 15-20 min (that can potentially be caused by the Exception listed) with a simple standard example, as your connectivity is Refinitiv Real Time Optimized, one of the best suited examples that would be EMA Java Market Price Consumer 450, steps to run described in EMA Quickstart - Connecting to RRTO?
That would narrow down the causes significantly for you. If the standard example reproduces the same issue - the issue is due to network connectivity or infra outages, if it does not, streams unabated- it's likely origin is in your custom application code.
Further, as suggested by @chavalit-jintamalit, please advise, if several connections are made with the same machine ID in parallel, and if you only run one connection at a time, does the issue still manifest?
@zoya faberov we dont seem to have access to the links you have provided. can you pls share them to us in an email?
Ok confirmed platform is supported as per the API compatibility matrix
As mentioned several times, only 1 connection is being made
Please be more specific. Are you running on Unix, as you have mentioned before?
You will need to be logged into dev portal to access these materials:
Quickstart guide is made available as online documentation, with linkage to other relevant and related documents.
Thanks @vuk.magdelinic,
This eliminates a subset of possible issues.
The code looks reasonable, however, troubleshooting custom code will not be reasonable in the framework of the discussion forum.
Do you have the complete SDK? SDK can be downloaded from Downloads, you will need to be logged in and may need to "request download" the first time.
Otherwise, SDK is open source and so the complete source code can also be downloaded from RTSDK on GitHub and built from source according to Installation Guide found in docs.
The example, in both cases, will be in ... \Ema\Examples, the example of interest, ServiceDiscovery, is com\refinitiv\ema\examples\training\consumer\series400\ex450_MP_QueryServiceDiscovery
Let us know how this works for you
I have looked into this example but I cant find the below package/class being used: Pls advise since this is the class we are using to fetch data
package com.refinitiv.platformservices.elektron.objects.marketprice;
Our examples use Refinitiv libraries that are part of RTSDK:
Have you been able to run the example successfully as described in Quickstart guide?
---
There was an influential article published, believe that it was at the introduction of Elektron SDK (currently Relatime SDK) EMA libraries, A simple MarketPrice object for EMA describing how to build value-add objects on top of EMA.
There was companion code included with the article, to illustrate the concepts that were explained by the author in the article, hosted on GiHub, that was last updated by the author 2 years ago, major updates 3 years ago, from what I see.
I read in Readme on GitHub that the article code is tested with EMA 1.1.0. This is a very old version and pre-dates not just the latest major version of RTSDK 2.0, but several versions before it.
I would suggest using the latest EMA from RTSDK 2.0 that you mention you use, directly, since the publication of the article, many issues have been fixed, many new features became available, and this is what you wish to use.
The techniques demonstrated in the article remain valid, and can be used for creating value- added objects, as suggests at the end "It should give you good indications for implementing the same kind of features in your own applications", however would not suggest using the companion code as is, would at least need to verify the parts that you wish to use with the current version of RTSDK prior to doing so, in my view.
Thanks for your response! If I skip using the value-add objects, then I would need to find the FID values for the fields I'm pulling. can you please help me where I can get those?
For ex -
theMarketPrice.getField("MID_PRICE").value().toString());
Instead of the above, I would need to do something like this right?
OmmArray array = EmaFactory.createOmmArray(); array.fixedWidth(2); array.add(EmaFactory.createOmmArrayEntry().intValue(???-what should go here?))
You can locate the data dictionary files under \Real-Time-SDK-2.0.1.L1.java\RTSDK-2.0.1.L1.java.rrg\Java\etc folder.
From RDMFieldDictionary:
MID_PRICE "MID-PRICE" 134 NULL PRICE 17 REAL64 7
I assume that you are trying to use "View" feature when you make a subscription.
Please refer to example 360.
The "View" feature limits the fields in the refresh or update message that your application receives regarding the data subscription.
You can add more fields and you make a subscription:
@chavalit-jintamalit do you know why when I use EMA directly as explained by you, I dont seem to get data for many of the fields but using ValueAddObject interface gets me data for those fields?
You provided a snippet code which is using "view" feature.
So I was assuming that you tried to achieve what "view" feature does.
When you refer to the sample code from EMA, it is suggested that you also review the readme file for each sample as well.
Summary=======The ex360_MP_View application is provided as an example of OMM Consumerapplication written to the EMA library.This application demonstrates basic usage of the EMA library for opening itemstream with a list of field IDs called "View" and parsing of OMM MarketPricedata from Refinitiv Data Feed Direct (RDF-D), directly from an OMM Providerapplication, or from an Advanced Distribution Server.ex360_MP_View showcases usage of view request feature supportedby OMM Consumer and server.
Please refer to this article for view (and batch) features.
@chavalit-jintamalit yes I'm using ex 360. But can you tell me when the refresh/update happens then do we get all the values/fields that we specified in the view or just the updated fields out of those?
For ex - I subscribe to Ask, Bid and Display name
OmmArray array = EmaFactory.createOmmArray(); array.fixedWidth(0); array.add(EmaFactory.createOmmArrayEntry().intValue(3)); array.add(EmaFactory.createOmmArrayEntry().intValue(22)); array.add(EmaFactory.createOmmArrayEntry().intValue(25);
But when I receive UpdateMsg, I only get Ask and Bid not Display Name. Is this expected behavior?
The update message only contains the fields that are updated.
In this case, Display Name value is not updated(compare to the previous value in the refresh message) so it won't be included in the update message.
How can I get all the values and not just the updated ones? When using ValueObject MarketPrice I was getting all values regardless of them being updated or not
You can obtain all fields, only when you need them, by using Snapshot- specifically requesting snapshot when you need it, Snapshot request example can be found with RTSDK EMA Java - Example Market Price 102 - Snapshot
Or you can cache/store the required static values, and retrieve and combine them with every update, analogous to what Simple Market Price Object handles.
I used interestAfterRefresh(false) to get snapshot but now it seems I dont get any message after the initial refreshMsg.
My Thread.sleep is set to 1 sec.
Why??
@zoya faberov
But in ex 102, it has a thread.sleep of 60 secs. So how is it getting the snapshots every 60 secs is what I'm trying to understand and replicate at my end.
Should I be calling this every few seconds to get the snapshot? Because I dont see that in ex 102
consumer.registerClient(reqMsg.serviceName("DIRECT_FEED").name("IBM.N").interestAfterRefresh(false), appClient);
@vuk.magdelinic,
It is not getting snapshots every 60 sec. This example is timing it's run and exiting after 60 secs
If you are looking to tweak RTSDK EMA Java example 102 to snapshot every 60 sec, how about something like this, please replace the relevant part of the code in your example:
try { AppClient appClient = new AppClient(); OmmConsumerConfig config = EmaFactory.createOmmConsumerConfig(); consumer = EmaFactory.createOmmConsumer(config.host("YOURHOST:14002").username("YOURUSER")); ReqMsg reqMsg = EmaFactory.createReqMsg(); while(true) { consumer.registerClient(reqMsg.serviceName("YOURSERVICE").name("IBM.N").interestAfterRefresh(false), appClient); Thread.sleep(60000); // API calls onRefreshMsg(), onUpdateMsg() and onStatusMsg() } }
this will run continuously and snapshot every 60 secs.
Let us know how this works for you?
Thanks @zoya faberov
This is exactly what I did. And it ran fine. I was getting the snapshots of the records with values for all fields not just updated ones.
I kept it running, however, when I checked after 90mins, it failed with the below error: (code is exactly similar to what you have mentioned - I'm calling registerClient on consumer after every 1 sec instead of 60 sec)
Exception in thread "main" java.lang.NullPointerException at com.refinitiv.ema.access.SingleItem.rsslSubmit(ItemCallbackClient.java:2994) at com.refinitiv.ema.access.SingleItem.open(ItemCallbackClient.java:2856) at com.refinitiv.ema.access.BatchItem.open(ItemCallbackClient.java:3823) at com.refinitiv.ema.access.ItemCallbackClient.registerClient(ItemCallbackClient.java:2156) at com.refinitiv.ema.access.OmmBaseImpl.registerClient(OmmBaseImpl.java:457) at com.refinitiv.ema.access.OmmConsumerImpl.registerClient(OmmConsumerImpl.java:143) at EMAConsumer.Consumer.main(Consumer.java:548)
Please confirm that you are still connecting to RTTO?
And the basis of your code is example MP 450 as per Quickstart Guide? (you have just added the loop to it) or you are doing something different?
If this is exactly the case, I can try to reproduce, by running the example with snapshots for 90 min+.
I have modified RTSDK EMA Java Example 450 (RRTO) to request snapshots every 60 sec and left it running for over two hours.
My modification of the example was:
ReqMsg reqMsg = EmaFactory.createReqMsg(); while(true) { consumer.registerClient(reqMsg.serviceName("ELEKTRON_DD").name(itemName).interestAfterRefresh(false), appClient); Thread.sleep(60000); // API calls onRefreshMsg(), onUpdateMsg() and onStatusMsg() }
I did not reproduce the NullPointer exception.
I attach the complete zipped code as well:
Consumer_450_SnapshotLoop.zip
Is this identical to what you have modified and that have run and your run has resulted in the exception?
Does the issue reproduce consistently in your environment by running this code?
If it is not the same, please run the attached, and let us know how this works for you?
Yes my code is exactly the same as you provided except you are refreshing at 60sec and I'm refreshing at 1sec. Even when I run your code it fails after sometime. The time when it fails is not consistent.
Can you try running running at your end for 1 sec and see if you reproduce the issue?
Snapshotting every second is not a sound design approach. The snapshot is comprised of registration, receipt of the image or status, complete cleanup of the registration, all handled asynchronously on your behalf, but required.
You have said before "getting snapshots every 60 seconds" is what you were looking to do, and I have tried to help you with design approach and providing the example.
RRTO is already a conflated stream, quotes capped at 3 updates per sec, trade-safe. You will not be facing the situation, for example, with 100+ updates per second.
If you require to have the complete image, all fields, refreshed, every second, I would suggest as your custom consumer design approach:
The article, the companion code of which you have originally used A simple MarketPrice object for EMA discusses a design approach to analogous task (without understanding the complete requirement can not tell if this is a 100% match with your requirements). It does state however:
"The MarketPrice class and the ValueAddObjectsForEMA example library have been written by Refinitiv for the only purpose of illustrating a series of articles published on the Refinitiv Developer. The MarketPrice class and the ValueAddObjectsForEMA example library have not been tested for usage in production environments. Refinitiv cannot be held responsible for any issues that may happen if these objects or the related source code is used in production or any other client environment. "
However, as the article and its code pre-date RTSDK and even RRTO, in my understanding, the article's material can be helpful to learn of the concepts involved and the approach that was used, but you would have to design, for the approach illustrated, rather then use the code as is.
I hope this helps
@zoya faberov what is the recommended and the minimum time to get snapshots? 60 secs is too long, can I go with snapshots every 10sec lets say?
It's a totally different question from what we have started with, and it is better to ask a new question as a new question, while including the link to the related previous discussions, so that the answers are readily visible and easily findable by the next developers with same ot similar question.
However, I hope this discussion is useful to you?
There is no "blanket" answer to this question. The best approach is to do a thorough testing in your local environment and once we identify what works, we always prefer to err on the side of safety. I.e. consider doubling the interval that you find to work safely for your instrument list size in your environment and under your network and internet conditions.
Attached is a rough port of A simple MarketPrice object for EMA article code, included are:
From my testing, the example runs as expected and does not drop connection with arguments:
-wu -s ELEKTRON_DD -u MACHINEID -p PASSWORD -kf KEYSTOREFILEWITHPATH -kp KEYSTOREPASSWORD -c CLIENTID JPY=
Article.EMA.Java.OlivierDavant.RTSDK20QuickPort.zip
Let us know if you find this useful
What is the refresh rate you have used in this? We want 1 sec refresh with full data.
As you have stated ValueAddedObject was not working right for you and you were most interested in getting ValueAddedObject working right, with RRTO, I went back and invested time into rough-porting it to RTSDK 2.0.
I have run MarketpriceSubscriber object, with the parameters noted above.
1 sec refresh:
This is streaming request, every update is received. RRTO connectivity is 3 quotes per sec, conflated stream, trade-safe. If you wish you can process the first update in a sec, and ignore the rest, of course.
Full data:
"Objects, like MarketPrices or Order Books that cache (retain) in memory the latest values sent by the platform so that your application can randomly get this data anytime and from any thread".
You can retrieve the full set of fields at any time, including on update. Where you have in code
print(marketPrice, UPDATE, update);
you can add
print(marketPrice, "ALLFIELDS", marketPrice.getFields());
or you can also getField(name) at any time, and only get the fields you need, not all couple of hundred of fields in the image.
Another option is creating a separate thread and sampling MarketPrice every second from that thread.
The attached should be helpful in giving you a solid start, allowing to next implement your requirements using it as a working base.
