Using HTTPS proxy in EMA only for authentication, not for market data

We are using the EMA API with AWS Endpoints to obtain market data.

We have direct connectivity to the endpoint both within AWS and on-premises (using AWS Direct Connect).

We note that even when using an AWS endpoint, the authentication process (obtaining an access token) required to talk to the endpoint needs to be done over the Internet, as 'api.refinitiv.com' is not reachable any other way.

While this is no problem in AWS (where we have native routed Internet connectivity via NAT), it is a problem on premises, because the use of an HTTPS proxy is required to access 'api.refinitiv.com', but that proxy should NOT be used for accessing the AWS endpoint over Direct Connect.

When we use EmaFactory.createOmmConsumer(), we have to configure the 'tunnelling proxy', but doing so results in EMA attempting to connect to the AWS endpoint over the HTTPS proxy, which is incorrect; we want to get the authentication token using the proxy, but connect to the endpoint directly.

How can we work around this? It seems to be a catch-22, because if we enable the proxy we can't reach the AWS endpoint, but the AWS endpoint requires a token that can only be fetched via the proxy.

Looking at the EMA Python examples it seems that the authentication function can be severed from the market data consumption, but that doesn't seem to be the case in the Java API.

Any hints on how we can do this would be welcome. Until then, we have to hack around the situation by making 'api.refinitiv.com' point to an internal host with Internet connectivity which then forwards anything received on port 443 to the real 'api.refinitiv.com', but I would prefer to avoid having to do that.

Find more posts tagged with

error

proxy

elektron

elektron-sdk

refinitiv-realtime

rrt

elektron-message-api

ema-api

Accepted answers

Gurpreet

Hi @eWarrant

I don't completely understand your infrastructure setup and unable to offer any advise, but just like python sample, you can use websockets API with Java as well. Please see this service discovery sample (MarketPriceEdpGwServiceDiscovery.java) in the github. Please note that this does not use EMA API (and hence RSSL protocol), which can be of advantage in relatively high throughput consumers.

If you need specific help with the EMA Java consumer, I would recommend that you raise an issue with Realtime SDK Github, where you can also view the EMA source code.

All comments

Gurpreet

Hi @eWarrant

If you need specific help with the EMA Java consumer, I would recommend that you raise an issue with Realtime SDK Github, where you can also view the EMA source code.

eWarrant

Hi Gurpreet,

WebSocket does not meet our requirements: we need to make use of EMA because that's what fits what we have already developed.

The issue is very simple - we want EMA to decouple the 'get the authorization token' from the 'consume market data' function.

Consider the case where EMA is being used to connect to a Refinitiv AWS endpoint (this is a service you offer and one that we are currently using, please ask internally if you are unsure): in this case, rather than using service endpoint discovery (which returns hostnames similar to 'amer 1.pricing.streaming.edp.thomsonreuters.com'), we will be connecting directly to the endpoint's IP address.

Thus, the connection is entirely within our network (we consider VPCs within AWS accessed over Direct Connect to be 'our network'), similar to what you would get with an on-premises Elektron Edge device.

However, unlike consuming from an EED, the endpoint requires an authentication token that must be obtained from <api.refinitiv.com> over the Internet. We consider this an extremely poor design choice and have discussed it with our Refinitiv contacts, but they are not able to do anything about it.

Due to infrastructure security policies, machines that are consuming market data are not permitted direct access to the Internet: this is only available via a proxy.

Using the current EMA implementation, EMA fetches the authentication token itself, then uses the same proxy settings to attempt to talk to the endpoint using RSSL. This will fail, because the endpoint is local: it is not to be reached through the proxy.

Since Refinitiv is offering the AWS endpoint service, we expect the EMA API to be enhanced so that the proxy can be used just to fetch authentication tokens, and not for the actual RSSL connection itself.

The WebSocket implementations behave that way, but WebSocket does not meet our needs, so we have to work with EMA.

I will look at the source code, but I would prefer not to have to patch it, as it is likely to result in issues in the future as we would need to fork it. It is much better if Refinitiv just properly supports the AWS endpoint service by either a) extending the authentication service to the endpoint, or b) improving EMA to handle the 'use a proxy to get authentication tokens, don't use a proxy for RSSL connectivity' requirement.

You can be sure that other customers will have this issue when attempting to integrate your ERT Cloud market data solution using AWS endpoints.

Gurpreet

I will ask around internally, but it is best that you raise a git-issue in the EMA repository. This will directly get you in touch with the development group.

EXPLORE OUR SITES