EIKON DEPRECATION: Fields that are returned from rd.eikon.get_data are not returned when using rd...

....get_data

I recently noticed the deprecation notice about Eikon on RD 1.5.0:

FutureWarning: The refinitiv.data.eikon module will be removed in future library version v2.0. Please install and use the 'eikon' Python library instead or migrate your code to the Refinitiv/LSEG Data Library

So I decided to check if I could transition to Refinitiv Data without rewriting and checking all my fields/formulas. It turns out that this is not possible. I need to check each field individually if it is returned by rd.getdata()

Take for example the following code snippet:

import refinitiv.data as rd

rd.open_session()

fields = ["CF_CURR", "CF_EXCHNG", "TR.PriceClose.Currency", "TR.InstrumentType", "TR.ExchangeName", "DSPLY_NAME"]

EikonData = rd.eikon.get_data(instruments= "AAPL.O", fields= fields)
print(EikonData)
print("-----------")
RD_Data = rd.get_data(universe="AAPL.O", fields= fields)
print(RD_Data)

As can be observed from below rd.get_data() does not return:

CF_CURR
DSPLY_NAME
CF_EXCHNG

As Eikon will be deprecated at a currently to me unknown date it is extremely important to know upfront which fields will be returned correctly and which will not be returned. I cannot and I don't want to deal with this kind of unexpected surprises in automated workflows.

Therefore a complete list of fields that are no longer working in rd.get_data() but are in rd.eikon.get_data() has to be made available!

Finally the lack of communication about the changes in RD is appalling. This is not a 100 USD a month service! Why is there no mailing list that automatically sends me updates on new versions and release notes within these versions. I pay serious money for this service and expect for this also a professional handled API service. Not some cowboys that randomly change the content of functions and then suddenly deprecates the cornerstone module in the API without any help or thought about transitioning!

Also maintaining two libraries (Eikon and RD) in one project is terrible and for sure unadvisable. It has already occurred that RD and Eikon python package had conflicting dependencies, which resulted in the situation that both packages could not be installed in the same environment!

eikon code book snapshot proving that this is a real problem:

Find more posts tagged with

eikon-data-api

refinitiv-dataplatform-eikon

refinitiv-data-platform

#product

workspace-upgrade

Accepted answers

olivier1.davant2

Dear Laurens,

I am the product manager for the Eikon Library and Refinitiv Data Libraries. Apologies for jumping into this conversation that late and – as you pointed out – for the communication that could have been handled better. This has created some confusion which I would like to clarify. Hopefully, this will give you better visibility into the upcoming changes and the potential impact on your code and your applications.

First, I would like to explain our strategy for the Eikon library and Refinitiv Data Libraries:

We continue to support and maintain the Eikon Library
This library is "Feature Complete", but we continue to support it and provide maintenance releases.
The Refinitiv Data Library will be renamed the LSEG Data Library (LD Lib)
Due to LSEG's acquisition of Refinitiv, Refinitiv products and libraries are rebranded as LSEG. Rebranding a library means changing its name, which is a breaking change. This change will occur in version 2.0 of our data library.
We are taking advantage of the name change of our data library to slightly improve its public interfaces
Some aspects of the public APIs will be slightly improved in the LSEG Data Library 2.0. The “eikon” module which is a duplicate of the Eikon library will be removed.
We will continue to support and maintain the Refinitiv 1.x data library
After the rebranding and release of LSEG Data Library 2.0, the Refinitiv Data Library will become " Feature Complete", but we will continue to support it and provide maintenance releases. This is essential for customers who cannot migrate to the LSEG Data Library 2.0. However, new features will only be added to the LSEG data library.
The “eikon” module will not be removed from the Refinitiv 1.x data library
The FutureWarning you noticed was about the change to the LSEG Data Library 2.0. Maintenance releases of RD Lib will continue to include the “eikon” module. There are no plans to remove it.
We will ensure that there are no dependency conflicts between the Eikon library, the Refinitiv data library and the LSEG data library
This is essential for CodeBook and many of our customers. This check will be integrated into our automated tests.
The underlying data sources used by the LSEG Data Library's get_data() function will be fully aligned with those used by the Eikon Library's ek.get_data()
This will ensure a smooth migration from the Eikon library to the LSEG Data Library.

In other words, you can continue to use the Refinitiv Data Library as you do today. New versions of this library will continue to include the “eikon” module. Whenever and wherever you want, you can migrate to the LSEG Data Library at your own pace. You will be able to use it in the same environment as the Eikon Library and the Refinitiv Data Library

I hope this information provides you with more clarity. My understanding is that these changes will not impact your code and will give you the flexibility to migrate to the LD Lib whenever you want and need to.

All comments

andrii.sidachenko01

Hello, Laurens.

Thank you for your feedback.

Future v2 version (2.0.0) is a major version that allow to have breaking changes. If any breaking changes will we have - we should introduce major changing version (https://semver.org/).

This is a normal practice to introduce breaking changes in product.

That’s why library have FutureWarnings in advance to let users know that changes will take place.

If you are currently use Eikon, you can still use it as separate package.

Regarding release news.

As soon we released new version, immediately we provide release notes for this version - https://developers.lseg.com/en/api-catalog/refinitiv-data-platform/refinitiv-data-library-for-python/documentation#release-notes

I ran you code example. Please take a look results below for 1.5.0 version. I attached the log file.

apple.req.log.zip

laurens

Dear Andrii,

1. Regarding deprecation.

Suppose you run a railway network and you suddenly decide to change the track width. You can imagine that all your customers that have trains who can only run on the old track will not be happy that they can not run their perfectly fine trains on the track anymore.

As stated before you always need a combination of RD and Eikon (let's say e.g. that you have perform a RD search command and then get an eikon timeseries command). Therefore you need both packages. Both packages can not be reliably used together due to conflicting dependencies (which is already currently the case) between the existing RD and Eikon python packages.

I don't want to get in a discussion regarding versioning semantics. The fact that something can be done doesn't mean that something should be done.

2. Regarding rd_GetData()

I am highly surprised you get the answer as stated above. Both in my own python environment and in Eikon CodeBook i am not able to reproduce this...

I checked what version codebook was running (this was 1.2.0) so that could explain something. However: Refinitiv is responsible for Codebook and this command is clearly not returning what it should be returning

Then I ran again the 1.5.0 code in my own environment and also could not reproduce your results (and please find my log attached):

import refinitiv.data as rd

rd.__version__

rd.open_session()missingfields.txt

fields= ["CF_CURR", "CF_EXCHNG", "TR.PriceClose.Currency", "TR.InstrumentType", "TR.ExchangeName"]

EikonData = rd.eikon.get_data(instruments= "AAPL.O", fields= fields)

print(EikonData)

RD_Data = rd.get_data(universe="AAPL.O", fields= fields)

print(RD_Data)

print(RD_Data.columns)

3. Regarding communication

let me ask 3 questions:

- Have users been asked if eikon could be removed and when?

- Has Refinitiv ever actively reached out to users about these changes? Wouldn't one expect a news letter that contains some sort of roadmap.

- if I pay thousands of pounds per month to use this service would one expect this deprecations?

4. Conclusion

I would like conclude that it is not normal practice that the future legacy function receives data and then it suddenly disappears when I use the next version of that function.

Furthermore active communication reaching users about product planning is lacking. Every week I get emails from a nice training manager about how I can do something in excel. Not a single time i receive an email regarding the api which all my workflows use...

Furthermore it is extremely disappointing that no questions are asked to users about why they pay for the product and what they expect from the product...

I don't use this product because I like programming. I use this product to earn money.

laurens

please find my response below, somehow I did not want to be pasted here.

andrii.sidachenko01

Laurens, thank you for the feedback.

Regarding rd.get_data().

As I see at your log file you have the permission issue. You've got a server response message:

{
  "ID": 5,
  "Type": "Status",
  "Key": {
    "Service": "IDN_FD3",
    "Name": "AAPL.O"
  },
  "State": {
    "Stream": "Closed",
    "Data": "Suspect",
    "Code": "NotEntitled",
    "Text": "Access Denied: User req to PE(74)"
  }
}

More generally, permissions (list of PEs) give access to specific markets (by region and/or by domain). PE (Permission Entitlement) give access to list of instruments.

I would recommend contact your Refinitiv Account team or Sales team directly to verify the permission to the requested data.

laurens

perfectOMMstream.txtDear Andrii,

Thank you for your reply, however I am bit disappointed that you only looked at the last lines of the logfile and did not try to identify the root cause of the difference between rd.eikon.get_data() and rd.get_data()

I was extremely surprised (again) that in rd.eikon.get_data() I don't have a permission error and that I suddenly have a permission error for a live data stream [OMM] in rd.get_data() while these fields ["CF_CURR", "CF_EXCHNG"] are simply static character fields that do not have much in common with live data as they won't change in time.

So I checked the previously supplied log (missingfields.txt) in detail which api (json) calls are made:

1. rd.eikon.get_data() call

HTTP Request id 2
url = http://localhost:9060/api/udf
method = POST
headers = {'Content-Type': 'application/json', 'Authorization': 'Bearer eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJBcGlWZXJzaW9uIjoiMSIsIkFwcEtleSI6IjEyMDU0YjRlN2JiMTRlZTU4YWM2MzQ3MjQ4ZDMzNzg2MGY1MmUxZDUiLCJBcHBTY29wZSI6InRyYXBpIiwiTGlicmFyeU5hbWUiOiJSRFAgUHl0aG9uIExpYnJhcnkiLCJMaWJyYXJ5VmVyc2lvbiI6IjEuNS4wIiwiaWF0IjoxNzAxMTIxMDEyLCJleHAiOjE3MDIzMzA2MTJ9.RobPtR-hlTdO_pDm-MbVrrq2h7lTSdQ-qr7XnLZyRis', 'x-tr-applicationid': 'DEFAULT_WORKSPACE_APP_KEY'}
json = {'Entity': {'E': 'DataGrid_StandardAsync', 'W': {'requests': [{'instruments': ['AAPL.O'], 'fields': [{'name': 'CF_CURR'}, {'name': 'CF_EXCHNG'}, {'name': 'TR.PriceClose.Currency'}, {'name': 'TR.InstrumentType'}, {'name': 'TR.ExchangeName'}]}]}}, 'ID': '123'}

as you can see clearly here udf is called for ["CF_CURR", "CF_EXCHNG"]

fields': [{'name': 'CF_CURR'}, {'name': 'CF_EXCHNG'}

is in the body of this json request

2. When I look then in rd.get_data()

HTTP Request id 3
url = http://localhost:9060/api/udf
headers = {'Content-Type': 'application/json', 'Authorization': 'Bearer eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJBcGlWZXJzaW9uIjoiMSIsIkFwcEtleSI6IjEyMDU0YjRlN2JiMTRlZTU4YWM2MzQ3MjQ4ZDMzNzg2MGY1MmUxZDUiLCJBcHBTY29wZSI6InRyYXBpIiwiTGlicmFyeU5hbWUiOiJSRFAgUHl0aG9uIExpYnJhcnkiLCJMaWJyYXJ5VmVyc2lvbiI6IjEuNS4wIiwiaWF0IjoxNzAxMTIxMDEyLCJleHAiOjE3MDIzMzA2MTJ9.RobPtR-hlTdO_pDm-MbVrrq2h7lTSdQ-qr7XnLZyRis', 'x-tr-applicationid': 'DEFAULT_WORKSPACE_APP_KEY'}
json = {'Entity': {'E': 'DataGrid_StandardAsync', 'W': {'requests': [{'instruments': ['AAPL.O'], 'fields': [{'name': 'TR.PRICECLOSE.CURRENCY'}, {'name': 'TR.INSTRUMENTTYPE'}, {'name': 'TR.EXCHANGENAME'}]}]}}}

I can then clearly see that here the fields are missing from the SAME SERVICE. the fields parameters here omits ["CF_CURR", "CF_EXCHNG"]

'fields': [{'name': 'TR.PRICECLOSE.CURRENCY'}

3. OMM stream.

It then tries to get these fields from the OMM stream. It is indeed not able to find this fields in this stream. Off course as proved above these fields have nothing to do with a live streaming. For the avoidance of any doubt I decided to set up a simple OMM stream to see if I really did have a permission issue with OMM streaming.

import refinitiv.data as rd
from refinitiv.data.delivery import omm_stream
import datetime
import json

rd.open_session()

# Callback function to display data or status events
def display_event(eventType, event):
    currentTime = datetime.datetime.now().time()
    print("----------------------------------------------------------")
    print(">>> {} event received at {}".format(eventType, currentTime))
    print(json.dumps(event, indent=2))
    return

stream = omm_stream.Definition(
    name="EUR=",
    fields=['BID', 'ASK','OPEN_PRC','HST_CLOSE','TIMACT'],
    domain='MarketPrice').get_stream()
# Update - as and when field values change, we receive updates from the server
stream.on_update(lambda event, item_stream : display_event("Update", event))

stream.open()

I checked this and the stream ran perfectly fine. (see also log perfectOMMstream.txt)

perfectOMMstream.txt

4. Conclusion

This is as proven above definitely not a permission problem. This is a bug in rd.get_data() that it asks the wrong fields to the OMM service. Furthermore it is bad design of the rd.get_data() function as the old rd.eikon.get_data() function can do it in a single api call. The new function rd.get_data() requires an additional call to the OMM service and fails because it can not find this field in the service and blames it on some permission. It is however very likely that these static fields are not even available in this OMM service!

So for the third time I have extensively proven that this is a real and serious issue. PLEASE fix this!

laurens

please see my comment in the next answer below

andrii.sidachenko01

Dear, Laurens.

I see that you disappointed.

The design of rd.get_data() gives User ability request fundamental fields data from UDF or RDP platform and pricing fields data. Different endpoints provides these data, that's why we have two requests accordingly.

I have assumption, that the issue you have causes by permission lack. I ask you kindly double check with next code examples:

Regarding OMMstream.

import refinitiv.data as rd
from refinitiv.data.delivery import omm_stream
import datetime
import json

rd.open_session()


def display_event(eventType, event):
    currentTime = datetime.datetime.now().time()
    print("----------------------------------------------------------")
    print(">>> {} event received at {}".format(eventType, currentTime))
    print(json.dumps(event, indent=2))


stream = omm_stream.Definition(
    name="AAPL.O", fields=["CF_CURR", "CF_EXCHNG", "DSPLY_NAME"], domain="MarketPrice"
).get_stream()
stream.on_refresh(lambda event, item_stream: display_event("Refresh", event))

stream.open(with_updates=False)

as you can see we use the same Instrument with the same fields from initial message. Please provide the log file for this run.

Regarding rd.get_data() I ran the same code for RD library 1.5.0 from initial message.

import refinitiv.data as rd

rd.open_session()

fields = ["CF_CURR", "CF_EXCHNG", "TR.PriceClose.Currency", "TR.InstrumentType", "TR.ExchangeName", "DSPLY_NAME"]

df = rd.get_data(universe="AAPL.O", fields=fields)
print(df.to_string())

I attached the log file.

rd.get_data.1.5.0.log.zip

laurens

I am sorry but this is not a serious response.

1. Apple is not trading at this moment so I cannot check this
2. even if there is a permission error i don't have it in rd.eikon and I also should not have it rd.get_data() as well these fields should simply be requested to UDF as I proved extensively above!

olivier1.davant2

Dear Laurens,

First, I would like to explain our strategy for the Eikon library and Refinitiv Data Libraries:

We continue to support and maintain the Eikon Library
This library is "Feature Complete", but we continue to support it and provide maintenance releases.
The Refinitiv Data Library will be renamed the LSEG Data Library (LD Lib)
Due to LSEG's acquisition of Refinitiv, Refinitiv products and libraries are rebranded as LSEG. Rebranding a library means changing its name, which is a breaking change. This change will occur in version 2.0 of our data library.
We are taking advantage of the name change of our data library to slightly improve its public interfaces
Some aspects of the public APIs will be slightly improved in the LSEG Data Library 2.0. The “eikon” module which is a duplicate of the Eikon library will be removed.
We will continue to support and maintain the Refinitiv 1.x data library
After the rebranding and release of LSEG Data Library 2.0, the Refinitiv Data Library will become " Feature Complete", but we will continue to support it and provide maintenance releases. This is essential for customers who cannot migrate to the LSEG Data Library 2.0. However, new features will only be added to the LSEG data library.
The “eikon” module will not be removed from the Refinitiv 1.x data library
The FutureWarning you noticed was about the change to the LSEG Data Library 2.0. Maintenance releases of RD Lib will continue to include the “eikon” module. There are no plans to remove it.
We will ensure that there are no dependency conflicts between the Eikon library, the Refinitiv data library and the LSEG data library
This is essential for CodeBook and many of our customers. This check will be integrated into our automated tests.
The underlying data sources used by the LSEG Data Library's get_data() function will be fully aligned with those used by the Eikon Library's ek.get_data()
This will ensure a smooth migration from the Eikon library to the LSEG Data Library.

EXPLORE OUR SITES