question

Upvotes
Accepted
5 0 0 1

Stale data from real-time RD retrieval

Trying to update Python app on Windows to switch from RDP to RD library API. Using RD library version 1.1.0 and the following code to make real-time snaphot retrieval:

import refinitiv.data as rd

session = rd.session.platform.Definition('rd_PPE').get_session()

rd.session.set_default(session)

rd.open_session()

response = rd.content.pricing.Definition(

['SOFR1MAVG=', 'SOFR3MAVG=', 'SOFR6MAVG=', 'USDSOFR=', '.SR1M', '.SR3M', '.SR6M', '.SR1Y'],

['VALUE_DT1', 'PRIMACT_1']

).get_data()


With RDP library, the value retrieved at 12:08 UTC for 'USDSOFR=' RIC is updated already for current day. However, after switch to RD library code above, the value is still yesterday's data until minutes later. Today I was not able to retrieve the current day data until 12:22 UTC. I can see using DataView that the current day data is already available at 12:08.

Please advise if anything can be done about the lag in the current data being available with RD get_data() call, or if I am doing something wrong.

python#technologyrefinitiv-realtime-optimisedrefinitiv-data-libraries
icon clock
10 |1500

Up to 2 attachments (including images) can be used with a maximum of 512.0 KiB each and 1.0 MiB total.

Upvotes
Accepted
18.3k 84 39 63

Hi @Belinda

It might be easier if we're using the same code. It appears you may be running in VS Code. So that is where I'm doing all my testing. For example, try this entire example:

import refinitiv.data as rd
import os

print(f'lib version: {rd.__version__}')

os.environ["RD_LIB_CONFIG_PATH"] = "C:/USDLIBORCashFallback/"
rd.open_session()

stream = rd.open_pricing_stream(
    universe = ['USDSOFR='],
    fields = ['VALUE_DT1', 'PRIMACT_1']
)

prices = stream.get_snapshot()
print(prices)

rd.close_session()

I've created the same root directory as you and in there my config file looks like this:

{
  "logs": {
    "level": "debug",
    "transports": {
      "console": {
        "enabled": false
      },
      "file": {
        "enabled": true,
        "name": "/tmp/refinitiv-data-lib.log"
      }
    }
  },
  "sessions": {
    "default": "platform.prod",
    "platform": {
        "prod": {
            "app-key": "<your app key>",
            "username": "<machine id>",
            "password": "<password>",
            "signon_control": true
        }
    }   
  }
}

I've enabled the logs within the config file instead. When I run this in VS Code:

1715299481873.png



1715299481873.png (46.7 KiB)
icon clock
10 |1500

Up to 2 attachments (including images) can be used with a maximum of 512.0 KiB each and 1.0 MiB total.

Upvotes
18.3k 84 39 63

Hi @Belinda

I performed the following test at approximately 21:40 UTC using version 1.6.0 of the Data Library for Python.

1715031488952.png

In the above, I don't see today's data (2024-05-06) but rather Friday's data (2024-05-03). If I pull the information using the SpeedGuide, I see the same values as above but do notice the TRADE_DATE field containing today's date.

1715031816372.png

Perhaps you can provide some screenshots to show the data you are retrieving. Also, I would suggest you update to the latest version of the Data Library for Python.


1715031488952.png (13.2 KiB)
1715031816372.png (67.1 KiB)
icon clock
10 |1500

Up to 2 attachments (including images) can be used with a maximum of 512.0 KiB each and 1.0 MiB total.

Upvotes
5 0 0 1

Hi @nick.zincone, thanks for response. Sorry I didn't describe the problem correctly... my app is looking for previous business date in the VALUE_DT1 field. Today it was getting back 2024-05-02 from get_data() call up until almost 12:22, although VALUE_DT1 had been updated to 2024-05-03 by 12:08.
I can't update immediately to latest version of RD library because it has breaking changes for other apps on my servers, I will need to find and make the modifications for all the apps at the same time.

icon clock
10 |1500

Up to 2 attachments (including images) can be used with a maximum of 512.0 KiB each and 1.0 MiB total.

Hi @Belinda

The library should be utilizing the RDP Pricing Snapshot endpoint when making the call to rd.content.pricing.Definition().get_data(). And to confirm, when you are using the same call within both the rdp and rd libraries, you are getting different results. And when you use DataView it is consistent with the rdp libraries?

If you can confirm the behavior when using the RDP Pricing Snapshot endpoint, that will help the development team confirm the odd behavior. In addition, what would be helpful is if you can try the following code as well (assuming you have access to streaming content):

1715120876718.png

The snapshot call (get_data()) you are using should be getting values directly from the streaming services. So, if it is possible to try the above to confirm the values you are getting back from the streaming services that will help narrow down what is going on.

1715120876718.png (17.2 KiB)
Upvotes
5 0 0 1

Hi @nick.zincone,

For some reason I am not able to succeed with code like above with get_stream() call, although that is much like the code we use with RDP realtime data request. Here is what the RDP code we use looks like, that is getting updated data for USDSOFR=/VALUE_DT1 updates in a timely manner:

1715279055784.png




1715279055784.png (31.5 KiB)
icon clock
10 |1500

Up to 2 attachments (including images) can be used with a maximum of 512.0 KiB each and 1.0 MiB total.

When I try similar code with get_stream() call and RD lib, I don't get any error back but the code just hangs after the "Opening Stream" log message below, it never gets to the "Getting snapshot" log message, so it seems to be hung in the stream.open() call. Here is what that code looks like:

1715281118527.png

1715281118527.png (17.4 KiB)
Upvotes
5 0 0 1

Since I can't seem to use RD get_stream() call, I tried get_data() instead which succeeds but the data is updated late.

icon clock
10 |1500

Up to 2 attachments (including images) can be used with a maximum of 512.0 KiB each and 1.0 MiB total.

Upvotes
18.3k 84 39 63

Hi @Belinda

Can you capture some logs within the rd library by doing the following before you open a session:

rd.get_config()["logs.transports.file.enabled"] = True
rd.get_config()["logs.transports.file.name"] = "data-lib.log"

This should create a log file that looks similar to this: 20240509-1922-776-data-lib.log (that is what was created on my machine).

thanks.

icon clock
10 |1500

Up to 2 attachments (including images) can be used with a maximum of 512.0 KiB each and 1.0 MiB total.

Upvotes
5 0 0 1

Hi @nick.zincone,

I don't seem to be able to find the log, I've tried adding the config directory to see if it would be put there but it doesn't help. It might not be saving the log because the app doesn't ever get to finish and I have to kill the window that's running it, or maybe I haven't added it right. I've tried above and below my code on lines 22-27:
1715285438587.pngPlease let me know if I am not working in the log capturing right with the way I'm setting up the session.


1715285438587.png (31.6 KiB)
icon clock
10 |1500

Up to 2 attachments (including images) can be used with a maximum of 512.0 KiB each and 1.0 MiB total.

Upvotes
5 0 0 1

Hi @nick.zincone, from log collected this morning it appears the date returned in VALUE_DT1 for USDSOFR= from get_data() call is updated as expected, so I need to look into why my app is not registering the update correctly.
I'm still a little concerned that I can't use the streaming snapshot if that is preferred by RD dev for snapshot data, so would appreciate if you can give any advice from looking at log included above.

icon clock
10 |1500

Up to 2 attachments (including images) can be used with a maximum of 512.0 KiB each and 1.0 MiB total.

Hi @Belinda

Based on the logs, you are unable to retrieve the streaming endpoints when calling:

https://api.refinitiv.com/streaming/pricing/v1/

The logs are showing the request is timing out. I don't know if you have any firewall issues or why that request isn't working. You can try a couple of things:

  1. Check to see if you can access the endpoint via the Playground

  2. Download the SpeedGuide utility and login using your credentials



Write an Answer

Hint: Notify or tag a user in this post by typing @username.

Up to 2 attachments (including images) can be used with a maximum of 512.0 KiB each and 1.0 MiB total.