Odd handling of bad fields

In addition to the response from my colleague, I would like to explain how the get_data method works.

df = rd.get_data(
        universe=['IBM.N', 'VOD.L'], 
        fields=['BID', 'ASK', 'TR.Revenue']
    )

With the request above, there are two types of fields (a TR field and real-time fields). The real-time fields don't have the TR prefixed.

To get data for the TR fields, the get_data method sends a request message to the DataGrid endpoint on RDP. The DataGrid endpoint on RDP doesn't support real-time fields (BID, ASK). Therefore, the get_data method sends an additional request to the Real-Time WebSocket streaming service to get data for the Real-Time fields.

Typically, most clients don't have permission to access real-time data on the Real-Time WebSocket streaming service. This is why the get_data method didn't display the BID and ASK values.

on_complete_stream_event ({'ID': 5, 'Type': 'Status', 'Key': {'Service': 'ELEKTRON_DD', 'Name': 'IBM.N'}, 'State': {'Stream': 'Closed', 'Data': 'Suspect', 'Code': 'NotEntitled', 'Text': 'Access Denied: User req to PE(62)'}},)

You can verify this behavior by enable logging in the RD library via the configuration file (refinitiv-data.config.json).

All comments

raksina.samasiri

Hi @jeff.kenyon0 ,

Thank you for the feedback. I can replicate a delay response when BID and/or ASK fields are part of the request (but not as much delay as yours). Here's the result

Hence, I'm discussing this with the team and will keep you updated

In addition to the response from my colleague, I would like to explain how the get_data method works.

df = rd.get_data(
        universe=['IBM.N', 'VOD.L'], 
        fields=['BID', 'ASK', 'TR.Revenue']
    )

With the request above, there are two types of fields (a TR field and real-time fields). The real-time fields don't have the TR prefixed.

Typically, most clients don't have permission to access real-time data on the Real-Time WebSocket streaming service. This is why the get_data method didn't display the BID and ASK values.

on_complete_stream_event ({'ID': 5, 'Type': 'Status', 'Key': {'Service': 'ELEKTRON_DD', 'Name': 'IBM.N'}, 'State': {'Stream': 'Closed', 'Data': 'Suspect', 'Code': 'NotEntitled', 'Text': 'Access Denied: User req to PE(62)'}},)

You can verify this behavior by enable logging in the RD library via the configuration file (refinitiv-data.config.json).

@raksina.samasiri, thank you for the response. Glad you noted the difference (which would be explained by the additional call to the real-time web socket that @Jirapongse pointed out).

@Jirapongse, thank you for the added information. A few notes:

Given that "Typically, most clients don't have permission to access real-time data on the Real-Time WebSocket streaming service," it seems an odd choice to include those fields in an example on the public-facing https://pypi.org/project/refinitiv-data/ page. Especially without mentioning the difference of how a "TR" field versus a "real-time" field are handled.
I had logging turned on (to both console and file), and went back and looked at the log file. There is no on_complete_stream_event in the file. No "access denied" message. No error beyond a timeout. Full log available on request.

Please share the log file and also check that the log file doesn't contain private information, such as your credentials.

The script:

import os
import ssl
import sys
import refinitiv.data as rd
from refinitiv.data.content import fundamental_and_reference

os.environ["RD_LIB_CONFIG_PATH"] = "D:/Python Utilities/"

ssl_context = ssl.create_default_context()
ssl_version=ssl.PROTOCOL_TLS

print('Python version:')
print(sys.version)
print('SSL version:')
print(ssl.OPENSSL_VERSION)
print('RD version:')
print(rd.__version__)

default_session = rd.open_session()

print('Basic get_data')
df = rd.get_data(
        universe=['IBM.N', 'VOD.L'], 
        fields=['BID', 'ASK', 'TR.Revenue']
    )
print(df)

default_session.close()


print('Done')

20230817-1459-25868-refinitiv-data-lib.txt : Log. Non-log output was:

  Instrument      Revenue
0      IBM.N  60530000000
1      VOD.L  45706000000

This (again) took ~ 9 minutes or so to run, so any insight on why would be appreciated (especially if there's something that I could do to improve on that, beyond just removing BID and ASK from the field list).

You are using the PPE environment.

[2023-08-17T15:04:30.661420+00:00] - [RetryTransportBase] - [DEBUG] - [31472 - AuthManager-Thread] - [_retry_transport] - [_handle_request] - Sending request to https://api.ppe.refinitiv.com/auth/oauth2/v1/token
[2023-08-17T15:04:44.852726+00:00] - [RetryTransportBase] - [DEBUG] - [1336 - OpenUniverseStreams-Thread_1] - [_retry_transport] - [_handle_request] - Sending request to https://api.ppe.refinitiv.com/streaming/pricing/v1/
[2023-08-17T15:05:11.666195+00:00] - [RetryTransportBase] - [DEBUG] - [31472 - AuthManager-Thread] - [_retry_transport] - [_handle_request] - Sending request to https://api.ppe.refinitiv.com/auth/oauth2/v1/token
[2023-08-17T15:05:26.883592+00:00] - [RetryTransportBase] - [DEBUG] - [1336 - OpenUniverseStreams-Thread_1] - [_retry_transport] - [_handle_request] - Sending request to https://api.ppe.refinitiv.com/streaming/pricing/v1/
[2023-08-17T15:05:53.672063+00:00] - [RetryTransportBase] - [DEBUG] - [31472 - AuthManager-Thread] - [_retry_transport] - [_handle_request] - Sending request to https://api.ppe.refinitiv.com/auth/oauth2/v1/token
[2023-08-17T15:06:10.890647+00:00] - [RetryTransportBase] - [DEBUG] - [1336 - OpenUniverseStreams-Thread_1] - [_retry_transport] - [_handle_request] - Sending request to https://api.ppe.refinitiv.com/streaming/pricing/v1/
[2023-08-17T15:06:37.678115+00:00] - [RetryTransportBase] - [DEBUG] - [31472 - AuthManager-Thread] - [_retry_transport] - [_handle_request] - Sending request to https://api.ppe.refinitiv.com/auth/oauth2/v1/token
[2023-08-17T15:06:58.899068+00:00] - [RetryTransportBase] - [DEBUG] - [1336 - OpenUniverseStreams-Thread_1] - [_retry_transport] - [_handle_request] - Sending request to https://api.ppe.refinitiv.com/streaming/pricing/v1/
[2023-08-17T15:07:25.684532+00:00] - [RetryTransportBase] - [DEBUG] - [31472 - AuthManager-Thread] - [_retry_transport] - [_handle_request] - Sending request to https://api.ppe.refinitiv.com/auth/oauth2/v1/token
[2023-08-17T15:07:54.905223+00:00] - [RetryTransportBase] - [DEBUG] - [1336 - OpenUniverseStreams-Thread_1] - [_retry_transport] - [_handle_request] - Sending request to https://api.ppe.refinitiv.com/streaming/pricing/v1/
[2023-08-17T15:08:21.690688+00:00] - [RetryTransportBase] - [DEBUG] - [31472 - AuthManager-Thread] - [_retry_transport] - [_handle_request] - Sending request to https://api.ppe.refinitiv.com/auth/oauth2/v1/token
[2023-08-17T15:08:34.909904+00:00] - [sessions.platform.rdp_PPE.0] - [ERROR] - [1336 - OpenUniverseStreams-Thread_1] - [http_service] - [request] - An error occurred while requesting URL('https://api.ppe.refinitiv.com/streaming/pricing/v1/').
    ConnectTimeout('timed out')

The log indicates that you can't connect to this endpoint: https://api.ppe.refinitiv.com/streaming/pricing/v1/.

Switching to PROD (base URL https://api.refinitiv.com/), I first tried an RT account. The time was roughly the same, and in this case, I didn't even get the values at the end (apparently the RT ID I used does not have access to historical values, and it errored out, presumably when going for BID and ASK).

I then tried another ID which has access to historical data. The time was roughly the same, as was the result.

Is there some other endpoint I should be using?

I understand not having access to RT fields (so would expect a quick rejection there), and I understand not having access to an endpoint (due to firewall issues or whatever). What I don't understand is waiting for ~10 minutes, then getting data. 20230822-1537-33032-refinitiv-data-lib.txt 20230822-1602-23764-refinitiv-data-lib.txt

It is ConnectionTimeout (ConnectTimeout('timed out')).

[2023-08-22T16:12:04.169045+00:00] - [sessions.platform.rdp_PROD.0] - [ERROR] - [21860 - OpenUniverseStreams-Thread_1] - [http_service] - [request] - An error occurred while requesting URL('https://api.refinitiv.com/streaming/pricing/v1/').
    ConnectTimeout('timed out')

I think the API may wait for the response from the real-time service before returning.

You can remove the real-time fields (BID and ASK) from the request.

Well, yes, removing BID and ASK was the solution, as I mentioned in the original post. I understand now that the mix of RT and historical fields was the problem. My questions now would be (a) why the delay/timeout, and (b) would this be considered standard/expected behavior by the API, or is this indicative of some other issue (e.g., network or firewall misconfiguration)? A 9-10 minute wait for the API to respond, but getting a response, does not seem normal.