question

Upvotes
Accepted
1 2 2 3

Odd handling of bad fields

In trying to get RD working, I ran through the examples posted on https://pypi.org/project/refinitiv-data/, within a platform session (desktop session not an option for my purposes). This example was taking 9 *minutes* to return a response, and only returned the revenue number:

df = rd.get_data(
        universe=['IBM.N', 'VOD.L'], 
        fields=['BID', 'ASK', 'TR.Revenue']
    )
    print(df)

This example, on the other hand, came back in ~9 seconds:

response = fundamental_and_reference.Definition(
			['IBM.N', 'VOD.L'],
			['TR.BidPrice', 'TR.AskPrice', 'TR.Revenue']
).get_data()
print(response.data.df)

After messing about with this a while longer, I discovered that (apparently) the issue is in the rd.get_data function's ability to handle bad fields names. In the first example, 'BID' and 'ASK' are not acceptable field names and get dropped from the internal request, and if you change them to TR.BidPrice and TR.AskPrice, rd.get_data returns a response in ~9 seconds (and actually gives you data). If I change the fields list to ['ARTICHOKE','TR.Revenue'], ARTICHOKE gets dropped from the internal request, but I'm back to ~9 minutes waiting for the response.

My reason for posting:

  1. Can anyone on the RD/RPD team replicate this response delay when BID and/or ASK fields are part of the request?
  2. If so, maybe change the example on the https://pypi.org/project/refinitiv-data/ page?
  3. Modify rd.get_data to TELL you when you're requesting a field it cannot provide? The only error I'm seeing in the logging is the ConnectTimeout('timed out') at close to the five-minute mark.
  4. Document that, if response is abnormally/absurdly slow, the presence of a bad field could be the issue.
#productrdp-apirdp
icon clock
10 |1500

Up to 2 attachments (including images) can be used with a maximum of 512.0 KiB each and 1.0 MiB total.

Upvote
Accepted
80.3k 259 52 75

@jeff.kenyon0

In addition to the response from my colleague, I would like to explain how the get_data method works.

df = rd.get_data(
        universe=['IBM.N', 'VOD.L'], 
        fields=['BID', 'ASK', 'TR.Revenue']
    )

With the request above, there are two types of fields (a TR field and real-time fields). The real-time fields don't have the TR prefixed.

To get data for the TR fields, the get_data method sends a request message to the DataGrid endpoint on RDP. The DataGrid endpoint on RDP doesn't support real-time fields (BID, ASK). Therefore, the get_data method sends an additional request to the Real-Time WebSocket streaming service to get data for the Real-Time fields.

1692160479522.png

Typically, most clients don't have permission to access real-time data on the Real-Time WebSocket streaming service. This is why the get_data method didn't display the BID and ASK values.

on_complete_stream_event ({'ID': 5, 'Type': 'Status', 'Key': {'Service': 'ELEKTRON_DD', 'Name': 'IBM.N'}, 'State': {'Stream': 'Closed', 'Data': 'Suspect', 'Code': 'NotEntitled', 'Text': 'Access Denied: User req to PE(62)'}},)

You can verify this behavior by enable logging in the RD library via the configuration file (refinitiv-data.config.json).

1692160619425.png



1692160479522.png (23.6 KiB)
1692160619425.png (62.0 KiB)
icon clock
10 |1500

Up to 2 attachments (including images) can be used with a maximum of 512.0 KiB each and 1.0 MiB total.

@Jirapongse, thank you for the added information. A few notes:

  1. Given that "Typically, most clients don't have permission to access real-time data on the Real-Time WebSocket streaming service," it seems an odd choice to include those fields in an example on the public-facing https://pypi.org/project/refinitiv-data/ page. Especially without mentioning the difference of how a "TR" field versus a "real-time" field are handled.
  2. I had logging turned on (to both console and file), and went back and looked at the log file. There is no on_complete_stream_event in the file. No "access denied" message. No error beyond a timeout. Full log available on request.

@jeff.kenyon0

Please share the log file and also check that the log file doesn't contain private information, such as your credentials.

Upvotes
14.5k 30 5 10

Hi @jeff.kenyon0 ,

Thank you for the feedback. I can replicate a delay response when BID and/or ASK fields are part of the request (but not as much delay as yours). Here's the result
1692159263999.png

Hence, I'm discussing this with the team and will keep you updated


1692159263999.png (53.4 KiB)
icon clock
10 |1500

Up to 2 attachments (including images) can be used with a maximum of 512.0 KiB each and 1.0 MiB total.

@raksina.samasiri, thank you for the response. Glad you noted the difference (which would be explained by the additional call to the real-time web socket that @Jirapongse pointed out).

Upvotes
1 2 2 3

The script:

import os
import ssl
import sys
import refinitiv.data as rd
from refinitiv.data.content import fundamental_and_reference

os.environ["RD_LIB_CONFIG_PATH"] = "D:/Python Utilities/"

ssl_context = ssl.create_default_context()
ssl_version=ssl.PROTOCOL_TLS

print('Python version:')
print(sys.version)
print('SSL version:')
print(ssl.OPENSSL_VERSION)
print('RD version:')
print(rd.__version__)

default_session = rd.open_session()

print('Basic get_data')
df = rd.get_data(
        universe=['IBM.N', 'VOD.L'], 
        fields=['BID', 'ASK', 'TR.Revenue']
    )
print(df)

default_session.close()


print('Done')

20230817-1459-25868-refinitiv-data-lib.txt : Log. Non-log output was:

  Instrument      Revenue
0      IBM.N  60530000000
1      VOD.L  45706000000

This (again) took ~ 9 minutes or so to run, so any insight on why would be appreciated (especially if there's something that I could do to improve on that, beyond just removing BID and ASK from the field list).



icon clock
10 |1500

Up to 2 attachments (including images) can be used with a maximum of 512.0 KiB each and 1.0 MiB total.

Upvotes
80.3k 259 52 75

@jeff.kenyon0

You are using the PPE environment.

[2023-08-17T15:04:30.661420+00:00] - [RetryTransportBase] - [DEBUG] - [31472 - AuthManager-Thread] - [_retry_transport] - [_handle_request] - Sending request to https://api.ppe.refinitiv.com/auth/oauth2/v1/token
[2023-08-17T15:04:44.852726+00:00] - [RetryTransportBase] - [DEBUG] - [1336 - OpenUniverseStreams-Thread_1] - [_retry_transport] - [_handle_request] - Sending request to https://api.ppe.refinitiv.com/streaming/pricing/v1/
[2023-08-17T15:05:11.666195+00:00] - [RetryTransportBase] - [DEBUG] - [31472 - AuthManager-Thread] - [_retry_transport] - [_handle_request] - Sending request to https://api.ppe.refinitiv.com/auth/oauth2/v1/token
[2023-08-17T15:05:26.883592+00:00] - [RetryTransportBase] - [DEBUG] - [1336 - OpenUniverseStreams-Thread_1] - [_retry_transport] - [_handle_request] - Sending request to https://api.ppe.refinitiv.com/streaming/pricing/v1/
[2023-08-17T15:05:53.672063+00:00] - [RetryTransportBase] - [DEBUG] - [31472 - AuthManager-Thread] - [_retry_transport] - [_handle_request] - Sending request to https://api.ppe.refinitiv.com/auth/oauth2/v1/token
[2023-08-17T15:06:10.890647+00:00] - [RetryTransportBase] - [DEBUG] - [1336 - OpenUniverseStreams-Thread_1] - [_retry_transport] - [_handle_request] - Sending request to https://api.ppe.refinitiv.com/streaming/pricing/v1/
[2023-08-17T15:06:37.678115+00:00] - [RetryTransportBase] - [DEBUG] - [31472 - AuthManager-Thread] - [_retry_transport] - [_handle_request] - Sending request to https://api.ppe.refinitiv.com/auth/oauth2/v1/token
[2023-08-17T15:06:58.899068+00:00] - [RetryTransportBase] - [DEBUG] - [1336 - OpenUniverseStreams-Thread_1] - [_retry_transport] - [_handle_request] - Sending request to https://api.ppe.refinitiv.com/streaming/pricing/v1/
[2023-08-17T15:07:25.684532+00:00] - [RetryTransportBase] - [DEBUG] - [31472 - AuthManager-Thread] - [_retry_transport] - [_handle_request] - Sending request to https://api.ppe.refinitiv.com/auth/oauth2/v1/token
[2023-08-17T15:07:54.905223+00:00] - [RetryTransportBase] - [DEBUG] - [1336 - OpenUniverseStreams-Thread_1] - [_retry_transport] - [_handle_request] - Sending request to https://api.ppe.refinitiv.com/streaming/pricing/v1/
[2023-08-17T15:08:21.690688+00:00] - [RetryTransportBase] - [DEBUG] - [31472 - AuthManager-Thread] - [_retry_transport] - [_handle_request] - Sending request to https://api.ppe.refinitiv.com/auth/oauth2/v1/token
[2023-08-17T15:08:34.909904+00:00] - [sessions.platform.rdp_PPE.0] - [ERROR] - [1336 - OpenUniverseStreams-Thread_1] - [http_service] - [request] - An error occurred while requesting URL('https://api.ppe.refinitiv.com/streaming/pricing/v1/').
    ConnectTimeout('timed out')

The log indicates that you can't connect to this endpoint: https://api.ppe.refinitiv.com/streaming/pricing/v1/.

icon clock
10 |1500

Up to 2 attachments (including images) can be used with a maximum of 512.0 KiB each and 1.0 MiB total.

Upvotes
1 2 2 3

Switching to PROD (base URL https://api.refinitiv.com/), I first tried an RT account. The time was roughly the same, and in this case, I didn't even get the values at the end (apparently the RT ID I used does not have access to historical values, and it errored out, presumably when going for BID and ASK).


I then tried another ID which has access to historical data. The time was roughly the same, as was the result.


Is there some other endpoint I should be using?


I understand not having access to RT fields (so would expect a quick rejection there), and I understand not having access to an endpoint (due to firewall issues or whatever). What I don't understand is waiting for ~10 minutes, then getting data. 20230822-1537-33032-refinitiv-data-lib.txt20230822-1602-23764-refinitiv-data-lib.txt


icon clock
10 |1500

Up to 2 attachments (including images) can be used with a maximum of 512.0 KiB each and 1.0 MiB total.

Upvotes
80.3k 259 52 75

@jeff.kenyon0

It is ConnectionTimeout (ConnectTimeout('timed out')).

[2023-08-22T16:12:04.169045+00:00] - [sessions.platform.rdp_PROD.0] - [ERROR] - [21860 - OpenUniverseStreams-Thread_1] - [http_service] - [request] - An error occurred while requesting URL('https://api.refinitiv.com/streaming/pricing/v1/').
    ConnectTimeout('timed out')

I think the API may wait for the response from the real-time service before returning.

You can remove the real-time fields (BID and ASK) from the request.

icon clock
10 |1500

Up to 2 attachments (including images) can be used with a maximum of 512.0 KiB each and 1.0 MiB total.

Upvotes
1 2 2 3

Well, yes, removing BID and ASK was the solution, as I mentioned in the original post. I understand now that the mix of RT and historical fields was the problem. My questions now would be (a) why the delay/timeout, and (b) would this be considered standard/expected behavior by the API, or is this indicative of some other issue (e.g., network or firewall misconfiguration)? A 9-10 minute wait for the API to respond, but getting a response, does not seem normal.

icon clock
10 |1500

Up to 2 attachments (including images) can be used with a maximum of 512.0 KiB each and 1.0 MiB total.

@jeff.kenyon0

It looks like to be a network or connection issue when connecting to https://api.refinitiv.com/streaming/pricing/v1/. You can check if a Python application can access this URL.

Write an Answer

Hint: Notify or tag a user in this post by typing @username.

Up to 2 attachments (including images) can be used with a maximum of 512.0 KiB each and 1.0 MiB total.