Odd handling of bad fields

Question

question

jeff.kenyon0

1 ●2 ●2 ●3

Odd handling of bad fields

In trying to get RD working, I ran through the examples posted on https://pypi.org/project/refinitiv-data/, within a platform session (desktop session not an option for my purposes). This example was taking 9 *minutes* to return a response, and only returned the revenue number:

df = rd.get_data(
        universe=['IBM.N', 'VOD.L'], 
        fields=['BID', 'ASK', 'TR.Revenue']
    )
    print(df)

This example, on the other hand, came back in ~9 seconds:

response = fundamental_and_reference.Definition(
			['IBM.N', 'VOD.L'],
			['TR.BidPrice', 'TR.AskPrice', 'TR.Revenue']
).get_data()
print(response.data.df)

After messing about with this a while longer, I discovered that (apparently) the issue is in the rd.get_data function's ability to handle bad fields names. In the first example, 'BID' and 'ASK' are not acceptable field names and get dropped from the internal request, and if you change them to TR.BidPrice and TR.AskPrice, rd.get_data returns a response in ~9 seconds (and actually gives you data). If I change the fields list to ['ARTICHOKE','TR.Revenue'], ARTICHOKE gets dropped from the internal request, but I'm back to ~9 minutes waiting for the response.

My reason for posting:

Can anyone on the RD/RPD team replicate this response delay when BID and/or ASK fields are part of the request?
If so, maybe change the example on the https://pypi.org/project/refinitiv-data/ page?
Modify rd.get_data to TELL you when you're requesting a field it cannot provide? The only error I'm seeing in the logging is the ConnectTimeout('timed out') at close to the five-minute mark.
Document that, if response is abnormally/absurdly slow, the presence of a bad field could be the issue.

rdp-api #product rdp

Aug 15, 2023 at 05:48 PM

10 |1500

Attachments: Up to 2 attachments (including images) can be used with a maximum of 512.0 KiB each and 1.0 MiB total.

raksina.samasiri Aug 16, 2023 at 04:25 AM

<AHS>

there're no error returned from datagrid as well, contacting RDP team

1692159886657.png (39.3 KiB)

raksina.samasiri raksina.samasiri Aug 16, 2023 at 04:27 AM

<AHS>

raw response (I cannot post the code snippet of this response so I put the screenshot instead)

1692160043859.png (43.9 KiB)

raksina.samasiri Sep 08, 2023 at 06:05 AM

sent an email to APP-SRE-Datacloud <APP-SRE-Datacloud@lseg.com> to follow up on this issue. (Subject: datagrid - Odd handling of bad fields - from customer)

Jirapongse Sep 21, 2023 at 04:57 AM

@jeff.kenyon0

Hi,

Thank you for your participation in the forum.

Are any of the replies below satisfactory in resolving your query?

If yes please click the 'Accept' text next to the most appropriate reply. This will guide all community members who have a similar question.

Otherwise please post again offering further insight into your question.

Thanks,

AHS

wasin.w Sep 28, 2023 at 07:29 AM

Please be informed that a reply has been verified as correct in answering the question, and has been marked as such.

Thanks,

AHS

Answer 1 · 2023-08-16T04:40:23Z

Jirapongse

79.2k ●251 ●52 ●74

@jeff.kenyon0

In addition to the response from my colleague, I would like to explain how the get_data method works.

df = rd.get_data(
        universe=['IBM.N', 'VOD.L'], 
        fields=['BID', 'ASK', 'TR.Revenue']
    )

With the request above, there are two types of fields (a TR field and real-time fields). The real-time fields don't have the TR prefixed.

To get data for the TR fields, the get_data method sends a request message to the DataGrid endpoint on RDP. The DataGrid endpoint on RDP doesn't support real-time fields (BID, ASK). Therefore, the get_data method sends an additional request to the Real-Time WebSocket streaming service to get data for the Real-Time fields.

Typically, most clients don't have permission to access real-time data on the Real-Time WebSocket streaming service. This is why the get_data method didn't display the BID and ASK values.

on_complete_stream_event ({'ID': 5, 'Type': 'Status', 'Key': {'Service': 'ELEKTRON_DD', 'Name': 'IBM.N'}, 'State': {'Stream': 'Closed', 'Data': 'Suspect', 'Code': 'NotEntitled', 'Text': 'Access Denied: User req to PE(62)'}},)

You can verify this behavior by enable logging in the RD library via the configuration file (refinitiv-data.config.json).

1692160479522.png (23.6 KiB)

1692160619425.png (62.0 KiB)

Aug 16, 2023 at 04:40 AM

10 |1500

Attachments: Up to 2 attachments (including images) can be used with a maximum of 512.0 KiB each and 1.0 MiB total.

jeff.kenyon0 Aug 16, 2023 at 02:25 PM

@Jirapongse, thank you for the added information. A few notes:

Given that "Typically, most clients don't have permission to access real-time data on the Real-Time WebSocket streaming service," it seems an odd choice to include those fields in an example on the public-facing https://pypi.org/project/refinitiv-data/ page. Especially without mentioning the difference of how a "TR" field versus a "real-time" field are handled.
I had logging turned on (to both console and file), and went back and looked at the log file. There is no on_complete_stream_event in the file. No "access denied" message. No error beyond a timeout. Full log available on request.

Jirapongse jeff.kenyon0 Aug 17, 2023 at 07:20 AM

@jeff.kenyon0

Please share the log file and also check that the log file doesn't contain private information, such as your credentials.

Answer 2 · 2023-08-16T04:15:39Z

raksina.samasiri

14.2k ●30 ●5 ●10

Hi @jeff.kenyon0 ,

Thank you for the feedback. I can replicate a delay response when BID and/or ASK fields are part of the request (but not as much delay as yours). Here's the result

Hence, I'm discussing this with the team and will keep you updated

1692159263999.png (53.4 KiB)

Aug 16, 2023 at 04:15 AM

10 |1500

Attachments: Up to 2 attachments (including images) can be used with a maximum of 512.0 KiB each and 1.0 MiB total.

jeff.kenyon0 Aug 16, 2023 at 02:19 PM

@raksina.samasiri, thank you for the response. Glad you noted the difference (which would be explained by the additional call to the real-time web socket that @Jirapongse pointed out).

Answer 3 · 2023-08-17T16:05:28Z

jeff.kenyon0

1 ●2 ●2 ●3

The script:

import os
import ssl
import sys
import refinitiv.data as rd
from refinitiv.data.content import fundamental_and_reference

os.environ["RD_LIB_CONFIG_PATH"] = "D:/Python Utilities/"

ssl_context = ssl.create_default_context()
ssl_version=ssl.PROTOCOL_TLS

print('Python version:')
print(sys.version)
print('SSL version:')
print(ssl.OPENSSL_VERSION)
print('RD version:')
print(rd.__version__)

default_session = rd.open_session()

print('Basic get_data')
df = rd.get_data(
        universe=['IBM.N', 'VOD.L'], 
        fields=['BID', 'ASK', 'TR.Revenue']
    )
print(df)

default_session.close()


print('Done')

20230817-1459-25868-refinitiv-data-lib.txt : Log. Non-log output was:

  Instrument      Revenue
0      IBM.N  60530000000
1      VOD.L  45706000000

This (again) took ~ 9 minutes or so to run, so any insight on why would be appreciated (especially if there's something that I could do to improve on that, beyond just removing BID and ASK from the field list).

20230817-1459-25868-refinitiv-data-lib.txt (25.9 KiB)

Aug 17, 2023 at 04:05 PM

10 |1500

Attachments: Up to 2 attachments (including images) can be used with a maximum of 512.0 KiB each and 1.0 MiB total.

Answer 4 · 2023-08-18T01:03:11Z

Jirapongse

79.2k ●251 ●52 ●74

@jeff.kenyon0

You are using the PPE environment.

[2023-08-17T15:04:30.661420+00:00] - [RetryTransportBase] - [DEBUG] - [31472 - AuthManager-Thread] - [_retry_transport] - [_handle_request] - Sending request to https://api.ppe.refinitiv.com/auth/oauth2/v1/token
[2023-08-17T15:04:44.852726+00:00] - [RetryTransportBase] - [DEBUG] - [1336 - OpenUniverseStreams-Thread_1] - [_retry_transport] - [_handle_request] - Sending request to https://api.ppe.refinitiv.com/streaming/pricing/v1/
[2023-08-17T15:05:11.666195+00:00] - [RetryTransportBase] - [DEBUG] - [31472 - AuthManager-Thread] - [_retry_transport] - [_handle_request] - Sending request to https://api.ppe.refinitiv.com/auth/oauth2/v1/token
[2023-08-17T15:05:26.883592+00:00] - [RetryTransportBase] - [DEBUG] - [1336 - OpenUniverseStreams-Thread_1] - [_retry_transport] - [_handle_request] - Sending request to https://api.ppe.refinitiv.com/streaming/pricing/v1/
[2023-08-17T15:05:53.672063+00:00] - [RetryTransportBase] - [DEBUG] - [31472 - AuthManager-Thread] - [_retry_transport] - [_handle_request] - Sending request to https://api.ppe.refinitiv.com/auth/oauth2/v1/token
[2023-08-17T15:06:10.890647+00:00] - [RetryTransportBase] - [DEBUG] - [1336 - OpenUniverseStreams-Thread_1] - [_retry_transport] - [_handle_request] - Sending request to https://api.ppe.refinitiv.com/streaming/pricing/v1/
[2023-08-17T15:06:37.678115+00:00] - [RetryTransportBase] - [DEBUG] - [31472 - AuthManager-Thread] - [_retry_transport] - [_handle_request] - Sending request to https://api.ppe.refinitiv.com/auth/oauth2/v1/token
[2023-08-17T15:06:58.899068+00:00] - [RetryTransportBase] - [DEBUG] - [1336 - OpenUniverseStreams-Thread_1] - [_retry_transport] - [_handle_request] - Sending request to https://api.ppe.refinitiv.com/streaming/pricing/v1/
[2023-08-17T15:07:25.684532+00:00] - [RetryTransportBase] - [DEBUG] - [31472 - AuthManager-Thread] - [_retry_transport] - [_handle_request] - Sending request to https://api.ppe.refinitiv.com/auth/oauth2/v1/token
[2023-08-17T15:07:54.905223+00:00] - [RetryTransportBase] - [DEBUG] - [1336 - OpenUniverseStreams-Thread_1] - [_retry_transport] - [_handle_request] - Sending request to https://api.ppe.refinitiv.com/streaming/pricing/v1/
[2023-08-17T15:08:21.690688+00:00] - [RetryTransportBase] - [DEBUG] - [31472 - AuthManager-Thread] - [_retry_transport] - [_handle_request] - Sending request to https://api.ppe.refinitiv.com/auth/oauth2/v1/token
[2023-08-17T15:08:34.909904+00:00] - [sessions.platform.rdp_PPE.0] - [ERROR] - [1336 - OpenUniverseStreams-Thread_1] - [http_service] - [request] - An error occurred while requesting URL('https://api.ppe.refinitiv.com/streaming/pricing/v1/').
    ConnectTimeout('timed out')

The log indicates that you can't connect to this endpoint: https://api.ppe.refinitiv.com/streaming/pricing/v1/.

Aug 18, 2023 at 01:03 AM

10 |1500

Attachments: Up to 2 attachments (including images) can be used with a maximum of 512.0 KiB each and 1.0 MiB total.

Answer 5 · 2023-08-22T16:21:58Z

jeff.kenyon0

1 ●2 ●2 ●3

Switching to PROD (base URL https://api.refinitiv.com/), I first tried an RT account. The time was roughly the same, and in this case, I didn't even get the values at the end (apparently the RT ID I used does not have access to historical values, and it errored out, presumably when going for BID and ASK).

I then tried another ID which has access to historical data. The time was roughly the same, as was the result.

Is there some other endpoint I should be using?

I understand not having access to RT fields (so would expect a quick rejection there), and I understand not having access to an endpoint (due to firewall issues or whatever). What I don't understand is waiting for ~10 minutes, then getting data. 20230822-1537-33032-refinitiv-data-lib.txt 20230822-1602-23764-refinitiv-data-lib.txt

20230822-1537-33032-refinitiv-data-lib.txt (56.4 KiB)

20230822-1602-23764-refinitiv-data-lib.txt (57.6 KiB)

Aug 22, 2023 at 04:21 PM

10 |1500

Attachments: Up to 2 attachments (including images) can be used with a maximum of 512.0 KiB each and 1.0 MiB total.

Answer 6 · 2023-08-23T01:37:43Z

Jirapongse

79.2k ●251 ●52 ●74

@jeff.kenyon0

It is ConnectionTimeout (ConnectTimeout('timed out')).

[2023-08-22T16:12:04.169045+00:00] - [sessions.platform.rdp_PROD.0] - [ERROR] - [21860 - OpenUniverseStreams-Thread_1] - [http_service] - [request] - An error occurred while requesting URL('https://api.refinitiv.com/streaming/pricing/v1/').
    ConnectTimeout('timed out')

I think the API may wait for the response from the real-time service before returning.

You can remove the real-time fields (BID and ASK) from the request.

Aug 23, 2023 at 01:37 AM

10 |1500

Attachments: Up to 2 attachments (including images) can be used with a maximum of 512.0 KiB each and 1.0 MiB total.

Answer 7 · 2023-08-23T10:50:43Z

jeff.kenyon0

1 ●2 ●2 ●3

Well, yes, removing BID and ASK was the solution, as I mentioned in the original post. I understand now that the mix of RT and historical fields was the problem. My questions now would be (a) why the delay/timeout, and (b) would this be considered standard/expected behavior by the API, or is this indicative of some other issue (e.g., network or firewall misconfiguration)? A 9-10 minute wait for the API to respond, but getting a response, does not seem normal.

Aug 23, 2023 at 10:50 AM

10 |1500

Attachments: Up to 2 attachments (including images) can be used with a maximum of 512.0 KiB each and 1.0 MiB total.

Jirapongse Aug 30, 2023 at 06:56 AM

@jeff.kenyon0

It looks like to be a network or connection issue when connecting to https://api.refinitiv.com/streaming/pricing/v1/. You can check if a Python application can access this URL.

Q&A Forum

question

Odd handling of bad fields

7 Answers

Write an Answer

question

Odd handling of bad fields

7 Answers

Write an Answer

Related Questions