Retrieval of Snapshot data with RDP for Python

Hi @wasin.w

There were several issues here

Corporate firewall/network issues - which were addressed internally by client
wrong credential values being used e.g. My.Refinitiv login details instead of MachineID
MachineID was not permissioned for RDP Snapshot API - only RTO - so the client was advised to use streaming interface with the Snapshot option

All comments

your code as far as I can read it looks fine. I tried the following and it worked OK

import refinitiv.dataplatform as rdp

print(rdp.__version__)

rdp.open_platform_session(
    APP_KEY, 
    rdp.GrantPassword(
        username = RDP_LOGIN, 
        password = RDP_PASSWORD
    )
)
universe = ['EUR='] 
fields = ['ASK', 'BID']
response = rdp.Pricing.get_snapshot(universe, fields)
print(response.data.df)
rdp.close_session()

can you confirm which version of the RDP library you are using?

I am using v 1.0.0a8.post1 from refinitiv-dataplatform · PyPI

If you are also using the same version, then please enable logging and attach the output - after removing any credentials etc)

import logging
logging.basicConfig(format='%(asctime)s - %(name)s - %(levelname)s - %(message)s', level=logging.DEBUG)

Also, if you are behind a corporate firewall/proxy etc - can you please try and run the code from your personal PC?

One further test, from your corporate PC, are you able to navigate to API Playground (refinitiv.com) in a browser? If so, can you then select the /data/pricing/beta3/snapshots test area and execute one of the GET requests:

refinitiv-dataplatform.20210525.13-41-47.txt

Hi Umer,

Thanks for your quick reply. I ran your code - with logging enabled. I still see the same error.

The version I have installed is 1.0.0a7.post7.

See output from the log created attached.

I tried connecting to the API Playground. This is not working as well:

I may be able to setup a test from my private pc tomorrow. It is not so easy to avoid the corp network.

I have discussed your issue with the RDP Lib Dev team and they have requested additional logging information,

Can you please add the following extra two lines to the start of your script and capture the output again, please

import logging
logging.basicConfig(format='%(asctime)s - %(name)s - %(levelname)s - %(message)s', level=logging.DEBUG)

import os
os.environ["HTTPX_LOG_LEVEL"] = "TRACE"

import refinitiv.dataplatform as rdp

#....
#open platform session etc...
#....

Please remove all 'authorization', b'Bearer sections from any output before uploading (looks like a very long line of alphanumeric output).

Thanks.

refinitiv-dataplatform.20210526.08-44-51.log.txt

Hi Umer,

I tried with the addtional login, but I don't see any additional output. When I run this code I get the following response - see attached log file created.

import os
import rf_settings as settings
import refinitiv.dataplatform as rdp
import logging
logging.basicConfig(
    format='%(asctime)s - %(name)s - %(levelname)s - %(message)s', level=logging.DEBUG)
os.environ["HTTPX_LOG_LEVEL"] = "TRACE"
print(rdp.__version__)
rdp.open_platform_session(
    settings.APP_KEY,
    rdp.GrantPassword(
        username=settings.RDP_LOGIN,
        password=settings.RDP_PASSWORD
    )
)
universe = ['EUR=']
fields = ['ASK', 'BID']
response = rdp.Pricing.get_snapshot(universe, fields)
print(response.data.df)
rdp.close_session()

1.0.0a7.post7

2021-05-26 08:44:51,602 - Session session.platform - Thread 1780 | MainThread
        server-mode : False
2021-05-26 08:44:51,602 - session.platform - DEBUG -         server-mode : False
2021-05-26 08:44:51,605 - Session session.platform - Thread 1780 | MainThread
AuthenticationTokenHandlerThread.authorize()
2021-05-26 08:44:51,605 - session.platform - DEBUG - AuthenticationTokenHandlerThread.authorize()
2021-05-26 08:44:51,606 - Session session.platform - Thread 12992 | AuthenticationTokenHandlerThread
STARTING :: AuthenticationTokenHandlerThread.run()
2021-05-26 08:44:51,606 - session.platform - DEBUG - STARTING :: AuthenticationTokenHandlerThread.run()
2021-05-26 08:44:51,607 - asyncio - DEBUG - Using proactor: IocpProactor
2021-05-26 08:44:51,608 - Session session.platform - Thread 12992 | AuthenticationTokenHandlerThread
Send a token request to https://api.refinitiv.com/auth/oauth2/v1/token
2021-05-26 08:44:51,608 - session.platform - DEBUG - Send a token request to https://api.refinitiv.com/auth/oauth2/v1/token
2021-05-26 08:44:51,608 - Session session.platform - Thread 12992 | AuthenticationTokenHandlerThread
{ 'grant_type' : 'password',
'client_id' : ****************6186,
'username' : vfm.im@devolksbank.nl,
'password' : *****************************************************************,
'scope' : trapi,
'takeExclusiveSignOnControl' : True }
2021-05-26 08:44:51,608 - session.platform - DEBUG - { 'grant_type' : 'password',
'client_id' : ****************6186,
'username' : vfm.im@devolksbank.nl,
'password' : *****************************************************************,
'scope' : trapi,
'takeExclusiveSignOnControl' : True }
2021-05-26 08:44:56,606 - Session session.platform - Thread 1780 | MainThread
RDP connection state = State.Open.
2021-05-26 08:44:56,606 - session.platform - DEBUG - RDP connection state = State.Open.
2021-05-26 08:44:56,607 - Session session.platform - Thread 1780 | MainThread
Send GET request asynchronously to https://api.refinitiv.com/data/pricing/beta3/snapshots?universe=EUR=&amp;fields=ASK,BID
2021-05-26 08:44:56,607 - session.platform - DEBUG - Send GET request asynchronously to https://api.refinitiv.com/data/pricing/beta3/snapshots?universe=EUR=&amp;fields=ASK,BID
2021-05-26 08:44:56,608 - Session session.platform - Thread 1780 | MainThread
Request to https://api.refinitiv.com/data/pricing/beta3/snapshots?universe=EUR=&amp;fields=ASK,BID
  headers = {'x-tr-applicationid': 'GE-A-00792776-3-6186'}
  params = None
2021-05-26 08:44:56,608 - session.platform - DEBUG - Request to https://api.refinitiv.com/data/pricing/beta3/snapshots?universe=EUR=&amp;fields=ASK,BID
  headers = {'x-tr-applicationid': 'GE-A-00792776-3-6186'}
  params = None
2021-05-26 08:44:56,617 - Session session.platform - Thread 1780 | MainThread
ERROR!!! An error occured while requesting URL('https://api.refinitiv.com/data/pricing/beta3/snapshots?universe=EUR=&amp;fields=ASK,BID&#39;).
2021-05-26 08:44:56,617 - session.platform - ERROR - ERROR!!! An error occured while requesting URL('https://api.refinitiv.com/data/pricing/beta3/snapshots?universe=EUR=&amp;fields=ASK,BID&#39;).
Traceback (most recent call last):
  File "D:\Program Files\Python39\lib\site-packages\httpx\_exceptions.py", line 326, in map_exceptions
    yield
  File "D:\Program Files\Python39\lib\site-packages\httpx\_client.py", line 1492, in _send_single_request
    (status_code, headers, stream, ext) = await transport.arequest(
  File "D:\Program Files\Python39\lib\site-packages\httpx\_transports\default.py", line 169, in arequest
    return await self._pool.arequest(
  File "D:\Program Files\Python39\lib\site-packages\httpcore\_async\connection_pool.py", line 218, in arequest
    response = await connection.arequest(
  File "D:\Program Files\Python39\lib\site-packages\httpcore\_async\connection.py", line 93, in arequest
    self.socket = await self._open_socket(timeout)
  File "D:\Program Files\Python39\lib\site-packages\httpcore\_async\connection.py", line 119, in _open_socket
    return await self.backend.open_tcp_stream(
  File "D:\Program Files\Python39\lib\site-packages\httpcore\_backends\auto.py", line 44, in open_tcp_stream
    return await self.backend.open_tcp_stream(
  File "D:\Program Files\Python39\lib\site-packages\httpcore\_backends\asyncio.py", line 270, in open_tcp_stream
    return SocketStream(
  File "D:\Program Files\Python39\lib\contextlib.py", line 135, in __exit__
    self.gen.throw(type, value, traceback)
  File "D:\Program Files\Python39\lib\site-packages\httpcore\_exceptions.py", line 12, in map_exceptions
    raise to_exc(exc) from None
httpcore.ConnectError: [Errno 11001] getaddrinfo failed
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
  File "d:\Project\refinitiv\rdp_library_connection.py", line 22, in <module>
    response = rdp.Pricing.get_snapshot(universe, fields)
  File "D:\Program Files\Python39\lib\site-packages\refinitiv\dataplatform\pricing\pricing_.py", line 142, in get_snapshot
    result = pricing._get_snapshot(universe=universe,
  File "D:\Program Files\Python39\lib\site-packages\refinitiv\dataplatform\pricing\pricing_.py", line 149, in _get_snapshot
    return self._snapshot_endpoint.session._loop.run_until_complete(self._get_snapshot_async(universe=universe,
  File "D:\Program Files\Python39\lib\site-packages\nest_asyncio.py", line 70, in run_until_complete
    return f.result()
  File "D:\Program Files\Python39\lib\asyncio\futures.py", line 201, in result
    raise self._exception
  File "D:\Program Files\Python39\lib\asyncio\tasks.py", line 256, in __step
    result = coro.send(None)
  File "D:\Program Files\Python39\lib\site-packages\refinitiv\dataplatform\pricing\pricing_.py", line 171, in _get_snapshot_async
    endpoint_response = await self._snapshot_endpoint.send_request_async(Endpoint.RequestMethod.GET,
  File "D:\Program Files\Python39\lib\site-packages\refinitiv\dataplatform\delivery\data\endpoint.py", line 482, in send_request_async
    _response = await self._params.session.http_request_async(_url,
  File "D:\Program Files\Python39\lib\site-packages\refinitiv\dataplatform\core\session\platform_session.py", line 464, in http_request_async
    return await Session.http_request_async(self, url, method=method, headers=headers,
  File "D:\Program Files\Python39\lib\site-packages\refinitiv\dataplatform\core\session\session.py", line 1181, in http_request_async
    raise error
  File "D:\Program Files\Python39\lib\site-packages\refinitiv\dataplatform\core\session\session.py", line 1178, in http_request_async
    request_response = await self._http_session.send(_http_request, **kwargs)
  File "D:\Program Files\Python39\lib\site-packages\httpx\_client.py", line 1396, in send
    response = await self._send_handling_auth(
  File "D:\Program Files\Python39\lib\site-packages\httpx\_client.py", line 1434, in _send_handling_auth
    response = await self._send_handling_redirects(
  File "D:\Program Files\Python39\lib\site-packages\httpx\_client.py", line 1466, in _send_handling_redirects
    response = await self._send_single_request(request, timeout)
  File "D:\Program Files\Python39\lib\site-packages\httpx\_client.py", line 1492, in _send_single_request
    (status_code, headers, stream, ext) = await transport.arequest(
  File "D:\Program Files\Python39\lib\contextlib.py", line 135, in __exit__
    self.gen.throw(type, value, traceback)
  File "D:\Program Files\Python39\lib\site-packages\httpx\_exceptions.py", line 343, in map_exceptions
    raise mapped_exc(message, **kwargs) from exc  # type: ignore
httpx.ConnectError: [Errno 11001] getaddrinfo failed
\Program Files\Python39\lib\site-packages\httpx\_client.py:1772: UserWarning: Unclosed <httpx.AsyncClient object at 0x000001C4595C1C40>. See https://www.python-httpx.org/async/#opening-and-closing-clients for details.
[Done] exited with code=1 in 6.399 seconds

https://api.refinitiv.com/data/pricing/beta3/snapshots?universe=EUR%3D&fields=ASK%2CBID

Thanks for trying that - but for some reason, the additional trace output is not being generated on your PC. I have asked the RDP Library team for more guidance.

Were you able to try the code on a non-corporate network?

Also, what error message do you see if you try to post the following into your browser on your corporate network?

Are you able to contact your internal network security and ask them to assist you as well - in terms of firewall / proxy help?

Hi Umer,

I have not tried setting up RDP outside of the corporate network yet. That will take some more time. I hope we can find another way to test this.

When I try the URL on the server I get the following return message:

I tried to run the code from my local machine (with Anaconda) - outside the corporate environment. In this case the logon to Refinitiv is failing.

2021-05-26 15:30:03,469 - asyncio - DEBUG - Using selector: SelectSelector
1.0.0a8.post1
2021-05-26 15:30:04,977 - httpx._client - DEBUG - HTTP Request: POST https://api.refinitiv.com/auth/oauth2/v1/token "HTTP/1.1 400 Bad Request"
ERROR!!! Failed to request a new token......
         Token requested error b'{"error":"access_denied"  ,"error_description":"Invalid username or password." } '
ERROR!!! response 400 while requesting URL('https://api.refinitiv.com/auth/oauth2/v1/token').
ERROR!!! authentication handler raise an exception.
HTTPStatusError('400 Client Error: Bad Request for url: https://api.refinitiv.com/auth/oauth2/v1/token\nFor more information check: https://httpstatuses.com/400')
ERROR!!! CANNOT authorize to RDP authentication endpoint.
Unexpected error 400 Client Error: Bad Request for url: https://api.refinitiv.com/auth/oauth2/v1/token
For more information check: https://httpstatuses.com/400.
(auto-reconnection is disbled)
---------------------------------------------------------------------------
PlatformSessionError                      Traceback (most recent call last)
<ipython-input-2-7f71bb52b82f> in <module>
     15 print(rdp.__version__)
     16 
---> 17 rdp.open_platform_session(
     18     APP_KEY,
     19     rdp.GrantPassword(

~\anaconda3\lib\site-packages\refinitiv\dataplatform\factory\core_factory.py in open_platform_session(app_key, grant, deployed_platform_host, deployed_platform_username)
     93     close_session()
     94     set_default_session(session)
---> 95     session.open()
     96     return session
     97 

~\anaconda3\lib\site-packages\refinitiv\dataplatform\core\session\session.py in open(self)
    755             # session is already opened or is opening
    756             return self._state
--> 757         self._loop.run_until_complete(self.open_async())
    758         return self._state
    759 

~\anaconda3\lib\site-packages\nest_asyncio.py in run_until_complete(self, future)
     96                 raise RuntimeError(
     97                     'Event loop stopped before Future completed.')
---> 98             return f.result()
     99         finally:
    100             self._thread_id = old_thread_id

~\anaconda3\lib\asyncio\futures.py in result(self)
    176         self.__log_traceback = False
    177         if self._exception is not None:
--> 178             raise self._exception
    179         return self._result
    180 

~\anaconda3\lib\asyncio\tasks.py in __step(***failed resolving arguments***)
    278                 # We use the `send` method directly, because coroutines
    279                 # don't have `__iter__` and `__next__` methods.
--> 280                 result = coro.send(None)
    281             else:
    282                 result = coro.throw(exc)

~\anaconda3\lib\site-packages\refinitiv\dataplatform\core\session\platform_session.py in open_async(self)
    315 
    316         #   do authentication process with Refinitiv Data Platform (RDP), if it's necessary
--> 317         self._connection.open()
    318 
    319         #   the platform session is ready,

~\anaconda3\lib\site-packages\refinitiv\dataplatform\core\session\connection.py in open(self)
    184 
    185         #   call authorize for do an authentication mechanism to the RDP platform by another thread
--> 186         self.authorize()
    187 
    188     def authorize(self):

~\anaconda3\lib\site-packages\refinitiv\dataplatform\core\session\connection.py in authorize(self)
    230 
    231                     #   raise the error
--> 232                     raise PlatformSessionError(-1, 'ERROR!!! Authentication handler failed to request a access token.\n'
    233                                                     f'{self._authentication_token_handler_thread.last_exception}')
    234 

PlatformSessionError: Error code -1 | ERROR!!! Authentication handler failed to request a access token.
400 Client Error: Bad Request for url: https://api.refinitiv.com/auth/oauth2/v1/token
For more information check: https://httpstatuses.com/400