Empty DF returned using async RDP library with multiple RICs and platform session
Hi,
When pulling prices for all index components using async RDP library part of results are returned empty, it happens only with platform session which I need for calculator I am creating, whereas with desktop session it is significantly slower but all results are returned correctly. Could you confirm why it is happening, is there any limit that I am hitting and how can I bypass this to get all instruments queried?
Sample results:
0#.FTSE first run - 37 RICs missing, rest returned correctly
No CLOSE for 35 RIC(s): ['BLND.L', 'SKG.L', 'BP.L', 'ADML.L', 'SBRY.L', 'PHNX.L', 'FLTRF.L', 'IHG.L', 'BT.L', 'MNDI.L', 'CRH.L', 'SPX.L', 'PSN.L', 'RIO.L', 'RDSb.L', 'CCH.L', 'RSA.L', 'RR.L', 'BMEB.L', 'SSE.L', 'LLOY.L', 'NWG.L', 'NXT.L', 'ITRK.L', 'BDEV.L', 'AVV.L', 'RDSa.L', 'FERG.L', 'ICP.L', 'BAES.L', 'DCC.L', 'ULVR.L', 'OCDO.L', 'TSCO.L', 'LGEN.L'].
0#.FTSE second run 15 RICs missing, rest returned correctly
No CLOSE for 15 RIC(s): ['MNDI.L', 'BATS.L', 'SPX.L', 'CCH.L', 'SSE.L', 'SMT.L', 'POLYP.L', 'GLEN.L', 'BARC.L', 'BAES.L', 'DCC.L', 'ULVR.L', 'LSEG.L', 'TSCO.L', 'LGEN.L'].
0#.SPX 139 RICs missing, rest returned correctly
No 139 RIC(s): ['AMAT.OQ', 'FCX.N', 'MAS.N', 'UAA.N', 'PVH.N', 'CBRE.N', 'CINF.OQ', 'SEE.N', 'WMB.N', 'EA.OQ', 'TMUS.OQ', 'CB.N', 'MDLZ.OQ', 'BLL.N', 'PFG.OQ', 'JPM.N', 'TTWO.OQ', 'RMD.N', 'CMG.N', 'PCAR.OQ', 'CHTR.OQ', 'PWR.N', 'COO.N', 'SNA.N', 'CMI.N', 'OXY.N', 'F.N', 'CVS.N', 'PH.N', 'GILD.OQ', 'PNW.N', 'DE.N', 'HSY.N', 'GL.N', 'SLB.N', 'AWK.N', 'FLT.N', 'KLAC.OQ', 'AME.N', 'XLNX.OQ', 'NUE.N', 'WU.N', 'D.N', 'SRE.N', 'AVGO.OQ', 'WRB.N', 'RF.N', 'AES.N', 'ANTM.N', 'GWW.N', 'K.N', 'GOOGL.OQ', 'CCI.N', 'ROP.N', 'C.N', 'ODFL.OQ', 'LEG.N', 'A.N', 'MET.N', 'WYNN.OQ', 'PSA.N', 'BK.N', 'FAST.OQ', 'CFG.N', 'NI.N', 'MO.N', 'CMCSA.OQ', 'EIX.N', 'UPS.N', 'BAC.N', 'VIAC.OQ', 'IP.N', 'MDT.N', 'INTU.OQ', 'EQIX.OQ', 'CHD.N', 'MTD.N', 'PEG.N', 'BIIB.OQ', 'CTSH.OQ', 'ALB.N', 'NCLH.N', 'WAT.N', 'MAR.OQ', 'MNST.OQ', 'FRC.N', 'BMY.N', 'MCHP.OQ', 'CAT.N', 'PG.N', 'ZTS.N', 'STE.N', 'AFL.N', 'CPB.N', 'FITB.OQ', 'XEL.OQ', 'HSIC.OQ', 'MPC.N', 'ICE.N', 'IPGP.OQ', 'SPGI.N', 'ADBE.OQ', 'ADSK.OQ', 'WRK.N', 'FOXA.OQ', 'TSN.N', 'LUV.N', 'TSCO.OQ', 'AON.N', 'AMZN.OQ', 'KR.N', 'BKNG.OQ', 'SWK.N', 'INTC.OQ', 'TRMB.OQ', 'WBA.OQ', 'DISCK.OQ', 'OMC.N', 'GLW.N', 'FTNT.OQ', 'HBI.N', 'DOW.N', 'PPL.N', 'VFC.N', 'ETN.N', 'LNC.N', 'CI.N', 'XYL.N', 'DISH.OQ', 'LUMN.N', 'NRG.N', 'LMT.N', 'PSX.N', 'FLIR.OQ', 'SCHW.N', 'DXC.N', 'SNPS.OQ', 'J.N', 'SIVB.OQ'].
Code used:
import refinitiv.dataplatform as rdp
import asyncio
import pandas as pd
# session = rdp.open_desktop_session('')
session = rdp.open_platform_session(
'',
rdp.GrantPassword(
username = '',
password = ''
)
)
session.open()
default_df = rdp.get_data('0#.FTSE',
fields = ['TR.AssetCategoryCode']
)
async def pull_raw_data():
df = pd.DataFrame()
tasks = []
responses = []
for ric in default_df.instrument:
task = asyncio.ensure_future(rdp.HistoricalPricing.get_summaries_async(universe = ric, start = '04/20/2020, 00:00:00', end = '04/20/2021, 00:00:00', interval = 'P1D', fields = ['TRDPRC_1']))
tasks.append(task)
responses = await asyncio.gather(*tasks)
remaining_instruments = []
for i in range(len(default_df)):
ts = responses[i].data.df
if ts is not None:
ts.rename(columns = {'TRDPRC_1': default_df.instrument[i]}, inplace = True)
df = pd.concat([df, ts], axis=1)
else:
remaining_instruments.append(default_df.instrument[i])
if len(remaining_instruments):
print(f"No {str(len(remaining_instruments))} RIC(s): {str(remaining_instruments)}.")
display(df)
await pull_raw_data()
Best Answer
-
You are mixing up the call from platform and desktop session, so your posted code does not work. Either way, I have found out the reason of the failure - it is too many data requests.
Please refer to the Rate limit section for the historical pricing service in the API playground. It does not specify the actual limit, but does mention that if the rate is exceeded, then the application will receive HTTP-429.
Here is the actual message I received for one of the missing data instrument, using your code and FTSE 100 index:
HTTP/1.1 429 Too Many Requests
{
"error": {
"id": "47828f29-454c-4423-aa73-3059b59d6e0c",
"code": "gw.userLimit",
"message": "too many requests for /data/historical-pricing/v1/views/intraday-summaries/{universe} [GET]",
"status": "Too Many Requests"
}
}You will have to pace out the requests to not breach this limit.
0
Answers
-
You can also check the error response in the failed instrument, by modifying your code like:
for i in range(len(default_df)):
ts = responses[i].data.df
if ts is not None:
ts.rename(columns = {'TRDPRC_1': default_df.instrument[i]}, inplace = True)
df = pd.concat([df, ts], axis=1)
else:
display(responses[i].data.raw)
remaining_instruments.append(default_df.instrument[i])0
Categories
- All Categories
- 3 Polls
- 6 AHS
- 36 Alpha
- 166 App Studio
- 6 Block Chain
- 4 Bot Platform
- 18 Connected Risk APIs
- 47 Data Fusion
- 34 Data Model Discovery
- 684 Datastream
- 1.4K DSS
- 614 Eikon COM
- 5.2K Eikon Data APIs
- 10 Electronic Trading
- Generic FIX
- 7 Local Bank Node API
- 3 Trading API
- 2.9K Elektron
- 1.4K EMA
- 248 ETA
- 552 WebSocket API
- 37 FX Venues
- 14 FX Market Data
- 1 FX Post Trade
- 1 FX Trading - Matching
- 12 FX Trading – RFQ Maker
- 5 Intelligent Tagging
- 2 Legal One
- 23 Messenger Bot
- 3 Messenger Side by Side
- 9 ONESOURCE
- 7 Indirect Tax
- 60 Open Calais
- 275 Open PermID
- 44 Entity Search
- 2 Org ID
- 1 PAM
- PAM - Logging
- 6 Product Insight
- Project Tracking
- ProView
- ProView Internal
- 22 RDMS
- 1.9K Refinitiv Data Platform
- 641 Refinitiv Data Platform Libraries
- 4 LSEG Due Diligence
- LSEG Due Diligence Portal API
- 4 Refinitiv Due Dilligence Centre
- Rose's Space
- 1.2K Screening
- 18 Qual-ID API
- 13 Screening Deployed
- 23 Screening Online
- 12 World-Check Customer Risk Screener
- 1K World-Check One
- 46 World-Check One Zero Footprint
- 45 Side by Side Integration API
- 2 Test Space
- 3 Thomson One Smart
- 10 TR Knowledge Graph
- 151 Transactions
- 143 REDI API
- 1.8K TREP APIs
- 4 CAT
- 26 DACS Station
- 121 Open DACS
- 1.1K RFA
- 104 UPA
- 192 TREP Infrastructure
- 228 TRKD
- 915 TRTH
- 5 Velocity Analytics
- 9 Wealth Management Web Services
- 89 Workspace SDK
- 11 Element Framework
- 5 Grid
- 18 World-Check Data File
- 1 Yield Book Analytics
- 46 中文论坛