ODPS - how to increase throughput?

odps8.2.0.L1.linux.rrg
Hi - we have an application team who are querying ways to increase throughput for lookups when RIC are translated to PE codes using ODPS.
Their current testing of sending 350 RIC codes that needed to be translated in batches of 100 RICs, took about 11 seconds to translate them all.
16:46:59.187 OdpsQuery.cpp:104 OdpsQuery:: CURL request took 1366ms for 9 rics 16:47:02.191 OdpsQuery.cpp:104 OdpsQuery:: CURL request took 3004ms for 100 rics 16:47:04.696 OdpsQuery.cpp:104 OdpsQuery:: CURL request took 2505ms for 100 rics 16:47:08.202 OdpsQuery.cpp:104 OdpsQuery:: CURL request took 3506ms for 100 rics 16:47:10.204 OdpsQuery.cpp:104 OdpsQuery:: CURL request took 2002ms for 39 rics
Is there something you could recommend to increase the throughput? Would larger batches be more efficient? Or running multiple queries in parallel?
Best Answer
-
Hello @paul.cockerham,
ODPS is not designed as a highly performant server, rather it is intended as convenient DACS view component, saving the need for active OpenDACS API integration effort within the custom application.
Please refer to INSTALL AND DEVELOPERS GUIDE on MyRefinitiv, section Performance, for the recommended performance configuration setup description and example performance statistics collected at the time of the release on testbed configured per recommended setup.
The throughput stats are similar, for different calls, and we can observe the specific stats for itemToPE in sub-section Message Throughput for Each Message Type, I am going to assume you are on HTTP 1.1, as it is the recommended, see sub-section HTTP 1.1 Connections, so we are looking at HTTP 1.1 stats for itemToPE for a ballpark expectation on the performance.
In terms of integration design, I would look into obtaining the map once and caching this information, rather then requesting on every call where it is required. This information is not expected to change often, so a periodic check should done to be safe, and sufficient for keeping the maps up to date.
Hope that this information is of help
0
Answers
-
Thanks for your response but the development team have asked if you could still possibly provide guidance on how to best use the ODPS batch lookups?
If they have say 500 RICs to translate, what would be the ideal size of an individual request in terms of RIC count?
Can it help reduce the overall time if they perform multiple HTTP calls in parallel?
0 -
Hello @paul.cockerham ,
Conveying additional information from our development:
itemToPE is a very slow call since there is a huge amount of processing to do. You can run a few different itemtope queries for different UUID’s of course, but doing multiple in parallel for the same UUID is not going to get you much for better performance if the profile has to be pulled from the DACS Servers instead of the cache.
In terms of speed of processing, it may be worthwhile to look into subscribing a RIC from ADS and obtaining RIC to PE map this way.
0
Categories
- All Categories
- 3 Polls
- 6 AHS
- 36 Alpha
- 166 App Studio
- 6 Block Chain
- 4 Bot Platform
- 18 Connected Risk APIs
- 47 Data Fusion
- 34 Data Model Discovery
- 690 Datastream
- 1.4K DSS
- 629 Eikon COM
- 5.2K Eikon Data APIs
- 11 Electronic Trading
- 1 Generic FIX
- 7 Local Bank Node API
- 3 Trading API
- 2.9K Elektron
- 1.4K EMA
- 255 ETA
- 559 WebSocket API
- 39 FX Venues
- 15 FX Market Data
- 1 FX Post Trade
- 1 FX Trading - Matching
- 12 FX Trading – RFQ Maker
- 5 Intelligent Tagging
- 2 Legal One
- 25 Messenger Bot
- 3 Messenger Side by Side
- 9 ONESOURCE
- 7 Indirect Tax
- 60 Open Calais
- 280 Open PermID
- 45 Entity Search
- 2 Org ID
- 1 PAM
- PAM - Logging
- 6 Product Insight
- Project Tracking
- ProView
- ProView Internal
- 23 RDMS
- 2K Refinitiv Data Platform
- 716 Refinitiv Data Platform Libraries
- 4 LSEG Due Diligence
- LSEG Due Diligence Portal API
- 4 Refinitiv Due Dilligence Centre
- Rose's Space
- 1.2K Screening
- 18 Qual-ID API
- 13 Screening Deployed
- 23 Screening Online
- 12 World-Check Customer Risk Screener
- 1K World-Check One
- 46 World-Check One Zero Footprint
- 45 Side by Side Integration API
- 2 Test Space
- 3 Thomson One Smart
- 10 TR Knowledge Graph
- 151 Transactions
- 143 REDI API
- 1.8K TREP APIs
- 4 CAT
- 27 DACS Station
- 121 Open DACS
- 1.1K RFA
- 106 UPA
- 194 TREP Infrastructure
- 229 TRKD
- 918 TRTH
- 5 Velocity Analytics
- 9 Wealth Management Web Services
- 95 Workspace SDK
- 11 Element Framework
- 5 Grid
- 19 World-Check Data File
- 1 Yield Book Analytics
- 48 中文论坛