ODPS - how to increase throughput?
odps8.2.0.L1.linux.rrg
Hi - we have an application team who are querying ways to increase throughput for lookups when RIC are translated to PE codes using ODPS.
Their current testing of sending 350 RIC codes that needed to be translated in batches of 100 RICs, took about 11 seconds to translate them all.
16:46:59.187 OdpsQuery.cpp:104 OdpsQuery:: CURL request took 1366ms for 9 rics 16:47:02.191 OdpsQuery.cpp:104 OdpsQuery:: CURL request took 3004ms for 100 rics 16:47:04.696 OdpsQuery.cpp:104 OdpsQuery:: CURL request took 2505ms for 100 rics 16:47:08.202 OdpsQuery.cpp:104 OdpsQuery:: CURL request took 3506ms for 100 rics 16:47:10.204 OdpsQuery.cpp:104 OdpsQuery:: CURL request took 2002ms for 39 rics
Is there something you could recommend to increase the throughput? Would larger batches be more efficient? Or running multiple queries in parallel?
Hello @paul.cockerham,
ODPS is not designed as a highly performant server, rather it is intended as convenient DACS view component, saving the need for active OpenDACS API integration effort within the custom application.
Please refer to INSTALL AND DEVELOPERS GUIDE on MyRefinitiv, section Performance, for the recommended performance configuration setup description and example performance statistics collected at the time of the release on testbed configured per recommended setup.
The throughput stats are similar, for different calls, and we can observe the specific stats for itemToPE in sub-section Message Throughput for Each Message Type, I am going to assume you are on HTTP 1.1, as it is the recommended, see sub-section HTTP 1.1 Connections, so we are looking at HTTP 1.1 stats for itemToPE for a ballpark expectation on the performance.
In terms of integration design, I would look into obtaining the map once and caching this information, rather then requesting on every call where it is required. This information is not expected to change often, so a periodic check should done to be safe, and sufficient for keeping the maps up to date.
Hope that this information is of help
Thanks for your response but the development team have asked if you could still possibly provide guidance on how to best use the ODPS batch lookups?
If they have say 500 RICs to translate, what would be the ideal size of an individual request in terms of RIC count?
Can it help reduce the overall time if they perform multiple HTTP calls in parallel?
Hello @paul.cockerham ,
Conveying additional information from our development:
itemToPE is a very slow call since there is a huge amount of processing to do. You can run a few different itemtope queries for different UUID’s of course, but doing multiple in parallel for the same UUID is not going to get you much for better performance if the profile has to be pulled from the DACS Servers instead of the cache.
In terms of speed of processing, it may be worthwhile to look into subscribing a RIC from ADS and obtaining RIC to PE map this way.