Hello @sergei.ermakov ,

Requesting of more then 10k result items (docs) is not supported, despite pagination within record set is supported, so that you should be able to request 5k and then another 5k. See API Playground -> Search -> Reference Guide for more details.

However, a set of TWEB RICs is much bigger then 10k and the question becomes how to request them all.

The working approach could be splitting the oversized single request into several smaller ones using filters.

I am not a content expert, so would select a straightforward approach to splitting:

{
  "View": "Entities",
  "Filter": "ContributorCode eq 'TWEB' and RIC eq '0*'",
  "Select": "RIC",
  "Top": 10000
}

resulting in

{
    "Total": 1048,
    "Hits": [
        {
            "RIC": "0#USBMK=TWEB"
        },
...

Consequently would do:

{
  "View": "Entities",
  "Filter": "ContributorCode eq 'TWEB' and not(RIC eq '0*')",
  "Select": "RIC",
  "Top": 10000
} 

and either examine the result and decide how to cut off the next subset <=10000 for example:

{
  "View": "Entities",
  "Filter": "ContributorCode eq 'TWEB' and RIC eq 'A*'",
  "Select": "RIC",
  "Top": 10000
}

Or, in a loop, just run through first letters alphabetically and digits 0-9, concatenating the resulting subsets into a single result set.

I expect that there may be more efficient approaches to "divide and conquer" approach, for people with more precise understanding of Tradeweb symbology...

Hope the approach information helps.

Hello @sergei.ermakov ,

RDP Search service is currently in Early Access, it is actively evolving and being improved.

For working with very large result sets, the approach that is presently available is to divide the very large result set into several smaller result subsets via Filters and Navigators on request.

I hear you, this is worth noting with product, as possible improvement area going forward.

I would suggest also reviewing article Building Search into your Application Workflow for the awareness on how to scout out Search Metadata, and work effectively with Filters and Navigators to narrow down the search results per your requirement.

Hello @sergei.ermakov and all,

Here is additional information, on RDP Search product direction, that you may find helpful:

Search is meant for fast-searching/retrieval of individual entities, or groups of 100s or at very most a few thousand. Bulk download of tens or hundreds of thousands of documents is beyond the scope and technical capabilities of this API, and there are currently no plans to change that.

thanks, @zoya farberov , appreciate your help

Hi @sergei.ermakov

I would suggest you review the Limits section within the article @zoya faberov referred to. There is also a working Notebook that provides greater details and suggestions that may allow for more accurate groupings for larger data sets.