DSS REST API- What are the limits on the number of instruments uploaded/downloaded

constantin.vulpescu
constantin.vulpescu LSEG
in DSS

The REST API Tutorials Introduction document states that: "It mirrors the DSS web GUI,
including its extraction limit, which is now 75000
instruments".

Questions:

1. Is this a fact for both a) intraday, and b) EOD queries across various templates, say T&C, Corporate Actions, etc.?

2. How is the API handling errors related to extralarge # of instruments uploaded/downloaded? Would I get an error if I upload, say 175,000 instruments? What exact error would I get?

3. What exact error would I get if looping does not work as expected in the above example.

@ Kai Tang

Tagged:

Best Answer

  • Christiaan Meihsl
    Christiaan Meihsl LSEG
    Answer ✓

    After further investigation, here is some more detailed information:

    1. Extractions have a limit on the instrument list size. That limit varies depending on the type
      of requested data      , from 500 to 150000 instruments.
    2. Extraction notes: if an instrument list is too long for a specific extraction request, the extraction notes file will mention it, with an error of type:
      WARNING: Instrument count (75000) > TNC limit (50000) -- extra instruments ignored.
      Here TNC stands for Terms and Conditions, for which the limit is 50000.
    3. DSS 10.6: we are in the process of adding
      pre-expansion limit checks and returning the appropriate 400 error to the requestor before the extraction is made. This change we are planning for the next
      release – 10.6.0

    From the DataScope Select user guide:

    Extraction Processing

    Extraction processing limits for the GUI, FTP and REST
    API platforms are identified by report type below. Note: SOAP API limits differ. See the DataScope Select SOAP API Programmer Guides for details.

    Pricing

    EOD Pricing: 75,000 instruments
    before file code and chain expansion; 150,000 instruments after expansion

    Intraday Pricing: 50,000
    instruments before and after file code and chain expansion

    Premium EOD Pricing: 25,000
    instruments before and after file code and chain expansion

    Premium Pricing: 50,000 instruments
    before file code and chain expansion; 150,000 instruments after expansion

    Single Historical Price: 50,000
    instruments

    Timeseries Pricing: 1,000
    instruments

    Reference Data

    Bond Schedules: 150,000 instruments

    Fund Allocation: 600 instruments
    before and after file code and chain expansion

    MBS-Factor History: 20,000
    instruments before and after file code and chain expansion

    Ownership: 500 instruments

    Ratings: 50,000 instruments before
    file code and chain expansion; 150,000 instruments after expansion; 1,000 legal
    entities before and after file code and chain expansion

    Symbol Cross Reference: 50,000
    instruments before file code and chain expansion; 150,000 instruments after
    expansion

    Terms and Conditions: 50,000
    instruments before file code and chain expansion; 150,000 instruments after
    expansion; 1,000 legal entities before file code and chain expansion; 150,000
    after instrument expansion

    Tranche-Factor History: 3,000
    instruments before and after file code and chain expansion

    Pricing &
    Reference Data

    Composite: 50,000 instruments
    before file code and chain expansion; 150,000 instruments after expansion

    Analytics Data

    Fixed Income Analytics: 50,000
    instruments before and after file code and chain expansion

    StarMine: 50,000 instruments before
    and after file code and chain expansion

    Technical Indicators: 50,000
    instruments before and after file code and chain expansion

    Corporate Actions

    Standard Events: 150,000
    instruments

    IPO Events: 150,000 instruments

    ISO 15022: 150,000 instruments

    Entity Data

    Entity Audit: 10,000 entities

    Entity Detail: 50,000 entities

    Entity Hierarchy: 2,000 entities

    Estimates Data

    Actual Estimates: 3,000 instruments
    before and after file code and chain expansion

    Detail Estimates: 3,000 instruments
    before and after file code and chain expansion

    Summary Estimates: 3,000
    instruments before and after file code and chain expansion

    Company Level Footnotes: 3,000
    instruments before and after file code and chain expansion

    Details Level Footnotes: 3,000
    instruments before and after file code and chain expansion

    News

    News Analytics & News Items:
    Instrument limits are dependent on the number of days for which you want to
    retrieve news:

    0-7
    Days – 50,000 instruments

    8-31
    Days – 25,000 instruments

    32-366
    Days – 10,000 instruments

    367+
    Days – 5,000 instruments

    The
    maximum number of records that can be returned for RIC-less extractions for News
    Analytics     and News Items is 400,000.

    News Analytics Commodities: For extractions with a
    specified date range of 32 or more days, only the first 15 commodity codes in
    your input list will return data.

Answers

  • Christiaan Meihsl
    Christiaan Meihsl LSEG

    I don't know the exact errors you'd receive, but the following should hopefully clarify how DSS handles large lists, and help you with any issues you are facing.

    An instrument list can be used to query different data types (intraday, EoD, etc.). The limits we are discussing here apply to the instrument list, independently of the data queries for which the instrument lists will be used (see my other answer for details on that, as queries can have lower limits depending on the data type).

    Additional details to the DSS handling of large instrument lists (and legal entities):

    1. APIs: the SOAP API has a limit of 1000 instruments. The REST API manages up to 75000 instruments.
    2. Instrument lists: DSS allows import of up to 85000 instruments, but only 75000 will be validated. The instrument import limit is set 10000 above the instrument list limit to allow for instrument validation. Instruments are validated after being imported; therefore it is possible that some imported instruments will not be added to the input list due to preference settings, instrument duplication, and not found instruments. The first 75000 validated instruments will be added to the input list. All other imported instruments will be ignored.
    3. Entity lists: a maximum of 50000 legal entities can be imported in an entity list.

    Additional note on chains: what is the limit to the number of chain instrument codes that can be added to an instrument list?

    The limit to the number of chain instrument codes that can be added to an instrument list is 150000. The chain instrument code does not expand immediately upon import. It expands when the extraction is run and your output file displays a row for each expanded item.

    The first N expanded items are processed in an extraction (N depends on the type of data you are querying, see my other answer).

    Sample code: .Net SDK Tutorial 7 describes an EoD request, with looping to handle more than 75000 instruments (which is the limit for an EoD request), download the code here. Caveat: it submits instruments in chunks of 75000, as read from an input CSV file. This counting method does not handle one particular case: instrument expansion. Chains in the imported list they will be expanded, so the final number of instruments will be longer than the original list ! As long as it does not exceed 150000 instruments after expansion, it ought to work all the same.

    Workaround: the limit of 75000 is a variable in the code; setting it to a lower value will result in the code submitting smaller chunks, thus mitigating these special cases (better having a few loops more than missing instruments).

  • Christiaan Meihsl
    Christiaan Meihsl LSEG
    https://community.developers.refinitiv.com/discussion/comment/6265#Comment_6265

    The list of request limits can be found online in the Extraction Limits help page (to access that page you need a valid account).

Categories