DSS REST API- What are the limits on the number of instruments uploaded/downloaded

The REST API Tutorials Introduction document states that: "It mirrors the DSS web GUI,
including its extraction limit, which is now 75000
instruments".

Questions:

1. Is this a fact for both a) intraday, and b) EOD queries across various templates, say T&C, Corporate Actions, etc.?

2. How is the API handling errors related to extralarge # of instruments uploaded/downloaded? Would I get an error if I upload, say 175,000 instruments? What exact error would I get?

3. What exact error would I get if looping does not work as expected in the above example.

@ Kai Tang

Find more posts tagged with

dss

rest-api

dss-rest-api

datascope-select

Accepted answers

Christiaan Meihsl

After further investigation, here is some more detailed information:

Extractions have a limit on the instrument list size. That limit varies depending on the type
of requested data, from 500 to 150000 instruments.
Extraction notes: if an instrument list is too long for a specific extraction request, the extraction notes file will mention it, with an error of type:
WARNING: Instrument count (75000) > TNC limit (50000) -- extra instruments ignored.
Here TNC stands for Terms and Conditions, for which the limit is 50000.
DSS 10.6: we are in the process of adding
pre-expansion limit checks and returning the appropriate 400 error to the requestor before the extraction is made. This change we are planning for the next
release – 10.6.0

From the DataScope Select user guide:

Extraction Processing

Extraction processing limits for the GUI, FTP and REST
API platforms are identified by report type below. Note: SOAP API limits differ. See the DataScope Select SOAP API Programmer Guides for details.

Pricing

EOD Pricing: 75,000 instruments
before file code and chain expansion; 150,000 instruments after expansion

Intraday Pricing: 50,000
instruments before and after file code and chain expansion

Premium EOD Pricing: 25,000
instruments before and after file code and chain expansion

Premium Pricing: 50,000 instruments
before file code and chain expansion; 150,000 instruments after expansion

Single Historical Price: 50,000
instruments

Timeseries Pricing: 1,000
instruments

Reference Data

Bond Schedules: 150,000 instruments

Fund Allocation: 600 instruments
before and after file code and chain expansion

MBS-Factor History: 20,000
instruments before and after file code and chain expansion

Ownership: 500 instruments

Ratings: 50,000 instruments before
file code and chain expansion; 150,000 instruments after expansion; 1,000 legal
entities before and after file code and chain expansion

Symbol Cross Reference: 50,000
instruments before file code and chain expansion; 150,000 instruments after
expansion

Terms and Conditions: 50,000
instruments before file code and chain expansion; 150,000 instruments after
expansion; 1,000 legal entities before file code and chain expansion; 150,000
after instrument expansion

Tranche-Factor History: 3,000
instruments before and after file code and chain expansion

Pricing &
Reference Data

Composite: 50,000 instruments
before file code and chain expansion; 150,000 instruments after expansion

Analytics Data

Fixed Income Analytics: 50,000
instruments before and after file code and chain expansion

StarMine: 50,000 instruments before
and after file code and chain expansion

Technical Indicators: 50,000
instruments before and after file code and chain expansion

Corporate Actions

Standard Events: 150,000
instruments

IPO Events: 150,000 instruments

ISO 15022: 150,000 instruments

Entity Data

Entity Audit: 10,000 entities

Entity Detail: 50,000 entities

Entity Hierarchy: 2,000 entities

Estimates Data

Actual Estimates: 3,000 instruments
before and after file code and chain expansion

Detail Estimates: 3,000 instruments
before and after file code and chain expansion

Summary Estimates: 3,000
instruments before and after file code and chain expansion

Company Level Footnotes: 3,000
instruments before and after file code and chain expansion

Details Level Footnotes: 3,000
instruments before and after file code and chain expansion

News

News Analytics & News Items:
Instrument limits are dependent on the number of days for which you want to
retrieve news:

0-7
Days – 50,000 instruments

8-31
Days – 25,000 instruments

32-366
Days – 10,000 instruments

367+
Days – 5,000 instruments

The
maximum number of records that can be returned for RIC-less extractions for News
Analytics and News Items is 400,000.

News Analytics Commodities: For extractions with a
specified date range of 32 or more days, only the first 15 commodity codes in
your input list will return data.

All comments

Christiaan Meihsl

I don't know the exact errors you'd receive, but the following should hopefully clarify how DSS handles large lists, and help you with any issues you are facing.

An instrument list can be used to query different data types (intraday, EoD, etc.). The limits we are discussing here apply to the instrument list, independently of the data queries for which the instrument lists will be used (see my other answer for details on that, as queries can have lower limits depending on the data type).

Additional details to the DSS handling of large instrument lists (and legal entities):

APIs: the SOAP API has a limit of 1000 instruments. The REST API manages up to 75000 instruments.
Instrument lists: DSS allows import of up to 85000 instruments, but only 75000 will be validated. The instrument import limit is set 10000 above the instrument list limit to allow for instrument validation. Instruments are validated after being imported; therefore it is possible that some imported instruments will not be added to the input list due to preference settings, instrument duplication, and not found instruments. The first 75000 validated instruments will be added to the input list. All other imported instruments will be ignored.
Entity lists: a maximum of 50000 legal entities can be imported in an entity list.

Additional note on chains: what is the limit to the number of chain instrument codes that can be added to an instrument list?

The limit to the number of chain instrument codes that can be added to an instrument list is 150000. The chain instrument code does not expand immediately upon import. It expands when the extraction is run and your output file displays a row for each expanded item.

The first N expanded items are processed in an extraction (N depends on the type of data you are querying, see my other answer).

Sample code: .Net SDK Tutorial 7 describes an EoD request, with looping to handle more than 75000 instruments (which is the limit for an EoD request), download the code here. Caveat: it submits instruments in chunks of 75000, as read from an input CSV file. This counting method does not handle one particular case: instrument expansion. Chains in the imported list they will be expanded, so the final number of instruments will be longer than the original list ! As long as it does not exceed 150000 instruments after expansion, it ought to work all the same.

Workaround: the limit of 75000 is a variable in the code; setting it to a lower value will result in the code submitting smaller chunks, thus mitigating these special cases (better having a few loops more than missing instruments).

Christiaan Meihsl

After further investigation, here is some more detailed information:

Extractions have a limit on the instrument list size. That limit varies depending on the type
of requested data, from 500 to 150000 instruments.
Extraction notes: if an instrument list is too long for a specific extraction request, the extraction notes file will mention it, with an error of type:
WARNING: Instrument count (75000) > TNC limit (50000) -- extra instruments ignored.
Here TNC stands for Terms and Conditions, for which the limit is 50000.
DSS 10.6: we are in the process of adding
pre-expansion limit checks and returning the appropriate 400 error to the requestor before the extraction is made. This change we are planning for the next
release – 10.6.0

From the DataScope Select user guide:

Extraction Processing

Pricing

EOD Pricing: 75,000 instruments
before file code and chain expansion; 150,000 instruments after expansion

Intraday Pricing: 50,000
instruments before and after file code and chain expansion

Premium EOD Pricing: 25,000
instruments before and after file code and chain expansion

Premium Pricing: 50,000 instruments
before file code and chain expansion; 150,000 instruments after expansion

Single Historical Price: 50,000
instruments

Timeseries Pricing: 1,000
instruments

Reference Data

Bond Schedules: 150,000 instruments

Fund Allocation: 600 instruments
before and after file code and chain expansion

MBS-Factor History: 20,000
instruments before and after file code and chain expansion

Ownership: 500 instruments

Ratings: 50,000 instruments before
file code and chain expansion; 150,000 instruments after expansion; 1,000 legal
entities before and after file code and chain expansion

Symbol Cross Reference: 50,000
instruments before file code and chain expansion; 150,000 instruments after
expansion

Tranche-Factor History: 3,000
instruments before and after file code and chain expansion

Pricing &
Reference Data

Composite: 50,000 instruments
before file code and chain expansion; 150,000 instruments after expansion

Analytics Data

Fixed Income Analytics: 50,000
instruments before and after file code and chain expansion

StarMine: 50,000 instruments before
and after file code and chain expansion

Technical Indicators: 50,000
instruments before and after file code and chain expansion

Corporate Actions

Standard Events: 150,000
instruments

IPO Events: 150,000 instruments

ISO 15022: 150,000 instruments

Entity Data

Entity Audit: 10,000 entities

Entity Detail: 50,000 entities

Entity Hierarchy: 2,000 entities

Estimates Data

Actual Estimates: 3,000 instruments
before and after file code and chain expansion

Detail Estimates: 3,000 instruments
before and after file code and chain expansion

Summary Estimates: 3,000
instruments before and after file code and chain expansion

Company Level Footnotes: 3,000
instruments before and after file code and chain expansion

Details Level Footnotes: 3,000
instruments before and after file code and chain expansion

News

News Analytics & News Items:
Instrument limits are dependent on the number of days for which you want to
retrieve news:

0-7
Days – 50,000 instruments

8-31
Days – 25,000 instruments

32-366
Days – 10,000 instruments

367+
Days – 5,000 instruments

The
maximum number of records that can be returned for RIC-less extractions for News
Analytics and News Items is 400,000.

News Analytics Commodities: For extractions with a
specified date range of 32 or more days, only the first 15 commodity codes in
your input list will return data.

Christiaan Meihsl

The list of request limits can be found online in the Extraction Limits help page (to access that page you need a valid account).

EXPLORE OUR SITES