DSS REST API- What are the limits on the number of instruments uploaded/downloaded
The REST API Tutorials Introduction document states that: "It mirrors the DSS web GUI, including its extraction limit, which is now 75000 instruments".
Questions:
1. Is this a fact for both a) intraday, and b) EOD queries across various templates, say T&C, Corporate Actions, etc.?
2. How is the API handling errors related to extralarge # of instruments uploaded/downloaded? Would I get an error if I upload, say 175,000 instruments? What exact error would I get?
3. What exact error would I get if looping does not work as expected in the above example.
@ Kai Tang
After further investigation, here is some more detailed information:
- Extractions have a limit on the instrument list size. That limit varies depending on the type of requested data, from 500 to 150000 instruments.
- Extraction notes: if an instrument list is too long for a specific extraction request, the extraction notes file will mention it, with an error of type:
WARNING: Instrument count (75000) > TNC limit (50000) -- extra instruments ignored.
Here TNC stands for Terms and Conditions, for which the limit is 50000. - DSS 10.6: we are in the process of adding pre-expansion limit checks and returning the appropriate 400 error to the requestor before the extraction is made. This change we are planning for the next release – 10.6.0
From the DataScope Select user guide:
Extraction Processing
Extraction processing limits for the GUI, FTP and REST API platforms are identified by report type below. Note: SOAP API limits differ. See the DataScope Select SOAP API Programmer Guides for details.
Pricing
EOD Pricing: 75,000 instruments before file code and chain expansion; 150,000 instruments after expansion
Intraday Pricing: 50,000 instruments before and after file code and chain expansion
Premium EOD Pricing: 25,000 instruments before and after file code and chain expansion
Premium Pricing: 50,000 instruments before file code and chain expansion; 150,000 instruments after expansion
Single Historical Price: 50,000 instruments
Timeseries Pricing: 1,000 instruments
Reference Data
Bond Schedules: 150,000 instruments
Fund Allocation: 600 instruments before and after file code and chain expansion
MBS-Factor History: 20,000 instruments before and after file code and chain expansion
Ownership: 500 instruments
Ratings: 50,000 instruments before file code and chain expansion; 150,000 instruments after expansion; 1,000 legal entities before and after file code and chain expansion
Symbol Cross Reference: 50,000 instruments before file code and chain expansion; 150,000 instruments after expansion
Terms and Conditions: 50,000 instruments before file code and chain expansion; 150,000 instruments after expansion; 1,000 legal entities before file code and chain expansion; 150,000 after instrument expansion
Tranche-Factor History: 3,000 instruments before and after file code and chain expansion
Pricing & Reference Data
Composite: 50,000 instruments before file code and chain expansion; 150,000 instruments after expansion
Analytics Data
Fixed Income Analytics: 50,000 instruments before and after file code and chain expansion
StarMine: 50,000 instruments before and after file code and chain expansion
Technical Indicators: 50,000 instruments before and after file code and chain expansion
Corporate Actions
Standard Events: 150,000 instruments
IPO Events: 150,000 instruments
ISO 15022: 150,000 instruments
Entity Data
Entity Audit: 10,000 entities
Entity Detail: 50,000 entities
Entity Hierarchy: 2,000 entities
Estimates Data
Actual Estimates: 3,000 instruments before and after file code and chain expansion
Detail Estimates: 3,000 instruments before and after file code and chain expansion
Summary Estimates: 3,000 instruments before and after file code and chain expansion
Company Level Footnotes: 3,000 instruments before and after file code and chain expansion
Details Level Footnotes: 3,000 instruments before and after file code and chain expansion
News
News Analytics & News Items: Instrument limits are dependent on the number of days for which you want to retrieve news:
0-7 Days – 50,000 instruments
8-31 Days – 25,000 instruments
32-366 Days – 10,000 instruments
367+ Days – 5,000 instruments
The maximum number of records that can be returned for RIC-less extractions for News Analytics and News Items is 400,000.
News Analytics Commodities: For extractions with a specified date range of 32 or more days, only the first 15 commodity codes in your input list will return data.
The list of request limits can be found online in the Extraction Limits help page (to access that page you need a valid account).
I don't know the exact errors you'd receive, but the following should hopefully clarify how DSS handles large lists, and help you with any issues you are facing.
An instrument list can be used to query different data types (intraday, EoD, etc.). The limits we are discussing here apply to the instrument list, independently of the data queries for which the instrument lists will be used (see my other answer for details on that, as queries can have lower limits depending on the data type).
Additional details to the DSS handling of large instrument lists (and legal entities):
- APIs: the SOAP API has a limit of 1000 instruments. The REST API manages up to 75000 instruments.
- Instrument lists: DSS allows import of up to 85000 instruments, but only 75000 will be validated. The instrument import limit is set 10000 above the instrument list limit to allow for instrument validation. Instruments are validated after being imported; therefore it is possible that some imported instruments will not be added to the input list due to preference settings, instrument duplication, and not found instruments. The first 75000 validated instruments will be added to the input list. All other imported instruments will be ignored.
- Entity lists: a maximum of 50000 legal entities can be imported in an entity list.
Additional note on chains: what is the limit to the number of chain instrument codes that can be added to an instrument list?
The limit to the number of chain instrument codes that can be added to an instrument list is 150000. The chain instrument code does not expand immediately upon import. It expands when the extraction is run and your output file displays a row for each expanded item.
The first N expanded items are processed in an extraction (N depends on the type of data you are querying, see my other answer).
Sample code: .Net SDK Tutorial 7 describes an EoD request, with looping to handle more than 75000 instruments (which is the limit for an EoD request), download the code here. Caveat: it submits instruments in chunks of 75000, as read from an input CSV file. This counting method does not handle one particular case: instrument expansion. Chains in the imported list they will be expanded, so the final number of instruments will be longer than the original list ! As long as it does not exceed 150000 instruments after expansion, it ought to work all the same.
Workaround: the limit of 75000 is a variable in the code; setting it to a lower value will result in the code submitting smaller chunks, thus mitigating these special cases (better having a few loops more than missing instruments).