The REST API Tutorials Introduction document states that: "It mirrors the DSS web GUI, including its extraction limit, which is now 75000 instruments".
Questions:
1. Is this a fact for both a) intraday, and b) EOD queries across various templates, say T&C, Corporate Actions, etc.?
2. How is the API handling errors related to extralarge # of instruments uploaded/downloaded? Would I get an error if I upload, say 175,000 instruments? What exact error would I get?
3. What exact error would I get if looping does not work as expected in the above example.
@ Kai Tang
After further investigation, here is some more detailed information:
From the DataScope Select user guide:
Extraction processing limits for the GUI, FTP and REST API platforms are identified by report type below. Note: SOAP API limits differ. See the DataScope Select SOAP API Programmer Guides for details.
EOD Pricing: 75,000 instruments before file code and chain expansion; 150,000 instruments after expansion
Intraday Pricing: 50,000 instruments before and after file code and chain expansion
Premium EOD Pricing: 25,000 instruments before and after file code and chain expansion
Premium Pricing: 50,000 instruments before file code and chain expansion; 150,000 instruments after expansion
Single Historical Price: 50,000 instruments
Timeseries Pricing: 1,000 instruments
Bond Schedules: 150,000 instruments
Fund Allocation: 600 instruments before and after file code and chain expansion
MBS-Factor History: 20,000 instruments before and after file code and chain expansion
Ownership: 500 instruments
Ratings: 50,000 instruments before file code and chain expansion; 150,000 instruments after expansion; 1,000 legal entities before and after file code and chain expansion
Symbol Cross Reference: 50,000 instruments before file code and chain expansion; 150,000 instruments after expansion
Terms and Conditions: 50,000 instruments before file code and chain expansion; 150,000 instruments after expansion; 1,000 legal entities before file code and chain expansion; 150,000 after instrument expansion
Tranche-Factor History: 3,000 instruments before and after file code and chain expansion
Composite: 50,000 instruments before file code and chain expansion; 150,000 instruments after expansion
Fixed Income Analytics: 50,000 instruments before and after file code and chain expansion
StarMine: 50,000 instruments before and after file code and chain expansion
Technical Indicators: 50,000 instruments before and after file code and chain expansion
Standard Events: 150,000 instruments
IPO Events: 150,000 instruments
ISO 15022: 150,000 instruments
Entity Audit: 10,000 entities
Entity Detail: 50,000 entities
Entity Hierarchy: 2,000 entities
Actual Estimates: 3,000 instruments before and after file code and chain expansion
Detail Estimates: 3,000 instruments before and after file code and chain expansion
Summary Estimates: 3,000 instruments before and after file code and chain expansion
Company Level Footnotes: 3,000 instruments before and after file code and chain expansion
Details Level Footnotes: 3,000 instruments before and after file code and chain expansion
News Analytics & News Items: Instrument limits are dependent on the number of days for which you want to retrieve news:
0-7 Days – 50,000 instruments
8-31 Days – 25,000 instruments
32-366 Days – 10,000 instruments
367+ Days – 5,000 instruments
The maximum number of records that can be returned for RIC-less extractions for News Analytics and News Items is 400,000.
News Analytics Commodities: For extractions with a specified date range of 32 or more days, only the first 15 commodity codes in your input list will return data.
The list of request limits can be found online in the Extraction Limits help page (to access that page you need a valid account).
I don't know the exact errors you'd receive, but the following should hopefully clarify how DSS handles large lists, and help you with any issues you are facing.
An instrument list can be used to query different data types (intraday, EoD, etc.). The limits we are discussing here apply to the instrument list, independently of the data queries for which the instrument lists will be used (see my other answer for details on that, as queries can have lower limits depending on the data type).
Additional details to the DSS handling of large instrument lists (and legal entities):
Additional note on chains: what is the limit to the number of chain instrument codes that can be added to an instrument list?
The limit to the number of chain instrument codes that can be added to an instrument list is 150000. The chain instrument code does not expand immediately upon import. It expands when the extraction is run and your output file displays a row for each expanded item.
The first N expanded items are processed in an extraction (N depends on the type of data you are querying, see my other answer).
Sample code: .Net SDK Tutorial 7 describes an EoD request, with looping to handle more than 75000 instruments (which is the limit for an EoD request), download the code here. Caveat: it submits instruments in chunks of 75000, as read from an input CSV file. This counting method does not handle one particular case: instrument expansion. Chains in the imported list they will be expanded, so the final number of instruments will be longer than the original list ! As long as it does not exceed 150000 instruments after expansion, it ought to work all the same.
Workaround: the limit of 75000 is a variable in the code; setting it to a lower value will result in the code submitting smaller chunks, thus mitigating these special cases (better having a few loops more than missing instruments).
How to schedule DSS Legal Entity Detail extraction with Java REST API
DSS RestAPI returning MonitorURL with localhost in the hostname. Is this a bug?
Sample C# code that excludes deleted items when pulling data from DSS Corporate Actions table
On demand extraction via HTTP requests with list and template ids?