I completely agree it should be a robust design; that's exactly my point!

I'm just saying the API makes it tedious and inefficient to do that. (i have to guess at retrieval segmentation based on history with no future promise of conformity to the historical distribution) and parse a text string to see if my data was arbitrarily decapitated)

anyway, as i said, thanks for your responses. Much as i dislike the answer, I've got it :-D

Hello @opsp,

Suggest to run the second request as ExtractWithNotes:

{
                {protocol}}{
                {host}}{
                {api}}Extractions/ExtractWithNotes 

If you examine the Extraction Notes for the second request, there is an explanation in them:

"Notes": [
        "Extraction Services Version 14.5.42294 (737b0965c07f), Built Apr  8 2021 13:46:52\r\nProcessing started at 06/09/2021 00:18:37.\r\nUser ID: 19945\r\nExtraction ID: 2000000268379715\r\nSchedule: 0x0795dce6adacb696 (ID = 0x0000000000000000)\r\nInput List (1 items):  (ID = 0x0795dce6adacb696) Created: 06/09/2021 00:18:36 Last Modified: 06/09/2021 00:18:36\r\nOnly 30000 constituents of CHAIN RIC 0#SPX*.U have been expanded because the limit of 30000 has been reached.\r\nTotal instruments after instrument expansion = 30000\r\nReport Template (22 fields): _OnD_0x0795dce6adacb696 (ID = 0x0795dce6adccb696) Created: 06/09/2021 00:11:07 Last Modified: 06/09/2021 00:11:07\r\nSchedule dispatched via message queue (0x0795dce6adacb696)\r\nSchedule Time: 06/09/2021 00:11:08\r\nHistorical Reference Date Range: 01/01/2015 to 06/06/2021\r\nProcessing completed successfully at 06/09/2021 00:40:59, taking 1342.897 Secs.\r\nExtraction finished at 06/09/2021 00:40:59 UTC, with servers: x02q18, DHS (1172.4 secs), QSHC17 (78.6 secs), QSSHA1 (0.2 secs)\r\nWriting RIC maintenance report.\r\n",
        "Identifier,IdentType,Source,RIC,RecordDate,MaintType,OldValue,NewValue,Factor,FactorType\r\n"
    ]

This is a large chain and a huge chunk of history being requested, you are hitting 30k Historical Reference extraction limit, please see Extraction Limits Guidance. Therefore, not all instruments that are requested are present in the result.

I would suggest to limit you time interval to a significantly smaller chunk of time, and progressively request all the Historical Reference that you require by sliding the time window. I would go with 1 month interval of Historical Reference for this chain and set of fields.

Hope this helps

ok. it's not huge, it's very small. it's a remarkably small amount of data.

tedious that the sever cant just manage the segmentation for me. hey ho. i'm new to this api, I guess we have a difference of opinion about what a big retrieval is. my view is that if I can fit it on a zipdisk from the 1990s, it isn't large.... anyway....

however, can you clarify that the chain is fully expanded for the ElektronTimeseriesExtractionRequest? or is there some limitation on that too?

Hello @opsp,

For a production integration solution, I would look into two aspects:

• Review DSS Extraction Limits Guidance and refer during design stage.
• At least for design stage, request with Extraction Notes and examine the notes the are received, learning of any errors, including limit, quota, etc.

As Elektron Timeseries requested is also limited to 30K instruments expanded, however you did not include an extensive interval of time, such as for Historical Reference request, so I would expect that you did not run into the limit on the request.

... my personal preference as a designer is to err on the side of caution, so during design/POC, I would examine all Extraction Notes to make sure there are no issues with the requests that go undetected, and structure the requests not to approach the product-imposed limits closely, give a bit of room, do you see?

yeah, i see the solution. it's just irritating having the limit so low, lol.

it's not exactly robust programmatically speaking though as I dont know the quantity of instruments that may or may not have turned over within a chain RIC.

is there a programmatic flag that i can check to see if my retrieval hit a limit?

how would you recommend programmatically inspecting my retrievals to be CERTAIN that i haven't hit a limit?

could you also please show me how I might at least get the notes for the first request as well?

(now I'm scared that I hit an arbitrary limit but I dont know it)

and thanks for the responses...certainly very useful

Hello @opsp,

In my opinion, the way to handle this in a robust way, is in design stage. You can request Extraction Notes and parse them programmatically to make sure neither Limits not Quotas are broken.

However, I think the right approach is to plan for it in design stage proactively, not to handle it reactively.

Elektron Timeseries request with Extraction Notes:

{
                {protocol}}{
                {host}}{
                {api}}Extractions/ExtractWithNotes 

yours looks good, "expanded to 9363 RICS", far from API limits.

Please note that in some cases, the requestor is only interested in obtaining the first N data points in the result, so as long as the limit is M, and M >= N, they are not concerned with hitting the limit.

When we tune our required request (the easy way is via Postman) if it's large, we define the appropriate chunk size, to partition the large request into.

Perhaps useful:

There is a lot of helpful into on DSS design contained in DSS Tutorials section, it may be helpful to over at least Tutorials Introduction to get a solid grasp on the approach to integration that is recommended, as well as the specifics of tasks at hand, Postman tutorials come with the downloadable Postman collection.

i hear what you're saying, and I think it'll *probably* be ok, but you do understand that without programmatically detecting limit triggers, even if I were to test and design my software today, what's to stop it breaking tomorrow?

for example, i test and decide that i need to refresh my data on a daily basis so as to not hit the 30k trigger.

what then if tomorow, 30,001 instruments are declared contemporaneously....your suggested segmentation method would fail.... probably wont happen, but I dont like "probably" in my code...

I'll have a look at the tutorials, but I'm suspicious!

thanks for helping though--I'd be completely stuck without you for such an arcane failure.

Hello @opsp,

In my view, if the new app is intended for prod, we need to design for robustness, from the start by partitioning any large requests according to Limit Guidance.

In addition to that, you may wish to examine Extraction Notes that report normal behavior, and Extraction Notes that report limit breach, and consequently, add failure-handling code that will parse Extraction Notes from every request bigger then certain size for potential issue detection. Once you have assured yourself that you are well within the limits, you may wish to remove this step, or to keep in in place.