For a deeper look into our DataScope Select REST API, look into:

Overview |  Quickstart |  Documentation |  Downloads |  Tutorials

question

Upvotes
Accepted
5 3 4 5

DSS .NET SDK performance issues - ondemand extract

We are trying to fetch EOD prices through on-demand extraction .NET SDK for more than 10k instruments.This takes 20+ minutes to complete which is almost 10 times longer than time taken while using http endpoint.

Is it to be expected or we can do something to speed it up. If you need more information, please let us know.


Regards

Mithun

dss-rest-apidatascope-selectdss.netperformance
icon clock
10 |1500

Up to 2 attachments (including images) can be used with a maximum of 512.0 KiB each and 1.0 MiB total.

Hi @mithun.ghosh,

Could you share the snippet code for .Net SDK and request's body for http endpoint? So, I can try to replicate the issue?

Upvotes
Accepted
82.8k 280 53 77

@mithun.ghosh

From the code, you are using the ExtractWithNotes endpoint. Therefore, the response should contain extraction information. For example:

  "Notes": [
        "Extraction Services Version 13.3.41369 (50b192356928), Built Feb  3 2020 17:30:35\r\nHoliday Rollover of Universal Close Price waived.\r\nProcessing started at 03/19/2020 05:50:17.\r\nUser ID: 9008895\r\nExtraction ID: 424491044\r\nSchedule: _OnD_0x07065045ef517094 (ID = 0x07065045fba17094)\r\nInput List (2 items): _OnD_0x07065045ef517094 (ID = 07065045f4917094) Created: 03/19/2020 05:50:15 Last Modified: 03/19/2020 05:50:16\r\nSchedule Time: 03/19/2020 05:50:16\r\nReport Template (73 fields): _OnD_0x07065045ef517094 (ID = 0x07065045f0417094) Created: 03/19/2020 05:50:15 Last Modified: 03/19/2020 05:50:15\r\n(CSP,438516AC0,CPN) is inactive.\r\nColumn 'CIN Code' suppressed for lack of 'CIN Code' permission.\r\nProcessing completed successfully at 03/19/2020 05:50:18, taking 0.728 Secs.\r\nExtraction finished at 03/19/2020 05:50:18 UTC, with servers: x04A06, QSHC16 (0.3 secs), QSSHA1 (0.0 secs)\r\nUsage Summary for User 9008895, Client 65508, Template Type EOD Pricing\r\nBase Usage\r\n        Instrument                          Instrument                   Terms          Price\r\n  Count Type                                Subtype                      Source         Source\r\n------- ----------------------------------- ---------------------------- -------------- ----------------------------------------\r\n      1 Corporate                                                        N/A            N/A\r\n      1 Equities                                                         N/A            N/A\r\n-------\r\n      2 Total instruments charged.\r\n      0 Instruments with no reported data.\r\n=======\r\n      2 Instruments in the input list.\r\nNo TRPS complex usage to report -- 2 Instruments in the input list had no reported data.\r\n"
    ]

From this information, you can compare the processing time between .NET and HTTP.

icon clock
10 |1500

Up to 2 attachments (including images) can be used with a maximum of 512.0 KiB each and 1.0 MiB total.

Upvotes
5 3 4 5

Hi Veerapath,

Thanks for your prompt response.

Please find the .Net SDK code below and attribute list attached with this mail.

public ExtractionResult CreateAndRunEodPricingExtraction(

InstrumentIdentifier[] instrumentIdentifiers, string[] contentFieldNames)

{

//Create an on demand extraction, of type EndOfDayPricingExtractionRequest:

PremiumEndOfDayPricingExtractionRequest extractionEod = new PremiumEndOfDayPricingExtractionRequest

{

IdentifierList = InstrumentIdentifierList.Create(instrumentIdentifiers),

ContentFieldNames = contentFieldNames

};

//Run the extraction.

//This call is blocking, it returns when the extraction is completed:

return extractionsContext.ExtractWithNotes(extractionEod);

}

EOD_ATTRIBUTE_LIST.txt

We have tried to fetch EOD prices through on-demand extraction with more than 8k instruments. But for the security reason we will not able to share the instrument list with you.

Find the sample one instrument list below :


Identifier Type, Identifier Code,BB_UNIQUE_ID

ISN,US0276XAA,COEK4854


icon clock
10 |1500

Up to 2 attachments (including images) can be used with a maximum of 512.0 KiB each and 1.0 MiB total.

Upvotes
11.3k 26 9 14

Hi @mithun.ghosh,

The ExtractWithNotes() function you are using with the .Net SDK will return when extraction is completed and data is received. Whereas the ExtractWithNotes HTTP endpoint uses Asynchronous mechanism which will return 202 status with Location URL if the request takes too long to process. Application needs to poll the location URL to get data when the extraction is completed.

Could you please confirm whether the time you get data with HTTP endpoint is 10 times faster than .Net SDK?

Also, as suggestion by Jirapongse, you can compare the Note information generated by .Net SDK and HTTP endpont. The Note information contains the time when extraction is started and completed.

icon clock
10 |1500

Up to 2 attachments (including images) can be used with a maximum of 512.0 KiB each and 1.0 MiB total.

Write an Answer

Hint: Notify or tag a user in this post by typing @username.

Up to 2 attachments (including images) can be used with a maximum of 512.0 KiB each and 1.0 MiB total.