question

Upvotes
Accepted
beera.rajesh01 avatar image
1 ●0 ●1 ●1

C# datascope select SDK : When we run more than one download concurrently, e.g., download 2 days of data at the same time, randomly data of some of the days are missing in the downloaded files

We have a client : tried to download tick data from datascope for below RICs:

CHK.O,Ric

GDX,Ric

SPY,Ric

GE,Ric

EEM,Ric

BAC,Ric

AMD.O,Ric

XLF,Ric

F,Ric

VXX,Ric

NIO,Ric

T,Ric

TVIXF.PK,Ric

We are using the C# datascope select SDK call for the download. The problem we encounter is: when we run more than one download concurrently, e.g., download 2 days of data at the same time, randomly data of some of the days are missing in the downloaded files. I attach a log file Concurrent_download.log Concurrent_download.txt to explain this in details.


The log shows that we wanted to download 4 days of above data, we run the C# downloader to open 4 connections in parallel each for a day. All of them were downloaded, the notes show correct tick count and everything looks fine. But in fact out of these 4 days, only one day's downloaded file contains full data. The rest 3 contains some data but miss most of it. Our log file shows 3 exceptions were thrown during writing the file to disk: One or more errors occurred. (Error while copying content to a stream.) This sounds like an exception thrown from HttpClient of .NET.


When we download these 4 days of data individually with single connection, all data can be downloaded successfully. It iindicates to us that we have problem with concurrent connection with datascope select. However this only happen with relatively bigger tick data, we use same C# program to download small daily data with 4 concurrent connections, we don't observe similar problem.

datascope-selecttick-history-rest-apic#download-file
concurrent-download.txt (436.0 KiB)
icon clock
10 |1500

Up to 2 attachments (including images) can be used with a maximum of 512.0 KiB each and 1.0 MiB total.

raksina.samasiri avatar image   raksina.samasiri

hi @beera.rajesh01

Thank you for your participation in the forum. Is the reply below satisfactory in resolving your query?
If so please can you click the 'Accept' text next to the appropriate reply? This will guide all community members who have a similar question.

Thanks,
AHS

Shadab Hussain avatar image   Shadab Hussain

Hi @beera.rajesh01,

Please be informed that a reply has been verified as correct in answering the question, and has been marked as such.


Thanks,

AHS

1 Answer

· Write an Answer
Upvotes
Accepted
zoya faberov avatar image
32.2k ●40 ●11 ●20

Hello @beera.rajesh01 ,

On the theoretical level:

For the majority of the request templates, the extraction requests are processed 2 at at time. Up to 50 can be submitted, but they are being queued for processing, see Tick History REST API User Guide->Concurrent Requests. Some templates it's only 1 request processed at a time.

So I personally prefer not to submit requests that are guaranteed to be queued instead of processing, especially large ones that will be queued longer.

However, if submitted, all requests are supposed to be processed completely.

---

To try to determine why this is not happening in this case, we will need to try to reproduce the issue by running the exact same test scenario on our side.

I see two ways:

The client can provide the complete requests, and you can submit 4 parallel requests via HTTP REST tool, for example Postman- verify if the results received will be complete. This is the cleaner test case.

Or the client can share their C# code and then you can try to test to verify using their code.

I hope that this information helps

icon clock
10 |1500

Up to 2 attachments (including images) can be used with a maximum of 512.0 KiB each and 1.0 MiB total.

Write an Answer

Hint: Notify or tag a user in this post by typing @username.

Up to 2 attachments (including images) can be used with a maximum of 512.0 KiB each and 1.0 MiB total.