C# datascope select SDK : When we run more than one download concurrently, e.g., download 2 days of data at the same time, randomly data of some of the days are missing in the downloaded files
We have a client : tried to download tick data from datascope for below RICs:
CHK.O,Ric
GDX,Ric
SPY,Ric
GE,Ric
EEM,Ric
BAC,Ric
AMD.O,Ric
XLF,Ric
F,Ric
VXX,Ric
NIO,Ric
T,Ric
TVIXF.PK,Ric
We are using the C# datascope select SDK call for the download. The problem we encounter is: when we run more than one download concurrently, e.g., download 2 days of data at the same time, randomly data of some of the days are missing in the downloaded files. I attach a log file Concurrent_download.log Concurrent_download.txt to explain this in details.
The log shows that we wanted to download 4 days of above data, we run the C# downloader to open 4 connections in parallel each for a day. All of them were downloaded, the notes show correct tick count and everything looks fine. But in fact out of these 4 days, only one day's downloaded file contains full data. The rest 3 contains some data but miss most of it. Our log file shows 3 exceptions were thrown during writing the file to disk: One or more errors occurred. (Error while copying content to a stream.) This sounds like an exception thrown from HttpClient of .NET.
When we download these 4 days of data individually with single connection, all data can be downloaded successfully. It iindicates to us that we have problem with concurrent connection with datascope select. However this only happen with relatively bigger tick data, we use same C# program to download small daily data with 4 concurrent connections, we don't observe similar problem.
Hello @beera.rajesh01 ,
On the theoretical level:
For the majority of the request templates, the extraction requests are processed 2 at at time. Up to 50 can be submitted, but they are being queued for processing, see Tick History REST API User Guide->Concurrent Requests. Some templates it's only 1 request processed at a time.
So I personally prefer not to submit requests that are guaranteed to be queued instead of processing, especially large ones that will be queued longer.
However, if submitted, all requests are supposed to be processed completely.
---
To try to determine why this is not happening in this case, we will need to try to reproduce the issue by running the exact same test scenario on our side.
I see two ways:
The client can provide the complete requests, and you can submit 4 parallel requests via HTTP REST tool, for example Postman- verify if the results received will be complete. This is the cleaner test case.
Or the client can share their C# code and then you can try to test to verify using their code.
I hope that this information helps