question

Upvotes
Accepted
helen.ristov avatar image
20 ●2 ●3 ●6

Extracting TRTH Time and Sales Data to a File

Hello, When I try to extract the Time and Sales Data to a file I only get partial data. I've attached a snippet of my code for the extraction request which writes to a file on my local machine. Am I missing a step like aggregating files? 

  StreamWriter clear = new StreamWriter(dataOutputFile, false);
            clear.Close();
            Console.WriteLine("Cleared data output file " + dataOutputFile + "\n");


            TickHistoryTimeAndSalesExtractionRequest extractionRequest = new TickHistoryTimeAndSalesExtractionRequest {
                IdentifierList = InstrumentIdentifierList.Create(instrumentIdentifiers),
                Condition = reportTemplate.Condition,
                ContentFieldNames = { "Trade - Price", "Trade - Volume", "Quote - Bid Price", "Quote - Bid Size", "Quote - Ask Price", "Quote - Ask Size" }
            };


            extractionsContext.Options.AutomaticDecompression = true; //Decompress gzip to plain text
            RawExtractionResult extractionResult = extractionsContext.ExtractRaw(extractionRequest);


            Console.Write("{0:T} Extraction complete ... ", DateTime.Now);
            DebugPrintAndWaitForEnter("");


            sw = new StreamWriter(dataOutputFile, true);
            Console.WriteLine("==================================== DATA =====================================");
            DssStreamResponse streamResponse = extractionsContext.GetReadStream(extractionResult);
            using (StreamReader reader = new StreamReader(streamResponse.Stream)) {
                string line = reader.ReadLine();
                if (string.IsNullOrEmpty(line)) {
                    Console.WriteLine("WARNING: no data returned. Check your request dates.");
                    sw.WriteLine("WARNING: no data returned. Check your request dates.");
                } else {
                    //The first line is the list of field names:
                    sw.WriteLine(line);
                    Console.WriteLine(line);
                    //The remaining lines are the data:
                    //Variant 1: write all lines individually to file and console (and set a limit on number of lines)
                    for (int lineCount = 0; lineCount < maxDataLines && !reader.EndOfStream; lineCount++) {
                        line = reader.ReadLine();
                        sw.WriteLine(line);
                        Console.WriteLine(line);
                    }
                    //Variant 2: write all lines to either file or console (not both !):
                    //sw.WriteLine(line = reader.ReadToEnd());
                    //Console.WriteLine(line = reader.ReadToEnd());
                }
            }
            sw.Close();
            DebugPrintAndWaitForEnter("===============================================================================");
        }
tick-history-rest-apiextraction
icon clock
10 |1500

Up to 2 attachments (including images) can be used with a maximum of 512.0 KiB each and 1.0 MiB total.

Christiaan Meihsl avatar image Christiaan Meihsl

This code is from our .Net Tutorial 5. It was just meant as a demo, not as productized code. We discovered lately that native libraries are not designed to handle merged GZip files, and are not very robust, especially when decompressing data on the fly. Thus they might fail with large data sets. I am reviewing the code for .Net Tutorials 4 and 5, and as soon as I have something more robust I will make it available.

helen.ristov avatar image helen.ristov Christiaan Meihsl

Yes, I was using the demos to write a program to automate a daily extraction. I want to get rid of the debugging, etc later. Do you have a dictionary for the TRTH API?

Christiaan Meihsl avatar image Christiaan Meihsl helen.ristov

@helen.ristov, there is a data dictionary under the documentation tab. Is that what you are looking for ?

On the code sample topic I made good progress. If all goes well tomorrow I will publish new samples and update .Net tutorials 4 & 5 with more robust code to display or save the data. But first I want to test them with large data sets, and that takes time ...

Show more comments

4 Answers

Upvotes
Accepted
moragodkrit.chumsri_1 avatar image
7.6k ●15 ●6 ●9

@helen.ristov

-What is maxDataLines you set in your test application?

if it's the same value as our tutorial which is 1000000, it may not enought to print unpacked result. Some report can generate more than 1000000 lines of data in csv format. You have to increase the value.

-Can you provide more details about reportTemplate.condition you are testing? I would like to test the instrument with the same date range.

-Please you change ExtractionsContext.Options.AutomaticDecompression to false and then change the codes that your write the result to file to 

//Download the result
using (var response = ExtractionsContext.RawExtractionResultOperations.GetReadStream(result))
using (var fileStream = File.Create("savedExtraction.gzip"))
	response.Stream.CopyTo(fileStream);

Then you can unpack .gzip file and open savedExtraction.csv to compare number of lines from the unpacked data.

icon clock
10 |1500

Up to 2 attachments (including images) can be used with a maximum of 512.0 KiB each and 1.0 MiB total.

Christiaan Meihsl avatar image Christiaan Meihsl

Following this query, a new set of C# code samples was created and uploaded. Changes:

  • Samples 4 and 5 enhanced with better code to treat data on the fly.
  • Sample 5 now has code to save the compressed data file.
  • Note: there is a requirement for an additional library (the DLL is in the new code sample package).

.Net Tutorials 4 & 5 were updated accordingly.

Christiaan Meihsl avatar image Christiaan Meihsl

See here for a similar query.

Upvotes
helen.ristov avatar image
20 ●2 ●3 ●6

Ok, thanks. I will increase the size. I try to extract the data from midnight to 1am, and I only get 13s of history in the file. I'll try downloading to gzip. My report condition is below. 

 reportTemplate.Condition = new TickHistoryTimeAndSalesCondition {


                MessageTimeStampIn = TickHistoryTimeOptions.GmtUtc,
                QueryStartDate = new DateTimeOffset(2017, 07, 20, 0, 0, 0, TimeSpan.FromHours(0)),
                QueryEndDate = new DateTimeOffset(2017, 07, 20, 1, 0, 0, TimeSpan.FromHours(0)),
                ReportDateRangeType = ReportDateRangeType.Range,
                ExtractBy = TickHistoryExtractByMode.Ric,
                SortBy = TickHistorySort.SingleByRic,
                Preview = PreviewMode.None,
                DisplaySourceRIC = false
            };
icon clock
10 |1500

Up to 2 attachments (including images) can be used with a maximum of 512.0 KiB each and 1.0 MiB total.

Upvotes
helen.ristov avatar image
20 ●2 ●3 ●6

I just tried running the code you mentioned to save the file and I got a a small file again . Attached here. 

            using (var response = extractionsContext.RawExtractionResultOperations.GetReadStream(extractionResult))
            using (var fileStream = File.Create("E:\\HomeShare\\apitest.txt"))
                response.Stream.CopyTo(fileStream);

apitest.txt (14.6 KiB)
icon clock
10 |1500

Up to 2 attachments (including images) can be used with a maximum of 512.0 KiB each and 1.0 MiB total.

Upvotes
helen.ristov avatar image
20 ●2 ●3 ●6

I got it to work by extracting to gzip and setting the decompression option to false

icon clock
10 |1500

Up to 2 attachments (including images) can be used with a maximum of 512.0 KiB each and 1.0 MiB total.