Discover Refinitiv
MyRefinitiv Refinitiv Perspectives Careers
Created with Sketch.
All APIs Questions & Answers  Register |  Login
Ask a question
  • Questions
  • Tags
  • Badges
  • Unanswered
Search:
  • Home /
  • TRTH /
avatar image
Question by helen.ristov · Jul 26, 2017 at 02:48 PM · trth apiextraction

Extracting TRTH Time and Sales Data to a File

Hello, When I try to extract the Time and Sales Data to a file I only get partial data. I've attached a snippet of my code for the extraction request which writes to a file on my local machine. Am I missing a step like aggregating files?

  StreamWriter clear = new StreamWriter(dataOutputFile, false);
            clear.Close();
            Console.WriteLine("Cleared data output file " + dataOutputFile + "\n");


            TickHistoryTimeAndSalesExtractionRequest extractionRequest = new TickHistoryTimeAndSalesExtractionRequest {
                IdentifierList = InstrumentIdentifierList.Create(instrumentIdentifiers),
                Condition = reportTemplate.Condition,
                ContentFieldNames = { "Trade - Price", "Trade - Volume", "Quote - Bid Price", "Quote - Bid Size", "Quote - Ask Price", "Quote - Ask Size" }
            };


            extractionsContext.Options.AutomaticDecompression = true; //Decompress gzip to plain text
            RawExtractionResult extractionResult = extractionsContext.ExtractRaw(extractionRequest);


            Console.Write("{0:T} Extraction complete ... ", DateTime.Now);
            DebugPrintAndWaitForEnter("");


            sw = new StreamWriter(dataOutputFile, true);
            Console.WriteLine("==================================== DATA =====================================");
            DssStreamResponse streamResponse = extractionsContext.GetReadStream(extractionResult);
            using (StreamReader reader = new StreamReader(streamResponse.Stream)) {
                string line = reader.ReadLine();
                if (string.IsNullOrEmpty(line)) {
                    Console.WriteLine("WARNING: no data returned. Check your request dates.");
                    sw.WriteLine("WARNING: no data returned. Check your request dates.");
                } else {
                    //The first line is the list of field names:
                    sw.WriteLine(line);
                    Console.WriteLine(line);
                    //The remaining lines are the data:
                    //Variant 1: write all lines individually to file and console (and set a limit on number of lines)
                    for (int lineCount = 0; lineCount < maxDataLines && !reader.EndOfStream; lineCount++) {
                        line = reader.ReadLine();
                        sw.WriteLine(line);
                        Console.WriteLine(line);
                    }
                    //Variant 2: write all lines to either file or console (not both !):
                    //sw.WriteLine(line = reader.ReadToEnd());
                    //Console.WriteLine(line = reader.ReadToEnd());
                }
            }
            sw.Close();
            DebugPrintAndWaitForEnter("===============================================================================");
        }

People who like this

0 Show 5
Comment
10 |1500 characters needed characters left characters exceeded
▼
  • Viewable by all users
  • Viewable by moderators
  • Viewable by moderators and the original poster
  • Advanced visibility
Viewable by all users

Up to 2 attachments (including images) can be used with a maximum of 512.0 KiB each and 1.0 MiB total.

avatar image
REFINITIV
Christiaan Meihsl ♦♦ · Jul 27, 2017 at 04:50 AM 0
Share

This code is from our .Net Tutorial 5. It was just meant as a demo, not as productized code. We discovered lately that native libraries are not designed to handle merged GZip files, and are not very robust, especially when decompressing data on the fly. Thus they might fail with large data sets. I am reviewing the code for .Net Tutorials 4 and 5, and as soon as I have something more robust I will make it available.

avatar image
helen.ristov Christiaan Meihsl ♦♦ · Jul 27, 2017 at 12:02 PM 0
Share

Yes, I was using the demos to write a program to automate a daily extraction. I want to get rid of the debugging, etc later. Do you have a dictionary for the TRTH API?

avatar image
REFINITIV
Christiaan Meihsl ♦♦ helen.ristov · Jul 27, 2017 at 12:13 PM 0
Share

@helen.ristov, there is a data dictionary under the documentation tab. Is that what you are looking for ?

On the code sample topic I made good progress. If all goes well tomorrow I will publish new samples and update .Net tutorials 4 & 5 with more robust code to display or save the data. But first I want to test them with large data sets, and that takes time ...

4 Replies

  • Sort: 
avatar image
REFINITIV
Best Answer
Answer by moragodkrit.chumsri_1 · Jul 27, 2017 at 01:35 AM

@helen.ristov

-What is maxDataLines you set in your test application?

if it's the same value as our tutorial which is 1000000, it may not enought to print unpacked result. Some report can generate more than 1000000 lines of data in csv format. You have to increase the value.

-Can you provide more details about reportTemplate.condition you are testing? I would like to test the instrument with the same date range.

-Please you change ExtractionsContext.Options.AutomaticDecompression to false and then change the codes that your write the result to file to

//Download the result
using (var response = ExtractionsContext.RawExtractionResultOperations.GetReadStream(result))
using (var fileStream = File.Create("savedExtraction.gzip"))
	response.Stream.CopyTo(fileStream);

Then you can unpack .gzip file and open savedExtraction.csv to compare number of lines from the unpacked data.

Comment

People who like this

0 Show 2 · Share
10 |1500 characters needed characters left characters exceeded
▼
  • Viewable by all users
  • Viewable by moderators
  • Viewable by moderators and the original poster
  • Advanced visibility
Viewable by all users

Up to 2 attachments (including images) can be used with a maximum of 512.0 KiB each and 1.0 MiB total.

avatar image
REFINITIV
Christiaan Meihsl ♦♦ · Jul 28, 2017 at 11:40 AM 0
Share

Following this query, a new set of C# code samples was created and uploaded. Changes:

  • Samples 4 and 5 enhanced with better code to treat data on the fly.
  • Sample 5 now has code to save the compressed data file.
  • Note: there is a requirement for an additional library (the DLL is in the new code sample package).

.Net Tutorials 4 & 5 were updated accordingly.

avatar image
REFINITIV
Christiaan Meihsl ♦♦ · May 24, 2018 at 09:07 AM 0
Share

See here for a similar query.

avatar image
Answer by helen.ristov · Jul 27, 2017 at 12:07 PM

Ok, thanks. I will increase the size. I try to extract the data from midnight to 1am, and I only get 13s of history in the file. I'll try downloading to gzip. My report condition is below.

 reportTemplate.Condition = new TickHistoryTimeAndSalesCondition {


                MessageTimeStampIn = TickHistoryTimeOptions.GmtUtc,
                QueryStartDate = new DateTimeOffset(2017, 07, 20, 0, 0, 0, TimeSpan.FromHours(0)),
                QueryEndDate = new DateTimeOffset(2017, 07, 20, 1, 0, 0, TimeSpan.FromHours(0)),
                ReportDateRangeType = ReportDateRangeType.Range,
                ExtractBy = TickHistoryExtractByMode.Ric,
                SortBy = TickHistorySort.SingleByRic,
                Preview = PreviewMode.None,
                DisplaySourceRIC = false
            };
Comment

People who like this

0 Show 0 · Share
10 |1500 characters needed characters left characters exceeded
▼
  • Viewable by all users
  • Viewable by moderators
  • Viewable by moderators and the original poster
  • Advanced visibility
Viewable by all users

Up to 2 attachments (including images) can be used with a maximum of 512.0 KiB each and 1.0 MiB total.

avatar image
Answer by helen.ristov · Jul 27, 2017 at 01:12 PM

I just tried running the code you mentioned to save the file and I got a a small file again . Attached here.

            using (var response = extractionsContext.RawExtractionResultOperations.GetReadStream(extractionResult))
            using (var fileStream = File.Create("E:\\HomeShare\\apitest.txt"))
                response.Stream.CopyTo(fileStream);

apitest.txt (14.6 KiB)
Comment

People who like this

0 Show 0 · Share
10 |1500 characters needed characters left characters exceeded
▼
  • Viewable by all users
  • Viewable by moderators
  • Viewable by moderators and the original poster
  • Advanced visibility
Viewable by all users

Up to 2 attachments (including images) can be used with a maximum of 512.0 KiB each and 1.0 MiB total.

avatar image
Answer by helen.ristov · Jul 27, 2017 at 02:57 PM

I got it to work by extracting to gzip and setting the decompression option to false

Comment

People who like this

0 Show 0 · Share
10 |1500 characters needed characters left characters exceeded
▼
  • Viewable by all users
  • Viewable by moderators
  • Viewable by moderators and the original poster
  • Advanced visibility
Viewable by all users

Up to 2 attachments (including images) can be used with a maximum of 512.0 KiB each and 1.0 MiB total.

Watch this question

Add to watch list
Add to your watch list to receive emailed updates for this question. Too many emails? Change your settings >
9 People are following this question.

Related Questions

Dates fields​

EOD Request does not return Open, High and Low. Also Intraday has similar problem

Is there any other way to identify RICs for Fix Income instruments via TRTH v2 API?

No price data for delisted

No extractions associated

  • Feedback
  • Copyright
  • Cookie Policy
  • Privacy Statement
  • Terms of Use
  • Careers
  • Anonymous
  • Sign in
  • Create
  • Ask a question
  • Spaces
  • Alpha
  • App Studio
  • Block Chain
  • Bot Platform
  • Calais
  • Connected Risk APIs
  • DSS
  • Data Fusion
  • Data Model Discovery
  • Datastream
  • Eikon COM
  • Eikon Data APIs
  • Electronic Trading
    • Generic FIX
    • Local Bank Node API
    • Trading API
  • Elektron
    • EMA
    • ETA
    • WebSocket API
  • Legal One
  • Messenger Bot
  • Messenger Side by Side
  • ONESOURCE
    • Indirect Tax
  • Open PermID
    • Entity Search
  • Org ID
  • PAM
    • PAM - Logging
  • ProView
  • ProView Internal
  • Product Insight
  • Project Tracking
  • Refinitiv Data Platform
    • Refinitiv Data Platform Libraries
  • Rose's Space
  • Screening
    • Qual-ID API
    • Screening Deployed
    • Screening Online
    • World-Check One
    • World-Check One Zero Footprint
  • Side by Side Integration API
  • TR Knowledge Graph
  • TREP APIs
    • CAT
    • DACS Station
    • Open DACS
    • RFA
    • UPA
  • TREP Infrastructure
  • TRIT
  • TRKD
  • TRTH
  • Thomson One Smart
  • Transactions
    • REDI API
  • Velocity Analytics
  • Wealth Management Web Services
  • World-Check Data File
  • Explore
  • Tags
  • Questions
  • Badges