time and sales extraction incomplete

ayoub.serroukheidrissi

Hello,

I'm trying to extract some time and sales data via an on demand raw extraction but I'm only getting a small subset of the expected results, whereas I'm able to get the full results when using the GUI. Is there any limitation when using the on demand requests ?

An excerpt of my request and the notes returned below:

request =

@{'ExtractionRequest': {'@odata.type': '#ThomsonReuters.Dss.Api.Extractions.ExtractionRequests.TickHistoryTimeAndSalesExtractionRequest',

'Condition': {'ApplyCorrectionsAndCancellations': True,

'DisplaySourceRIC': True,

'ExtractBy': 'Entity',

'MessageTimeStampIn': 'LocalExchangeTime',

'QueryEndDate': '2019-10-25T20:15:00',

'QueryStartDate': '2019-10-25T19:45:00',

'ReportDateRangeType': 'Range',

'SortBy': 'SingleByRic',

'TimeRangeMode': 'Inclusive'},

'ContentFieldNames': ['Quote - Bid Price',

'Quote - Bid Size',

'Quote - Ask Price',

'Quote - Ask Size',

'Trade - Bid Price',

'Trade - Bid Size',

'Trade - Ask Price',

'Trade - Ask Size'],

'IdentifierList': {'@odata.type': '#ThomsonReuters.Dss.Api.Extractions.ExtractionRequests.InstrumentIdentifierList',

'InstrumentIdentifiers': [{'Identifier': 'SPXWw011927250.U',

'IdentifierType': 'Ric'},

{'Identifier': 'SPXWw011927300.U', 'IdentifierType': 'Ric'},

{'Identifier': 'SPXWw011927350.U', 'IdentifierType': 'Ric'},

{'Identifier': 'SPXWw011927400.U', 'IdentifierType': 'Ric'},

{'Identifier': 'SPXWw011927450.U', 'IdentifierType': 'Ric'},

{'Identifier': 'SPXWw011927500.U', 'IdentifierType': 'Ric'},

{'Identifier': 'SPXWw011927550.U', 'IdentifierType': 'Ric'},

{'Identifier': 'SPXWw011927600.U', 'IdentifierType': 'Ric'},

{'Identifier': 'SPXWw011927650.U', 'IdentifierType': 'Ric'},

{'Identifier': 'SPXWw011927700.U', 'IdentifierType': 'Ric'},

{'Identifier': 'SPXWw011927750.U', 'IdentifierType': 'Ric'},

{'Identifier': 'SPXWw011927800.U', 'IdentifierType': 'Ric'},

{'Identifier': 'SPXWw011927850.U', 'IdentifierType': 'Ric'},

{'Identifier': 'SPXWw011927900.U', 'IdentifierType': 'Ric'},

{'Identifier': 'SPXWw011927950.U', 'IdentifierType': 'Ric'},

{'Identifier': 'SPXWw011928000.U', 'IdentifierType': 'Ric'},

{'Identifier': 'SPXWw011928050.U', 'IdentifierType': 'Ric'},

{'Identifier': 'SPXWw011928100.U', 'IdentifierType': 'Ric'},

{'Identifier': 'SPXWw011928150.U', 'IdentifierType': 'Ric'},

{'Identifier': 'SPXWw011928200.U', 'IdentifierType': 'Ric'},],

'UseUserPreferencesForValidationOptions': True}}}

notes =

'Extraction Services Version 13.2.41189 (85522a4c8b22), Built Nov 11 2019 22:20:39',

'User ID: ',

'Extraction ID: 2000000113205251',

'Schedule: 0x06dfaee189312bd0 (ID = 0x0000000000000000)',

'Input List (83 items): (ID = 0x06dfaee189312bd0) Created: 11/20/2019 08:31:46 Last Modified: 11/20/2019 08:31:46',

'Report Template (8 fields): _OnD_0x06dfaee189312bd0 (ID = 0x06dfaee189b12bd0) Created: 11/20/2019 08:29:34 Last Modified: 11/20/2019 08:29:34',

'Schedule dispatched via message queue (0x06dfaee189312bd0), Data source identifier (93ECB38A43154CAD8A1154D930EDB5C2)',

'Schedule Time: 11/20/2019 08:29:35',

'Processing started at 11/20/2019 08:29:36',

'Processing completed successfully at 11/20/2019 08:31:46',

'Extraction finished at 11/20/2019 08:31:46 UTC, with servers: tm02n02, TRTH (71.93 secs)',

'Historical Instrument <RIC,SPXWw011927250.U> expanded to 1 RIC: SPXWw011927250.U.',

'Historical Instrument <RIC,SPXWw011927300.U> expanded to 1 RIC: SPXWw011927300.U.',

'Historical Instrument <RIC,SPXWw011927350.U> expanded to 1 RIC: SPXWw011927350.U.',

'Historical Instrument <RIC,SPXWw011927400.U> expanded to 1 RIC: SPXWw011927400.U.',

'Historical Instrument <RIC,SPXWw011927450.U> expanded to 1 RIC: SPXWw011927450.U.',

'Historical Instrument <RIC,SPXWw011927500.U> expanded to 1 RIC: SPXWw011927500.U.',

'Historical Instrument <RIC,SPXWw011927550.U> expanded to 1 RIC: SPXWw011927550.U.',

'Historical Instrument <RIC,SPXWw011927600.U> expanded to 1 RIC: SPXWw011927600.U.',

'Historical Instrument <RIC,SPXWw011927650.U> expanded to 1 RIC: SPXWw011927650.U.',

'Historical Instrument <RIC,SPXWw011927700.U> expanded to 1 RIC: SPXWw011927700.U.',

'Historical Instrument <RIC,SPXWw011927750.U> expanded to 1 RIC: SPXWw011927750.U.',

'Total instruments after instrument expansion = 83',

'Range Query from 2019-10-25T19:45:00.000 to 2019-10-25T20:15:00.000 (UTC)',

'Quota Message: INFO: Tick History Options Quota Count Before Extraction: 47; Instruments Approved for Extraction: 1; Tick History Options Quota Count After Extraction: 47, 78.3333333333333% of Limit; Tick History Options Quota Limit: 60',

'Manifest: #RIC,Domain,Start,End,Status,Count',

'Manifest: SPXWk011930200.U,Market Price,2019-10-25T19:45:00.084016861Z,2019-10-25T20:14:59.197502072Z,Active,678',

'Manifest: SPXWk011930250.U,Market Price,2019-10-25T19:45:00.416646846Z,2019-10-25T20:14:59.528750063Z,Active,619',

'Manifest: SPXWk011930300.U,Market Price,2019-10-25T19:45:00.751666355Z,2019-10-25T20:14:59.860912512Z,Active,690',

'Manifest: SPXWk011930350.U,Market Price,2019-10-25T19:45:00.083406164Z,2019-10-25T20:14:59.201640345Z,Active,639',

'Manifest: SPXWk011930400.U,Market Price,2019-10-25T19:45:00.418477881Z,2019-10-25T20:14:59.529562257Z,Active,659',

'Manifest: SPXWk011930450.U,Market Price,2019-10-25T19:45:00.751679571Z,2019-10-25T20:14:59.861306049Z,Active,650',

'Manifest: SPXWk011930500.U,Market Price,2019-10-25T19:45:00.084726519Z,2019-10-25T20:14:59.200826455Z,Active,674',

'Manifest: SPXWk011930550.U,Market Price,2019-10-25T19:45:01.442438658Z,2019-10-25T20:14:58.534407536Z,Active,658',

'Manifest: SPXWk011930600.U,Market Price,2019-10-25T19:45:01.793153265Z,2019-10-25T20:14:58.863615075Z,Active,631',

'Manifest: SPXWk011930650.U,Market Price,2019-10-25T19:45:01.110501067Z,2019-10-25T20:14:58.201620846Z,Active,616',

'Manifest: SPXWk011930700.U,Market Price,2019-10-25T19:45:01.442281722Z,2019-10-25T20:14:58.534447688Z,Active,600',

'Manifest: SPXWk011930750.U,Market Price,2019-10-25T19:45:01.441433928Z,2019-10-25T20:14:58.531512914Z,Active,594',

'Manifest: SPXWk011930800.U,Market Price,2019-10-25T19:45:01.108691063Z,2019-10-25T20:14:58.200188611Z,Active,604',

'Manifest: SPXWk011930850.U,Market Price,2019-10-25T19:45:01.443025611Z,2019-10-25T20:14:58.533451126Z,Active,607',

'Manifest: SPXWk011930900.U,Market Price,2019-10-25T19:45:03.843663254Z,2019-10-25T20:14:58.865526983Z,Active,550',

'Manifest: SPXWk011930950.U,Market Price,2019-10-25T19:45:01.111667870Z,2019-10-25T20:14:58.200801172Z,Active,514',

'Manifest: SPXWk011931000.U,Market Price,2019-10-25T19:45:01.107683661Z,2019-10-25T20:14:58.196788052Z,Active,473',

'Manifest: SPXWk011931050.U,Market Price,2019-10-25T19:45:00.086552971Z,2019-10-25T20:14:31.029838257Z,Active,493',

'Manifest: SPXWk011931100.U,Market Price,2019-10-25T19:45:44.034855495Z,2019-10-25T20:14:56.525360521Z,Active,476',

Christiaan Meihsl

ayoub.serroukheidrissi,

We have Python samples described here that download compressed data, you might want to compare their code to yours. You might also find this article useful in this context.

Christiaan Meihsl

ayoub.serroukheidrissi,

You do not mention exactly how you are retrieving the data, but if you are decompressing the extracted data on the fly you can end up with truncated results. On that topic see this advisory.

ayoub.serroukheidrissi

Thanks Christiaan

I'm retrieving via http requests in python using the requests module. I'm unable to desactivate the decompression and I get the same truncated results in my browser with the request's url:

https://hosted.datascopeapi.reuters.com/RestApi/v1/Extractions/RawExtractionResults('0x06dfb21d04412bda')/$value

ayoub.serroukheidrissi

https://community.developers.refinitiv.com/discussion/comment/52047#Comment_52047

It works with aws. Thanks a lot

Christiaan Meihsl

https://community.developers.refinitiv.com/discussion/comment/52018#Comment_52018

ayoub.serroukheidrissi, great, glad to know it is solved. And by using AWS you get faster downloads as well