time and sales extraction incomplete

Hello,
I'm trying to extract some time and sales data via an on demand raw extraction but I'm only getting a small subset of the expected results, whereas I'm able to get the full results when using the GUI. Is there any limitation when using the on demand requests ?
An excerpt of my request and the notes returned below:
request =
@{'ExtractionRequest': {'@odata.type': '#ThomsonReuters.Dss.Api.Extractions.ExtractionRequests.TickHistoryTimeAndSalesExtractionRequest',
'Condition': {'ApplyCorrectionsAndCancellations': True,
'DisplaySourceRIC': True,
'ExtractBy': 'Entity',
'MessageTimeStampIn': 'LocalExchangeTime',
'QueryEndDate': '2019-10-25T20:15:00',
'QueryStartDate': '2019-10-25T19:45:00',
'ReportDateRangeType': 'Range',
'SortBy': 'SingleByRic',
'TimeRangeMode': 'Inclusive'},
'ContentFieldNames': ['Quote - Bid Price',
'Quote - Bid Size',
'Quote - Ask Price',
'Quote - Ask Size',
'Trade - Bid Price',
'Trade - Bid Size',
'Trade - Ask Price',
'Trade - Ask Size'],
'IdentifierList': {'@odata.type': '#ThomsonReuters.Dss.Api.Extractions.ExtractionRequests.InstrumentIdentifierList',
'InstrumentIdentifiers': [{'Identifier': 'SPXWw011927250.U',
'IdentifierType': 'Ric'},
{'Identifier': 'SPXWw011927300.U', 'IdentifierType': 'Ric'},
{'Identifier': 'SPXWw011927350.U', 'IdentifierType': 'Ric'},
{'Identifier': 'SPXWw011927400.U', 'IdentifierType': 'Ric'},
{'Identifier': 'SPXWw011927450.U', 'IdentifierType': 'Ric'},
{'Identifier': 'SPXWw011927500.U', 'IdentifierType': 'Ric'},
{'Identifier': 'SPXWw011927550.U', 'IdentifierType': 'Ric'},
{'Identifier': 'SPXWw011927600.U', 'IdentifierType': 'Ric'},
{'Identifier': 'SPXWw011927650.U', 'IdentifierType': 'Ric'},
{'Identifier': 'SPXWw011927700.U', 'IdentifierType': 'Ric'},
{'Identifier': 'SPXWw011927750.U', 'IdentifierType': 'Ric'},
{'Identifier': 'SPXWw011927800.U', 'IdentifierType': 'Ric'},
{'Identifier': 'SPXWw011927850.U', 'IdentifierType': 'Ric'},
{'Identifier': 'SPXWw011927900.U', 'IdentifierType': 'Ric'},
{'Identifier': 'SPXWw011927950.U', 'IdentifierType': 'Ric'},
{'Identifier': 'SPXWw011928000.U', 'IdentifierType': 'Ric'},
{'Identifier': 'SPXWw011928050.U', 'IdentifierType': 'Ric'},
{'Identifier': 'SPXWw011928100.U', 'IdentifierType': 'Ric'},
{'Identifier': 'SPXWw011928150.U', 'IdentifierType': 'Ric'},
{'Identifier': 'SPXWw011928200.U', 'IdentifierType': 'Ric'},],
'UseUserPreferencesForValidationOptions': True}}}
notes =
'Extraction Services Version 13.2.41189 (85522a4c8b22), Built Nov 11 2019 22:20:39',
'User ID: ',
'Extraction ID: 2000000113205251',
'Schedule: 0x06dfaee189312bd0 (ID = 0x0000000000000000)',
'Input List (83 items): (ID = 0x06dfaee189312bd0) Created: 11/20/2019 08:31:46 Last Modified: 11/20/2019 08:31:46',
'Report Template (8 fields): _OnD_0x06dfaee189312bd0 (ID = 0x06dfaee189b12bd0) Created: 11/20/2019 08:29:34 Last Modified: 11/20/2019 08:29:34',
'Schedule dispatched via message queue (0x06dfaee189312bd0), Data source identifier (93ECB38A43154CAD8A1154D930EDB5C2)',
'Schedule Time: 11/20/2019 08:29:35',
'Processing started at 11/20/2019 08:29:36',
'Processing completed successfully at 11/20/2019 08:31:46',
'Extraction finished at 11/20/2019 08:31:46 UTC, with servers: tm02n02, TRTH (71.93 secs)',
'Historical Instrument <RIC,SPXWw011927250.U> expanded to 1 RIC: SPXWw011927250.U.',
'Historical Instrument <RIC,SPXWw011927300.U> expanded to 1 RIC: SPXWw011927300.U.',
'Historical Instrument <RIC,SPXWw011927350.U> expanded to 1 RIC: SPXWw011927350.U.',
'Historical Instrument <RIC,SPXWw011927400.U> expanded to 1 RIC: SPXWw011927400.U.',
'Historical Instrument <RIC,SPXWw011927450.U> expanded to 1 RIC: SPXWw011927450.U.',
'Historical Instrument <RIC,SPXWw011927500.U> expanded to 1 RIC: SPXWw011927500.U.',
'Historical Instrument <RIC,SPXWw011927550.U> expanded to 1 RIC: SPXWw011927550.U.',
'Historical Instrument <RIC,SPXWw011927600.U> expanded to 1 RIC: SPXWw011927600.U.',
'Historical Instrument <RIC,SPXWw011927650.U> expanded to 1 RIC: SPXWw011927650.U.',
'Historical Instrument <RIC,SPXWw011927700.U> expanded to 1 RIC: SPXWw011927700.U.',
'Historical Instrument <RIC,SPXWw011927750.U> expanded to 1 RIC: SPXWw011927750.U.',
'Total instruments after instrument expansion = 83',
'Range Query from 2019-10-25T19:45:00.000 to 2019-10-25T20:15:00.000 (UTC)',
'Quota Message: INFO: Tick History Options Quota Count Before Extraction: 47; Instruments Approved for Extraction: 1; Tick History Options Quota Count After Extraction: 47, 78.3333333333333% of Limit; Tick History Options Quota Limit: 60',
'Manifest: #RIC,Domain,Start,End,Status,Count',
'Manifest: SPXWk011930200.U,Market Price,2019-10-25T19:45:00.084016861Z,2019-10-25T20:14:59.197502072Z,Active,678',
'Manifest: SPXWk011930250.U,Market Price,2019-10-25T19:45:00.416646846Z,2019-10-25T20:14:59.528750063Z,Active,619',
'Manifest: SPXWk011930300.U,Market Price,2019-10-25T19:45:00.751666355Z,2019-10-25T20:14:59.860912512Z,Active,690',
'Manifest: SPXWk011930350.U,Market Price,2019-10-25T19:45:00.083406164Z,2019-10-25T20:14:59.201640345Z,Active,639',
'Manifest: SPXWk011930400.U,Market Price,2019-10-25T19:45:00.418477881Z,2019-10-25T20:14:59.529562257Z,Active,659',
'Manifest: SPXWk011930450.U,Market Price,2019-10-25T19:45:00.751679571Z,2019-10-25T20:14:59.861306049Z,Active,650',
'Manifest: SPXWk011930500.U,Market Price,2019-10-25T19:45:00.084726519Z,2019-10-25T20:14:59.200826455Z,Active,674',
'Manifest: SPXWk011930550.U,Market Price,2019-10-25T19:45:01.442438658Z,2019-10-25T20:14:58.534407536Z,Active,658',
'Manifest: SPXWk011930600.U,Market Price,2019-10-25T19:45:01.793153265Z,2019-10-25T20:14:58.863615075Z,Active,631',
'Manifest: SPXWk011930650.U,Market Price,2019-10-25T19:45:01.110501067Z,2019-10-25T20:14:58.201620846Z,Active,616',
'Manifest: SPXWk011930700.U,Market Price,2019-10-25T19:45:01.442281722Z,2019-10-25T20:14:58.534447688Z,Active,600',
'Manifest: SPXWk011930750.U,Market Price,2019-10-25T19:45:01.441433928Z,2019-10-25T20:14:58.531512914Z,Active,594',
'Manifest: SPXWk011930800.U,Market Price,2019-10-25T19:45:01.108691063Z,2019-10-25T20:14:58.200188611Z,Active,604',
'Manifest: SPXWk011930850.U,Market Price,2019-10-25T19:45:01.443025611Z,2019-10-25T20:14:58.533451126Z,Active,607',
'Manifest: SPXWk011930900.U,Market Price,2019-10-25T19:45:03.843663254Z,2019-10-25T20:14:58.865526983Z,Active,550',
'Manifest: SPXWk011930950.U,Market Price,2019-10-25T19:45:01.111667870Z,2019-10-25T20:14:58.200801172Z,Active,514',
'Manifest: SPXWk011931000.U,Market Price,2019-10-25T19:45:01.107683661Z,2019-10-25T20:14:58.196788052Z,Active,473',
'Manifest: SPXWk011931050.U,Market Price,2019-10-25T19:45:00.086552971Z,2019-10-25T20:14:31.029838257Z,Active,493',
'Manifest: SPXWk011931100.U,Market Price,2019-10-25T19:45:44.034855495Z,2019-10-25T20:14:56.525360521Z,Active,476',
Best Answer
-
We have Python samples described here that download compressed data, you might want to compare their code to yours. You might also find this article useful in this context.
0
Answers
-
You do not mention exactly how you are retrieving the data, but if you are decompressing the extracted data on the fly you can end up with truncated results. On that topic see this advisory.
0 -
Thanks Christiaan
I'm retrieving via http requests in python using the requests module. I'm unable to desactivate the decompression and I get the same truncated results in my browser with the request's url:
0 -
It works with aws. Thanks a lot
0 -
ayoub.serroukheidrissi, great, glad to know it is solved. And by using AWS you get faster downloads as well
0
Categories
- All Categories
- 3 Polls
- 6 AHS
- 37 Alpha
- 167 App Studio
- 6 Block Chain
- 4 Bot Platform
- 18 Connected Risk APIs
- 47 Data Fusion
- 34 Data Model Discovery
- 705 Datastream
- 1.5K DSS
- 633 Eikon COM
- 5.2K Eikon Data APIs
- 14 Electronic Trading
- 1 Generic FIX
- 7 Local Bank Node API
- 6 Trading API
- 3K Elektron
- 1.5K EMA
- 259 ETA
- 569 WebSocket API
- 40 FX Venues
- 16 FX Market Data
- 1 FX Post Trade
- 1 FX Trading - Matching
- 12 FX Trading – RFQ Maker
- 5 Intelligent Tagging
- 2 Legal One
- 25 Messenger Bot
- 4 Messenger Side by Side
- 9 ONESOURCE
- 7 Indirect Tax
- 60 Open Calais
- 284 Open PermID
- 47 Entity Search
- 2 Org ID
- 1 PAM
- PAM - Logging
- 6 Product Insight
- Project Tracking
- ProView
- ProView Internal
- 24 RDMS
- 2.2K Refinitiv Data Platform
- 879 Refinitiv Data Platform Libraries
- 5 LSEG Due Diligence
- 1 LSEG Due Diligence Portal API
- 4 Refinitiv Due Dilligence Centre
- Rose's Space
- 1.2K Screening
- 18 Qual-ID API
- 13 Screening Deployed
- 23 Screening Online
- 12 World-Check Customer Risk Screener
- 1K World-Check One
- 46 World-Check One Zero Footprint
- 45 Side by Side Integration API
- 2 Test Space
- 3 Thomson One Smart
- 10 TR Knowledge Graph
- 151 Transactions
- 143 REDI API
- 1.8K TREP APIs
- 4 CAT
- 27 DACS Station
- 123 Open DACS
- 1.1K RFA
- 108 UPA
- 196 TREP Infrastructure
- 232 TRKD
- 919 TRTH
- 5 Velocity Analytics
- 9 Wealth Management Web Services
- 103 Workspace SDK
- 11 Element Framework
- 5 Grid
- 19 World-Check Data File
- 1 Yield Book Analytics
- 48 中文论坛