Tick History Raw Extraction result does not retrieve all the rows; the manifests are always fine ...

...for all jobs, but not all rows are available when I extract the results for the finished jobs.

Tick History Raw Extraction result does not retrieve all the rows; the manifests are always fine for all jobs, but not all rows are available when I extract the results for the finished jobs.

I am using Python aiohttp package to execute ClientSession.request() method.

Then we parse all lines of json content which we received. The results are consistent between Chrome and my Python application.

https://selectapi.datascope.refinitiv.com/RestApi/v1/Extractions/RawExtractionResults('0x079b2cbf54ecd44e')/$value

Below extraction result; we see only 2 of 4 rows.

Extraction Services Version 15.0.42358 (01a7f7ea050d), Built May 20 2021 18:20:45

User ID: 9028636

Extraction ID: 2000000274338174

Schedule: 0x079ae3f6c06cd2d6 (ID = 0x0000000000000000)

Input List (2 items): (ID = 0x079ae3f6c06cd2d6) Created: 2021-06-24 19:21:56 Last Modified: 2021-06-24 19:21:56

Report Template: _OnD_0x079ae3f6c06cd2d6 (ID = 0x079ae3f6c08cd2d6) Created: 2021-06-24 19:20:53 Last Modified: 2021-06-24 19:20:53

Schedule dispatched via message queue (0x079ae3f6c06cd2d6), Data source identifier (A665CFE4E0C34D93BE691CF39752B49D)

Schedule Time: 2021-06-24 19:20:53

Processing started at 2021-06-24 19:20:53

Processing completed successfully at 2021-06-24 19:21:57

Extraction finished at 2021-06-24 17:21:57 UTC, with servers: tm01n01, TRTH (41.752 secs)

Instrument <RIC,SPXWf252132750.U> expanded to 1 RIC: SPXWf252132750.U.

Instrument <RIC,SPXWc312228500.U> expanded to 1 RIC: SPXWc312228500.U.

Total instruments after instrument expansion = 2

Quota Message: INFO: Tick History Options Quota Count Before Extraction: 9; Instruments Approved for Extraction: 1; Tick History Options Quota Count After Extraction: 9, 0.428571428571429% of Limit; Tick History Options Quota Limit: 2100

Manifest: #RIC,Domain,Start,End,Status,Count

Manifest: SPXWc312228500.U,Market Price,2021-06-23T06:45:44.690108670Z,2021-06-23T10:30:41.362530994Z,Active,2

Manifest: SPXWf252132750.U,Market Price,2021-06-23T06:46:14.906572283Z,2021-06-23T10:31:11.921941834Z,Active,2

19:22:25.007065000 [Info ] [app.refinitiv_client] 6: TX GET https://transfer01:10180/RestApi/v1/Extractions/RawExtractionResults('0x079ae3f6c06cd2d6')/$value

19:22:25.449803000 [Info ] [app.refinitiv_client] all response : {'_body': None,

'_cache': {'headers': <CIMultiDictProxy('Cache-Control': 'no-cache', 'Pragma': 'no-cache', 'Content-Length': '440', 'Content-Type': 'text/plain', 'Content-Encoding': 'gzip', 'Expires': '-1', 'Accept-Ranges': 'bytes', 'Server': 'Microsoft-IIS/10.0', 'BeginRequestTime': '17:22:25.0514990', 'BeginRequestDate': '2021-06-24', 'X-Request-Execution-Correlation-Id': 'CiD/9028636/AAAAAA.079b2cc3246cd44e/RA', 'X-App-Id': 'Custom.RestApi', 'X-App-Version': '15.0.155.64', 'CPUUtilization': '3.048618', 'RequestsPerSec': '0', 'W3WP-PrivateBytes': '1214820', 'Date': 'Thu, 24 Jun 2021 17:22:25 GMT')>,

'url': URL('https://transfer01:10180/RestApi/v1/Extractions/RawExtractionResults('0x079ae3f6c06cd2d6')/$value')},

'_closed': True,

'_connection': None,

'_continue': None,

'_headers': <CIMultiDictProxy('Cache-Control': 'no-cache', 'Pragma': 'no-cache', 'Content-Length': '440', 'Content-Type': 'text/plain', 'Content-Encoding': 'gzip', 'Expires': '-1', 'Accept-Ranges': 'bytes', 'Server': 'Microsoft-IIS/10.0', 'BeginRequestTime': '17:22:25.0514990', 'BeginRequestDate': '2021-06-24', 'X-Request-Execution-Correlation-Id': 'CiD/9028636/AAAAAA.079b2cc3246cd44e/RA', 'X-App-Id': 'Custom.RestApi', 'X-App-Version': '15.0.155.64', 'CPUUtilization': '3.048618', 'RequestsPerSec': '0', 'W3WP-PrivateBytes': '1214820', 'Date': 'Thu, 24 Jun 2021 17:22:25 GMT')>,

'_history': (),

'_loop': <_UnixSelectorEventLoop running=True closed=False debug=False>,

'_protocol': <aiohttp.client_proto.ResponseHandler object at 0x7f0a0d370730>,

'_raw_headers': ((b'Cache-Control', b'no-cache'),

(b'Pragma', b'no-cache'),

(b'Content-Length', b'440'),

(b'Content-Type', b'text/plain'),

(b'Content-Encoding', b'gzip'),

(b'Expires', b'-1'),

(b'Accept-Ranges', b'bytes'),

(b'Server', b'Microsoft-IIS/10.0'),

(b'BeginRequestTime', b'17:22:25.0514990'),

(b'BeginRequestDate', b'2021-06-24'),

(b'X-Request-Execution-Correlation-Id',

b'CiD/9028636/AAAAAA.079b2cc3246cd44e/RA'),

(b'X-App-Id', b'Custom.RestApi'),

(b'X-App-Version', b'15.0.155.64'),

(b'CPUUtilization', b'3.048618'),

(b'RequestsPerSec', b'0'),

(b'W3WP-PrivateBytes', b'1214820'),

(b'Date', b'Thu, 24 Jun 2021 17:22:25 GMT')),

'_real_url': URL('https://transfer01:10180/RestApi/v1/Extractions/RawExtractionResults('0x079ae3f6c06cd2d6')/$value'),

'_request_info': RequestInfo(url=URL('https://transfer01:10180/RestApi/v1/Extractions/RawExtractionResults('0x079ae3f6c06cd2d6')/$value'), method='GET', headers=<CIMultiDictProxy('Host': 'transfer01:10180', 'Authorization': 'Token ', 'Prefer': 'odata.maxpagesize=20000', 'Accept': '*/*', 'Accept-Encoding': 'gzip, deflate', 'User-Agent': 'Python/3.6 aiohttp/3.7.3')>, real_url=URL('https://transfer01:10180/RestApi/v1/Extractions/RawExtractionResults('0x079ae3f6c06cd2d6')/$value')),

'_session': None,

'_timer': <aiohttp.helpers.TimerContext object at 0x7f0a12a30e10>,

'_traces': [],

'_url': URL('https://transfer01:10180/RestApi/v1/Extractions/RawExtractionResults('0x079ae3f6c06cd2d6')/$value'),

'_writer': None,

'content': <StreamReader 426 bytes eof>,

'cookies': <SimpleCookie: >,

'method': 'GET',

'reason': 'OK',

'status': 200,

'version': HttpVersion(major=1, minor=1)}

19:22:25.450062000 [Info ] [app.refinitiv_client] 6: RX 200 OK

19:22:25.450225000 [Info ] [__main__] raw_template header: b'#RIC,Domain,Date-Time,GMT Offset,Type,MsgClass/FID number,UpdateType/Action,FID Name,FID Value,FID Enum String,PE Code,Template Number,Key/Msg Sequence Number,Number of FIDs\n'

19:22:25.450816000 [Info ] [__main__] raw template row: b'SPXWc312228500.U,,2021-06-23T06:45:44.690108670Z,,FID,64,,OPINT_1,0,'

19:22:25.450909000 [Info ] [__main__] raw template row: b'SPXWc312228500.U,,2021-06-23T10:30:41.362530994Z,,FID,64,,OPINT_1,0,'

@vinod.amarnath

Find more posts tagged with

tick-history-rest-api

Accepted answers

Jirapongse

The issue may relate to the automatic decompression in the library used by the application.

The client needs to disable the automatic decompression and download the gzip file. Then, decompress the file to get the data.

All comments

zoya faberov

Hello @Vinod A ,

I fully sympathize that it can be very frustrating when one's API request only succeeds partially, and one can see no obvious reason for this to happen.

On DSS API developers forums, we can try to help with DSS API usage, and if you would like to include your complete request, we can be helpful, by reviewing it, and testing the same on our side?

If you would like DSS Support to investigate the potential failure of your request based on the job ID, on service side logs, to tell if there were any failures on the backend, please open a support ticket with DSS Support via Refinitiv Helpdesk Online -> DSS and include the extraction ID, etc.

Hope this makes sense, please let us know how you would like to proceed?

Beera.Rajesh

@zoya faberov Hi, I have recommend the client to test the below API call and client still observe the same issue. < Even though manifest says 2 instruments with two transactions. Output shows only one RIC>

Below is the client log:

POST call:
END POINT: https://selectapi.datascope.refinitiv.com/RestApi/v1/Extractions/ExtractRaw
HEADERS:
Content-Type:application/json
Authorization:Token <Token Value>
Prefer:respond-async
BODY:

@{ "ExtractionRequest": { "@odata.type": "#DataScope.Select.Api.Extractions.ExtractionRequests.TickHistoryRawExtractionRequest",
"IdentifierList": { "@odata.type": "#DataScope.Select.Api.Extractions.ExtractionRequests.InstrumentIdentifierList",
"ValidationOptions": { "AllowHistoricalInstruments": true },
"UseUserPreferencesForValidationOptions": false,
"InstrumentIdentifiers": [
{ "Identifier": "SPXWf252132750.U", "IdentifierType": "Ric" },
{ "Identifier": "SPXWc312228500.U", "IdentifierType": "Ric" } ] },
"Condition": { "MessageTimeStampIn": "GmtUtc",
"ReportDateRangeType": "Relative",
"RelativeStartDaysAgo": 5,
"RelativeStartTime": "02:00:00",
"RelativeEndDaysAgo": 4,
"RelativeEndTime": "01:59:59",
"ExtractBy": "Ric",
"SortBy": "SingleByRic",
"DomainCode": "MarketPrice",
"Fids": "64",
"DisplaySourceRIC": false,
"FidListOperator": "OR"
}
}
}

GET call
END POINT: https://selectapi.datascope.refinitiv.com/RestApi/v1/Extractions/RawExtractionResults('0x079c6d855f9cdac1')/$value
HEADERS:
Authorization:Token <Token Value>
Prefer:respond-async
Content-Type:application/json
Accept-Encoding:gzip, deflate

Client response:

Extraction Services Version 15.0.42358 (01a7f7ea050d), Built May 20 2021 18:20:45

User ID: 9028636

Extraction ID: 2000000276116294

Schedule: 0x079c7267195cdacb (ID = 0x0000000000000000)

Input List (2 items): (ID = 0x079c7267195cdacb) Created: 2021-06-29 13:16:17 Last Modified: 2021-06-29 13:16:17

Report Template: _OnD_0x079c7267195cdacb (ID = 0x079c7267197cdacb) Created: 2021-06-29 13:15:13 Last Modified: 2021-06-29 13:15:13

Schedule dispatched via message queue (0x079c7267195cdacb), Data source identifier (F8FBFBB1704E4BDF9E2448BEA70607A8)

Schedule Time: 2021-06-29 13:15:14

Processing started at 2021-06-29 13:15:14

Processing completed successfully at 2021-06-29 13:16:18

Extraction finished at 2021-06-29 11:16:18 UTC, with servers: tm16n01, TRTH (44.58 secs)

Instrument <RIC,SPXWc312228500.U> expanded to 1 RIC: SPXWc312228500.U.

Instrument <RIC,SPXWo312227750.U> expanded to 1 RIC: SPXWo312227750.U.

Total instruments after instrument expansion = 2

Manifest: #RIC,Domain,Start,End,Status,Count

Manifest: SPXWc312228500.U,Market Price,2021-06-28T06:45:35.607655899Z,2021-06-28T10:30:31.711417855Z,Active,2

Manifest: SPXWo312227750.U,Market Price,2021-06-28T06:46:05.752490195Z,2021-06-28T10:31:01.528677885Z,Active,2

13:16:45.621125000 [Info ] [app.refinitiv_client] all response : {'_body': None,

'_cache': {'headers': <CIMultiDictProxy('Cache-Control': 'no-cache', 'Pragma': 'no-cache', 'Content-Length': '439', 'Content-Type': 'text/plain', 'Content-Encoding': 'gzip', 'Expires': '-1', 'Accept-Ranges': 'bytes', 'Server': 'Microsoft-IIS/10.0', 'BeginRequestTime': '11:16:45.2335949', 'BeginRequestDate': '2021-06-29', 'X-Request-Execution-Correlation-Id': 'CiD/9028636/AAAAAA.079c7267307cdacb/RA', 'X-App-Id': 'Custom.RestApi', 'X-App-Version': '15.0.155.64', 'CPUUtilization': '1.823213', 'RequestsPerSec': '5.772524', 'W3WP-PrivateBytes': '1222976', 'Date': 'Tue, 29 Jun 2021 11:16:44 GMT')>,

'url': URL('https://transfer01:10180/RestApi/v1/Extractions/RawExtractionResults('0x079c7267195cdacb')/$value')},

'_closed': True,

'_connection': None,

'_continue': None,

'_headers': <CIMultiDictProxy('Cache-Control': 'no-cache', 'Pragma': 'no-cache', 'Content-Length': '439', 'Content-Type': 'text/plain', 'Content-Encoding': 'gzip', 'Expires': '-1', 'Accept-Ranges': 'bytes', 'Server': 'Microsoft-IIS/10.0', 'BeginRequestTime': '11:16:45.2335949', 'BeginRequestDate': '2021-06-29', 'X-Request-Execution-Correlation-Id': 'CiD/9028636/AAAAAA.079c7267307cdacb/RA', 'X-App-Id': 'Custom.RestApi', 'X-App-Version': '15.0.155.64', 'CPUUtilization': '1.823213', 'RequestsPerSec': '5.772524', 'W3WP-PrivateBytes': '1222976', 'Date': 'Tue, 29 Jun 2021 11:16:44 GMT')>,

'_history': (),

'_loop': <_UnixSelectorEventLoop running=True closed=False debug=False>,

'_protocol': <aiohttp.client_proto.ResponseHandler object at 0x7fd2c4a93730>,

'_raw_headers': ((b'Cache-Control', b'no-cache'),

(b'Pragma', b'no-cache'),

(b'Content-Length', b'439'),

(b'Content-Type', b'text/plain'),

(b'Content-Encoding', b'gzip'),

(b'Expires', b'-1'),

(b'Accept-Ranges', b'bytes'),

(b'Server', b'Microsoft-IIS/10.0'),

(b'BeginRequestTime', b'11:16:45.2335949'),

(b'BeginRequestDate', b'2021-06-29'),

(b'X-Request-Execution-Correlation-Id',

b'CiD/9028636/AAAAAA.079c7267307cdacb/RA'),

(b'X-App-Id', b'Custom.RestApi'),

(b'X-App-Version', b'15.0.155.64'),

(b'CPUUtilization', b'1.823213'),

(b'RequestsPerSec', b'5.772524'),

(b'W3WP-PrivateBytes', b'1222976'),

(b'Date', b'Tue, 29 Jun 2021 11:16:44 GMT')),

'_real_url': URL('https://transfer01:10180/RestApi/v1/Extractions/RawExtractionResults('0x079c7267195cdacb')/$value'),

'_request_info': RequestInfo(url=URL('https://transfer01:10180/RestApi/v1/Extractions/RawExtractionResults('0x079c7267195cdacb')/$value'), method='GET', headers=<CIMultiDictProxy('Host': 'transfer01:10180', 'Authorization': 'Token _KbSouNjEzjwo16KBkDEJPlyeE2fcN0OSiyEl12t7YkogcNI_UEvYygfS4K0gVwlrXho4iR7Vdy4tVSKz6Rvbx5hv94uwcGTxJ8S5NljbL-EOFgCP5EegwWJO6f7cK2AnXJ-U_0vSkMV4pVbMUBZff26xgGGIn5tz-tdYsBq23CcHrTxOPUGH-09Fry71T6YO0TYcJTSxpBdA2-r-Gtd77PpLRpX9O-SxDdQAX7oChmmgz5J5dyqq32W-ucATXkE1SI9PB3Wmos8-SDcz76ybhkIH5FICAqaeB5FwwUV1OL4', 'Prefer': 'respond-async', 'Accept-Encoding': 'gzip, deflate', 'Content-Type': 'application/json', 'Accept': '*/*', 'User-Agent': 'Python/3.6 aiohttp/3.7.3')>, real_url=URL('https://transfer01:10180/RestApi/v1/Extractions/RawExtractionResults('0x079c7267195cdacb')/$value')),

'_session': None,

'_timer': <aiohttp.helpers.TimerContext object at 0x7fd2c9d0aeb8>,

'_traces': [],

'_url': URL('https://transfer01:10180/RestApi/v1/Extractions/RawExtractionResults('0x079c7267195cdacb')/$value'),

'_writer': None,

'content': <StreamReader 426 bytes eof>,

'cookies': <SimpleCookie: >,

'method': 'GET',

'reason': 'OK',

'status': 200,

'version': HttpVersion(major=1, minor=1)}

13:16:45.621279000 [Info ] [app.refinitiv_client] 6: RX 200 OK

13:16:45.621435000 [Info ] [__main__] raw_template header: b'#RIC,Domain,Date-Time,GMT Offset,Type,MsgClass/FID number,UpdateType/Action,FID Name,FID Value,FID Enum String,PE Code,Template Number,Key/Msg Sequence Number,Number of FIDs\n'

13:16:45.621883000 [Info ] [__main__] raw template row: b'#RIC,Domain,Date-Time,GMT Offset,Type,MsgClass/FID number,UpdateType/Action,FID Name,FID Value,FID Enum String,PE Code,Template Number,Key/Msg Sequence Number,Number of FIDs\n'

13:16:45.621994000 [Info ] [__main__] raw template row: b'SPXWc312228500.U,,2021-06-28T06:45:35.607655899Z,,FID,64,,OPINT_1,0,'

13:16:45.622096000 [Info ] [__main__] raw template row: b'SPXWc312228500.U,,2021-06-28T10:30:31.711417855Z,,FID,64,,OPINT_1,0,'

Jirapongse

If I request only one RIC (SPXWf252132750.U), it returns "Historical Instrument <RIC,SPXWf252132750.U> expanded to 0 RICS.\n\nReport suppressed because there are no instruments".

{
    "@odata.context": "https://selectapi.datascope.refinitiv.com/RestApi/v1/$metadata#RawExtractionResults/$entity",
    "JobId": "0x079e4734529ce32d",
    "Notes": [
        "Extraction Services Version 15.0.42358 (01a7f7ea050d), Built May 20 2021 18:20:45\nUser ID: 9008895\nExtraction ID: 2000000278211939\nSchedule: 0x079e4734529ce32d (ID = 0x0000000000000000)\nInput List (1 items):  (ID = 0x079e4734529ce32d) Created: 07/05/2021 04:31:03 Last Modified: 07/05/2021 04:31:03\nReport Template: _OnD_0x079e4734529ce32d (ID = 0x079e473452bce32d) Created: 07/05/2021 04:30:57 Last Modified: 07/05/2021 04:30:57\nSchedule dispatched via message queue (0x079e4734529ce32d)\nSchedule Time: 07/05/2021 04:30:58\nProcessing started at 07/05/2021 04:30:58\nProcessing completed successfully at 07/05/2021 04:31:03\nExtraction finished at 07/05/2021 03:31:03 UTC, with servers: x03q13\nHistorical Instrument <RIC,SPXWf252132750.U> expanded to 0 RICS.\n\nReport suppressed because there are no instruments\n"
    ]
}

Please directly contact the DSS server team to verify the problem.

Jirapongse

The issue may relate to the automatic decompression in the library used by the application.

The client needs to disable the automatic decompression and download the gzip file. Then, decompress the file to get the data.

EXPLORE OUR SITES