Tick History Raw Extraction result does not retrieve all the rows; the manifests are always fine ...

Vinod A
Vinod A LSEG
in TRTH

...for all jobs, but not all rows are available when I extract the results for the finished jobs.

Tick History Raw Extraction result does not retrieve all the rows; the manifests are always fine for all jobs, but not all rows are available when I extract the results for the finished jobs.

I am using Python aiohttp package to execute ClientSession.request() method.

Then we parse all lines of json content which we received. The results are consistent between Chrome and my Python application.

https://selectapi.datascope.refinitiv.com/RestApi/v1/Extractions/RawExtractionResults('0x079b2cbf54ecd44e')/$value

Below extraction result; we see only 2 of 4 rows.

Extraction Services Version 15.0.42358 (01a7f7ea050d), Built May 20 2021 18:20:45

User ID: 9028636

Extraction ID: 2000000274338174

Schedule: 0x079ae3f6c06cd2d6 (ID = 0x0000000000000000)

Input List (2 items): (ID = 0x079ae3f6c06cd2d6) Created: 2021-06-24 19:21:56 Last Modified: 2021-06-24 19:21:56

Report Template: _OnD_0x079ae3f6c06cd2d6 (ID = 0x079ae3f6c08cd2d6) Created: 2021-06-24 19:20:53 Last Modified: 2021-06-24 19:20:53

Schedule dispatched via message queue (0x079ae3f6c06cd2d6), Data source identifier (A665CFE4E0C34D93BE691CF39752B49D)

Schedule Time: 2021-06-24 19:20:53

Processing started at 2021-06-24 19:20:53

Processing completed successfully at 2021-06-24 19:21:57

Extraction finished at 2021-06-24 17:21:57 UTC, with servers: tm01n01, TRTH (41.752 secs)

Instrument <RIC,SPXWf252132750.U> expanded to 1 RIC: SPXWf252132750.U.

Instrument <RIC,SPXWc312228500.U> expanded to 1 RIC: SPXWc312228500.U.

Total instruments after instrument expansion = 2

Quota Message: INFO: Tick History Options Quota Count Before Extraction: 9; Instruments Approved for Extraction: 1; Tick History Options Quota Count After Extraction: 9, 0.428571428571429% of Limit; Tick History Options Quota Limit: 2100

Manifest: #RIC,Domain,Start,End,Status,Count

Manifest: SPXWc312228500.U,Market Price,2021-06-23T06:45:44.690108670Z,2021-06-23T10:30:41.362530994Z,Active,2

Manifest: SPXWf252132750.U,Market Price,2021-06-23T06:46:14.906572283Z,2021-06-23T10:31:11.921941834Z,Active,2

19:22:25.007065000 [Info ] [app.refinitiv_client] 6: TX GET https://transfer01:10180/RestApi/v1/Extractions/RawExtractionResults('0x079ae3f6c06cd2d6')/$value

19:22:25.449803000 [Info ] [app.refinitiv_client] all response : {'_body': None,

'_cache': {'headers': <CIMultiDictProxy('Cache-Control': 'no-cache', 'Pragma': 'no-cache', 'Content-Length': '440', 'Content-Type': 'text/plain', 'Content-Encoding': 'gzip', 'Expires': '-1', 'Accept-Ranges': 'bytes', 'Server': 'Microsoft-IIS/10.0', 'BeginRequestTime': '17:22:25.0514990', 'BeginRequestDate': '2021-06-24', 'X-Request-Execution-Correlation-Id': 'CiD/9028636/AAAAAA.079b2cc3246cd44e/RA', 'X-App-Id': 'Custom.RestApi', 'X-App-Version': '15.0.155.64', 'CPUUtilization': '3.048618', 'RequestsPerSec': '0', 'W3WP-PrivateBytes': '1214820', 'Date': 'Thu, 24 Jun 2021 17:22:25 GMT')>,

'url': URL('https://transfer01:10180/RestApi/v1/Extractions/RawExtractionResults('0x079ae3f6c06cd2d6')/$value')},

'_closed': True,

'_connection': None,

'_continue': None,

'_headers': <CIMultiDictProxy('Cache-Control': 'no-cache', 'Pragma': 'no-cache', 'Content-Length': '440', 'Content-Type': 'text/plain', 'Content-Encoding': 'gzip', 'Expires': '-1', 'Accept-Ranges': 'bytes', 'Server': 'Microsoft-IIS/10.0', 'BeginRequestTime': '17:22:25.0514990', 'BeginRequestDate': '2021-06-24', 'X-Request-Execution-Correlation-Id': 'CiD/9028636/AAAAAA.079b2cc3246cd44e/RA', 'X-App-Id': 'Custom.RestApi', 'X-App-Version': '15.0.155.64', 'CPUUtilization': '3.048618', 'RequestsPerSec': '0', 'W3WP-PrivateBytes': '1214820', 'Date': 'Thu, 24 Jun 2021 17:22:25 GMT')>,

'_history': (),

'_loop': <_UnixSelectorEventLoop running=True closed=False debug=False>,

'_protocol': <aiohttp.client_proto.ResponseHandler object at 0x7f0a0d370730>,

'_raw_headers': ((b'Cache-Control', b'no-cache'),

(b'Pragma', b'no-cache'),

(b'Content-Length', b'440'),

(b'Content-Type', b'text/plain'),

(b'Content-Encoding', b'gzip'),

(b'Expires', b'-1'),

(b'Accept-Ranges', b'bytes'),

(b'Server', b'Microsoft-IIS/10.0'),

(b'BeginRequestTime', b'17:22:25.0514990'),

(b'BeginRequestDate', b'2021-06-24'),

(b'X-Request-Execution-Correlation-Id',

b'CiD/9028636/AAAAAA.079b2cc3246cd44e/RA'),

(b'X-App-Id', b'Custom.RestApi'),

(b'X-App-Version', b'15.0.155.64'),

(b'CPUUtilization', b'3.048618'),

(b'RequestsPerSec', b'0'),

(b'W3WP-PrivateBytes', b'1214820'),

(b'Date', b'Thu, 24 Jun 2021 17:22:25 GMT')),

'_real_url': URL('https://transfer01:10180/RestApi/v1/Extractions/RawExtractionResults('0x079ae3f6c06cd2d6')/$value'),

'_request_info': RequestInfo(url=URL('https://transfer01:10180/RestApi/v1/Extractions/RawExtractionResults('0x079ae3f6c06cd2d6')/$value'), method='GET', headers=<CIMultiDictProxy('Host': 'transfer01:10180', 'Authorization': 'Token ', 'Prefer': 'odata.maxpagesize=20000', 'Accept': '*/*', 'Accept-Encoding': 'gzip, deflate', 'User-Agent': 'Python/3.6 aiohttp/3.7.3')>, real_url=URL('https://transfer01:10180/RestApi/v1/Extractions/RawExtractionResults('0x079ae3f6c06cd2d6')/$value')),

'_session': None,

'_timer': <aiohttp.helpers.TimerContext object at 0x7f0a12a30e10>,

'_traces': [],

'_url': URL('https://transfer01:10180/RestApi/v1/Extractions/RawExtractionResults('0x079ae3f6c06cd2d6')/$value'),

'_writer': None,

'content': <StreamReader 426 bytes eof>,

'cookies': <SimpleCookie: >,

'method': 'GET',

'reason': 'OK',

'status': 200,

'version': HttpVersion(major=1, minor=1)}

19:22:25.450062000 [Info ] [app.refinitiv_client] 6: RX 200 OK

19:22:25.450225000 [Info ] [__main__] raw_template header: b'#RIC,Domain,Date-Time,GMT Offset,Type,MsgClass/FID number,UpdateType/Action,FID Name,FID Value,FID Enum String,PE Code,Template Number,Key/Msg Sequence Number,Number of FIDs\n'

19:22:25.450816000 [Info ] [__main__] raw template row: b'SPXWc312228500.U,,2021-06-23T06:45:44.690108670Z,,FID,64,,OPINT_1,0,'

19:22:25.450909000 [Info ] [__main__] raw template row: b'SPXWc312228500.U,,2021-06-23T10:30:41.362530994Z,,FID,64,,OPINT_1,0,'

@vinod.amarnath

Tagged:

Best Answer

  • Jirapongse
    Jirapongse ✭✭✭✭✭
    Answer ✓

    The issue may relate to the automatic decompression in the library used by the application.

    The client needs to disable the automatic decompression and download the gzip file. Then, decompress the file to get the data.

Answers

  • zoya faberov
    zoya faberov ✭✭✭✭✭

    Hello @Vinod A ,

    I fully sympathize that it can be very frustrating when one's API request only succeeds partially, and one can see no obvious reason for this to happen.

    On DSS API developers forums, we can try to help with DSS API usage, and if you would like to include your complete request, we can be helpful, by reviewing it, and testing the same on our side?

    If you would like DSS Support to investigate the potential failure of your request based on the job ID, on service side logs, to tell if there were any failures on the backend, please open a support ticket with DSS Support via Refinitiv Helpdesk Online -> DSS and include the extraction ID, etc.

    Hope this makes sense, please let us know how you would like to proceed?

  • Beera.Rajesh
    Beera.Rajesh LSEG

    @zoya faberov Hi, I have recommend the client to test the below API call and client still observe the same issue. < Even though manifest says 2 instruments with two transactions. Output shows only one RIC>

    Below is the client log:

    POST call:
    END POINT: https://selectapi.datascope.refinitiv.com/RestApi/v1/Extractions/ExtractRaw
    HEADERS:
    Content-Type:application/json
    Authorization:Token <Token Value>
    Prefer:respond-async
    BODY:

    @{ "ExtractionRequest": { "@odata.type": "#DataScope.Select.Api.Extractions.ExtractionRequests.TickHistoryRawExtractionRequest",
    "IdentifierList": { "@odata.type": "#DataScope.Select.Api.Extractions.ExtractionRequests.InstrumentIdentifierList",
    "ValidationOptions": { "AllowHistoricalInstruments": true },
    "UseUserPreferencesForValidationOptions": false,
    "InstrumentIdentifiers": [
    { "Identifier": "SPXWf252132750.U", "IdentifierType": "Ric" },
    { "Identifier": "SPXWc312228500.U", "IdentifierType": "Ric" } ] },
    "Condition": { "MessageTimeStampIn": "GmtUtc",
    "ReportDateRangeType": "Relative",
    "RelativeStartDaysAgo": 5,
    "RelativeStartTime": "02:00:00",
    "RelativeEndDaysAgo": 4,
    "RelativeEndTime": "01:59:59",
    "ExtractBy": "Ric",
    "SortBy": "SingleByRic",
    "DomainCode": "MarketPrice",
    "Fids": "64",
    "DisplaySourceRIC": false,
    "FidListOperator": "OR"
    }
    }
    }

    GET call
    END POINT: https://selectapi.datascope.refinitiv.com/RestApi/v1/Extractions/RawExtractionResults('0x079c6d855f9cdac1')/$value
    HEADERS:
    Authorization:Token <Token Value>
    Prefer:respond-async
    Content-Type:application/json
    Accept-Encoding:gzip, deflate



    Client response:

    Extraction Services Version 15.0.42358 (01a7f7ea050d), Built May 20 2021 18:20:45

    User ID: 9028636

    Extraction ID: 2000000276116294

    Schedule: 0x079c7267195cdacb (ID = 0x0000000000000000)

    Input List (2 items): (ID = 0x079c7267195cdacb) Created: 2021-06-29 13:16:17 Last Modified: 2021-06-29 13:16:17

    Report Template: _OnD_0x079c7267195cdacb (ID = 0x079c7267197cdacb) Created: 2021-06-29 13:15:13 Last Modified: 2021-06-29 13:15:13

    Schedule dispatched via message queue (0x079c7267195cdacb), Data source identifier (F8FBFBB1704E4BDF9E2448BEA70607A8)

    Schedule Time: 2021-06-29 13:15:14

    Processing started at 2021-06-29 13:15:14

    Processing completed successfully at 2021-06-29 13:16:18

    Extraction finished at 2021-06-29 11:16:18 UTC, with servers: tm16n01, TRTH (44.58 secs)

    Instrument <RIC,SPXWc312228500.U> expanded to 1 RIC: SPXWc312228500.U.

    Instrument <RIC,SPXWo312227750.U> expanded to 1 RIC: SPXWo312227750.U.

    Total instruments after instrument expansion = 2

    Quota Message: INFO: Tick History Options Quota Count Before Extraction: 9; Instruments Approved for Extraction: 1; Tick History Options Quota Count After Extraction: 9, 0.428571428571429% of Limit; Tick History Options Quota Limit: 2100

    Manifest: #RIC,Domain,Start,End,Status,Count

    Manifest: SPXWc312228500.U,Market Price,2021-06-28T06:45:35.607655899Z,2021-06-28T10:30:31.711417855Z,Active,2

    Manifest: SPXWo312227750.U,Market Price,2021-06-28T06:46:05.752490195Z,2021-06-28T10:31:01.528677885Z,Active,2

    13:16:45.621125000 [Info ] [app.refinitiv_client] all response : {'_body': None,

    '_cache': {'headers': <CIMultiDictProxy('Cache-Control': 'no-cache', 'Pragma': 'no-cache', 'Content-Length': '439', 'Content-Type': 'text/plain', 'Content-Encoding': 'gzip', 'Expires': '-1', 'Accept-Ranges': 'bytes', 'Server': 'Microsoft-IIS/10.0', 'BeginRequestTime': '11:16:45.2335949', 'BeginRequestDate': '2021-06-29', 'X-Request-Execution-Correlation-Id': 'CiD/9028636/AAAAAA.079c7267307cdacb/RA', 'X-App-Id': 'Custom.RestApi', 'X-App-Version': '15.0.155.64', 'CPUUtilization': '1.823213', 'RequestsPerSec': '5.772524', 'W3WP-PrivateBytes': '1222976', 'Date': 'Tue, 29 Jun 2021 11:16:44 GMT')>,

    'url': URL('https://transfer01:10180/RestApi/v1/Extractions/RawExtractionResults('0x079c7267195cdacb')/$value')},

    '_closed': True,

    '_connection': None,

    '_continue': None,

    '_headers': <CIMultiDictProxy('Cache-Control': 'no-cache', 'Pragma': 'no-cache', 'Content-Length': '439', 'Content-Type': 'text/plain', 'Content-Encoding': 'gzip', 'Expires': '-1', 'Accept-Ranges': 'bytes', 'Server': 'Microsoft-IIS/10.0', 'BeginRequestTime': '11:16:45.2335949', 'BeginRequestDate': '2021-06-29', 'X-Request-Execution-Correlation-Id': 'CiD/9028636/AAAAAA.079c7267307cdacb/RA', 'X-App-Id': 'Custom.RestApi', 'X-App-Version': '15.0.155.64', 'CPUUtilization': '1.823213', 'RequestsPerSec': '5.772524', 'W3WP-PrivateBytes': '1222976', 'Date': 'Tue, 29 Jun 2021 11:16:44 GMT')>,

    '_history': (),

    '_loop': <_UnixSelectorEventLoop running=True closed=False debug=False>,

    '_protocol': <aiohttp.client_proto.ResponseHandler object at 0x7fd2c4a93730>,

    '_raw_headers': ((b'Cache-Control', b'no-cache'),

    (b'Pragma', b'no-cache'),

    (b'Content-Length', b'439'),

    (b'Content-Type', b'text/plain'),

    (b'Content-Encoding', b'gzip'),

    (b'Expires', b'-1'),

    (b'Accept-Ranges', b'bytes'),

    (b'Server', b'Microsoft-IIS/10.0'),

    (b'BeginRequestTime', b'11:16:45.2335949'),

    (b'BeginRequestDate', b'2021-06-29'),

    (b'X-Request-Execution-Correlation-Id',

    b'CiD/9028636/AAAAAA.079c7267307cdacb/RA'),

    (b'X-App-Id', b'Custom.RestApi'),

    (b'X-App-Version', b'15.0.155.64'),

    (b'CPUUtilization', b'1.823213'),

    (b'RequestsPerSec', b'5.772524'),

    (b'W3WP-PrivateBytes', b'1222976'),

    (b'Date', b'Tue, 29 Jun 2021 11:16:44 GMT')),

    '_real_url': URL('https://transfer01:10180/RestApi/v1/Extractions/RawExtractionResults('0x079c7267195cdacb')/$value'),

    '_request_info': RequestInfo(url=URL('https://transfer01:10180/RestApi/v1/Extractions/RawExtractionResults('0x079c7267195cdacb')/$value'), method='GET', headers=<CIMultiDictProxy('Host': 'transfer01:10180', 'Authorization': 'Token _KbSouNjEzjwo16KBkDEJPlyeE2fcN0OSiyEl12t7YkogcNI_UEvYygfS4K0gVwlrXho4iR7Vdy4tVSKz6Rvbx5hv94uwcGTxJ8S5NljbL-EOFgCP5EegwWJO6f7cK2AnXJ-U_0vSkMV4pVbMUBZff26xgGGIn5tz-tdYsBq23CcHrTxOPUGH-09Fry71T6YO0TYcJTSxpBdA2-r-Gtd77PpLRpX9O-SxDdQAX7oChmmgz5J5dyqq32W-ucATXkE1SI9PB3Wmos8-SDcz76ybhkIH5FICAqaeB5FwwUV1OL4', 'Prefer': 'respond-async', 'Accept-Encoding': 'gzip, deflate', 'Content-Type': 'application/json', 'Accept': '*/*', 'User-Agent': 'Python/3.6 aiohttp/3.7.3')>, real_url=URL('https://transfer01:10180/RestApi/v1/Extractions/RawExtractionResults('0x079c7267195cdacb')/$value')),

    '_session': None,

    '_timer': <aiohttp.helpers.TimerContext object at 0x7fd2c9d0aeb8>,

    '_traces': [],

    '_url': URL('https://transfer01:10180/RestApi/v1/Extractions/RawExtractionResults('0x079c7267195cdacb')/$value'),

    '_writer': None,

    'content': <StreamReader 426 bytes eof>,

    'cookies': <SimpleCookie: >,

    'method': 'GET',

    'reason': 'OK',

    'status': 200,

    'version': HttpVersion(major=1, minor=1)}

    13:16:45.621279000 [Info ] [app.refinitiv_client] 6: RX 200 OK

    13:16:45.621435000 [Info ] [__main__] raw_template header: b'#RIC,Domain,Date-Time,GMT Offset,Type,MsgClass/FID number,UpdateType/Action,FID Name,FID Value,FID Enum String,PE Code,Template Number,Key/Msg Sequence Number,Number of FIDs\n'

    13:16:45.621883000 [Info ] [__main__] raw template row: b'#RIC,Domain,Date-Time,GMT Offset,Type,MsgClass/FID number,UpdateType/Action,FID Name,FID Value,FID Enum String,PE Code,Template Number,Key/Msg Sequence Number,Number of FIDs\n'

    13:16:45.621994000 [Info ] [__main__] raw template row: b'SPXWc312228500.U,,2021-06-28T06:45:35.607655899Z,,FID,64,,OPINT_1,0,'

    13:16:45.622096000 [Info ] [__main__] raw template row: b'SPXWc312228500.U,,2021-06-28T10:30:31.711417855Z,,FID,64,,OPINT_1,0,'

    even-on-gui.png

  • Jirapongse
    Jirapongse ✭✭✭✭✭

    If I request only one RIC (SPXWf252132750.U), it returns "Historical Instrument <RIC,SPXWf252132750.U> expanded to 0 RICS.\n\nReport suppressed because there are no instruments".

    {
        "@odata.context": "https://selectapi.datascope.refinitiv.com/RestApi/v1/$metadata#RawExtractionResults/$entity",
        "JobId": "0x079e4734529ce32d",
        "Notes": [
            "Extraction Services Version 15.0.42358 (01a7f7ea050d), Built May 20 2021 18:20:45\nUser ID: 9008895\nExtraction ID: 2000000278211939\nSchedule: 0x079e4734529ce32d (ID = 0x0000000000000000)\nInput List (1 items):  (ID = 0x079e4734529ce32d) Created: 07/05/2021 04:31:03 Last Modified: 07/05/2021 04:31:03\nReport Template: _OnD_0x079e4734529ce32d (ID = 0x079e473452bce32d) Created: 07/05/2021 04:30:57 Last Modified: 07/05/2021 04:30:57\nSchedule dispatched via message queue (0x079e4734529ce32d)\nSchedule Time: 07/05/2021 04:30:58\nProcessing started at 07/05/2021 04:30:58\nProcessing completed successfully at 07/05/2021 04:31:03\nExtraction finished at 07/05/2021 03:31:03 UTC, with servers: x03q13\nHistorical Instrument <RIC,SPXWf252132750.U> expanded to 0 RICS.\n\nReport suppressed because there are no instruments\n"
        ]
    }

    Please directly contact the DSS server team to verify the problem.

Categories