TickHistoryRawExtractionRequest REST API throws an error using Curl

I have followed below steps using curl:

1) get the authentication token which is working and generated token:

OdiOSCommand "-OUT_FILE=/home/oracle/Documents/OutFile/refinitiv_file.txt" "-CAPTURE_OUT_STREAM=ON_ERROR,10,50" "-CAPTURE_ERR_STREAM=ON_ERROR,10,50"
curl -H "Content-Type: application/json" -H "Prefer: respond-async" -X POST -v -d "{\"Credentials\": {\"Username\": \"501\",\"Password\": \"xxxxxx\"}}" https://selectapi.datascope.refinitiv.com/RestApi/v1/Authentication/RequestToken

2) Use this authentication token in the below curl request:

OdiOSCommand "-OUT_FILE=/home/oracle/Documents/OutFile/refinitiv_file_extraction_response_json.txt" "-CAPTURE_OUT_STREAM=ON_ERROR,10,50" "-CAPTURE_ERR_STREAM=ON_ERROR,10,50"
curl -X POST \
    https://selectapi.datascope.refinitiv.com/RestApi/v1/Extractions/ExtractRaw \
    -H 'Authorization: Token __7OGZ1r1Hp4HaTam98Gp9kmovQlNv8NeKEHe6boyIRJXnzNmdLSvnfJXcHysqu9JCgX6PH9nk30671SwRb1ol_dj9oG46BY3mG6JGMMNs17J_TZ7tP9RL2Y2q-6KaqL63EjjfWt9Bm-5usuI3alHDy5Q_AwVXoSW140O8x3AwNfhvOSUpxAllzXidWNXn27yfgHrMm28ZjKEycVqJOwfztmcr75tHj9ijvHogyaXzmiKpPv2sxc45aqEW2KnrKASN1zEhzw3ICrtDBm4Xhx9TzL2TB-XpBPDDWpaUi4jFdU' \
    -H 'Content-Type: application/json' \
    -d '{
        "ExtractionRequest": {
        "@odata.type": "#DataScope.Select.Api.Extractions.ExtractionRequests.TickHistoryRawExtractionRequest",
        "ContentFieldNames": [  
       
        ],
        "IdentifierList": {
          "@odata.type": "#DataScope.Select.Api.Extractions.ExtractionRequests.InstrumentIdentifierList",
          "InstrumentIdentifiers": [
            {"Identifier":"IBM.N","IdentifierType":"Ric"},{"Identifier":"CARR.PA","IdentifierType":"Ric"},{"Identifier":"BEGRs.TQE","IdentifierType":"Ric"},{"Identifier":"US1156371007.GTX","IdentifierType":"Ric"},{"Identifier":"JP3249600002.GTX","IdentifierType":"Ric"}
          ]
        },
        "Condition": {
            "MessageTimeStampIn": "GmtUtc",
            "ReportDateRangeType": "Range",
            "QueryStartDate": "2021-08-20T00:00:00.000Z",
            "QueryEndDate": "2021-08-23T00:00:00.000Z",
            "Fids": "3,5,12,13,15,16,19,22,23,26,27,4196,5632,5633,2130",
            "DisplaySourceRIC": true
      }
      }
 }'

The output response for the above request is as below:

@{"@odata.context":"https://hosted.datascopeapi.reuters.com/RestApi/v1/$metadata#RawExtractionResults/$entity","JobId":"0x07ae207dfead0ec8","Notes":["Extraction Services Version 15.0.42358 (01a7f7ea050d), Built May 20 2021 18:20:45\nUser ID: 7701\nExtraction ID: 2000000296050738\nSchedule: 0x07ae207dfead0ec8 (ID = 0x0000000000000000)\nInput List (1 items):  (ID = 0x07ae207dfead0ec8) Created: 23.08.2021 15:58:07 Last Modified: 23.08.2021 15:58:07\nReport Template: _OnD_0x07ae207dfead0ec8 (ID = 0x07ae207dfecd0ec8) Created: 23.08.2021 15:56:55 Last Modified: 23.08.2021 15:56:55\nSchedule dispatched via message queue (0x07ae207dfead0ec8), Data source identifier (8A3F4C373BA140F98836FA4F5B502FC8)\nSchedule Time: 23.08.2021 15:56:56\nProcessing started at 23.08.2021 15:56:56\nProcessing completed successfully at 23.08.2021 15:58:07\nExtraction finished at 23.08.2021 13:58:07 UTC, with servers: tm02n02, TRTH (41.83 secs)\nInstrument <RIC,IBM.N> expanded to 1 RIC: IBM.N.\nTotal instruments after instrument expansion = 1\n\nManifest: #RIC,Domain,Start,End,Status,Count\nManifest: IBM.N,Market Price,,,Inactive,0\n"]}

3) This step i am not clear yet. In this step i tried to get the JobID and also previous token i have used from step 1 and use it another curl request to get the response values for the fids:

OdiOSCommand "-OUT_FILE=/home/oracle/Documents/OutFile/refinitiv_file_extraction_response_json_values.txt" "-CAPTURE_OUT_STREAM=ON_ERROR,10,50" "-CAPTURE_ERR_STREAM=ON_ERROR,10,50"
curl -X GET \
    https://selectapi.datascope.refinitiv.com/RestApi/v1/Extractions/RawExtractionResults('0x07ae2e231f1d0eeb')/$value \
    -H 'Authorization: Token __7OGZ1r1Hp4HaTam98Gp9kmovQlNv8NeKEHe6boyIRJXnzNmdLSvnfJXcHysqu9JCgX6PH9nk30671SwRb1ol_dj9oG46BY3mG6JGMMNs17J_TZ7tP9RL2Y2q-6KaqL63EjjfWt9Bm-5usuI3alHDy5Q_AwVXoSW140O8x3AwNfhvOSUpxAllzXidWNXn27yfgHrMm28ZjKEycVqJOwfztmcr75tHj9ijvHogyaXzmiKpPv2sxc45aqEW2KnrKASN1zEhzw3ICrtDBm4Xhx9TzL2TB-XpBPDDWpaUi4jFdU' \
    -H 'Content-Type: application/gzip' \
    -H 'X-Direct-Download: True'

And this step is throwing an error as below:

ODI-1226: Step Copy of Refinitiv_RESTAPI_REQUEST fails after 1 attempt(s).
ODI-1241: Oracle Data Integrator tool execution fails.
Caused By: com.sunopsis.dwg.function.SnpsFunctionBaseException: ODI-30038: OS command returned 1. Error details are [sh: -c: line 1: syntax error near unexpected token `('
sh: -c: line 1: `    https://selectapi.datascope.refinitiv.com/RestApi/v1/Extractions/RawExtractionResults('0x07ae2e231f1d0eeb')/$value \'
].
    at com.sunopsis.dwg.tools.OSCommand.actionExecute(OSCommand.java:535)
    at com.sunopsis.dwg.function.SnpsFunctionBaseRepositoryConnected.execute(SnpsFunctionBaseRepositoryConnected.java:235)
    at oracle.odi.runtime.agent.execution.SessionTask.execIntegratedFunction(SessionTask.java:949)
    at oracle.odi.runtime.agent.execution.SessionTask.executeOdiCommand(SessionTask.java:607)
    at oracle.odi.runtime.agent.execution.cmd.OdiCommandExecutor.execute(OdiCommandExecutor.java:32)
    at oracle.odi.runtime.agent.execution.cmd.OdiCommandExecutor.execute(OdiCommandExecutor.java:18)
    at oracle.odi.runtime.agent.execution.TaskExecutionHandler.handleTask(TaskExecutionHandler.java:52)
    at oracle.odi.runtime.agent.execution.SessionTask.processTask(SessionTask.java:216)
    at oracle.odi.runtime.agent.execution.SessionTask.doExecuteTask(SessionTask.java:128)
    at oracle.odi.runtime.agent.execution.AbstractSessionTask.execute(AbstractSessionTask.java:886)
    at oracle.odi.runtime.agent.execution.SessionExecutor$SerialTrain.runTasks(SessionExecutor.java:2225)
    at oracle.odi.runtime.agent.execution.SessionExecutor.executeSession(SessionExecutor.java:610)
    at oracle.odi.runtime.agent.processor.TaskExecutorAgentRequestProcessor$1.doAction(TaskExecutorAgentRequestProcessor.java:718)
    at oracle.odi.runtime.agent.processor.TaskExecutorAgentRequestProcessor$1.doAction(TaskExecutorAgentRequestProcessor.java:611)
    at oracle.odi.core.persistence.dwgobject.DwgObjectTemplate.execute(DwgObjectTemplate.java:203)
    at oracle.odi.runtime.agent.processor.TaskExecutorAgentRequestProcessor.doProcessStartAgentTask(TaskExecutorAgentRequestProcessor.java:800)
    at oracle.odi.runtime.agent.processor.impl.StartSessRequestProcessor.access$1400(StartSessRequestProcessor.java:74)
    at oracle.odi.runtime.agent.processor.impl.StartSessRequestProcessor$StartSessTask.doExecute(StartSessRequestProcessor.java:702)
    at oracle.odi.runtime.agent.processor.task.AgentTask.execute(AgentTask.java:180)
    at oracle.odi.runtime.agent.support.DefaultAgentTaskExecutor$2.run(DefaultAgentTaskExecutor.java:108)
    at java.lang.Thread.run(Thread.java:748)

Find more posts tagged with

extraction

tick-history-rest-api

Accepted answers

zoya faberov

Hello @rahul.deshmukh ,

Working with RTH REST via curl is stable. However, curl is a test tool, rather then a programming language. Requests via curl returns a lot of verbose output that you have indicated you would prefer not to have.

Have you had a chance to review Programming Without SDK tutorials? They go over the recommended steps and link to RTH Downloads that includes RTH Python Code Samples.

I would suggest using On-Demand examples( please see tutorials for the complete information on On-Demand and Scheduled/GUI) and replace the request in the example (IntradayBars request) with your required RTH request (TickHistoryRaw).

I have just used this approach, to make a python example for TickHistoryRaw on demand:

RTH_OnDemand_Raw.20210827.zip

Please find it attached, replace the credentials with yours, and I hope you find this example useful

All comments

rahul.deshmukh

@zoya faberov @Gurpreet can you please help me here...thanks

zoya faberov

Hello @rahul.deshmukh ,

There was a mandatory migration of the endpoints, for DSS and RTH,

Please find the requirement details on DSS API front page and more in-depth from RTH API front page

The tutorial on how to migrate that should be helpful can be found at Refinitiv Datascope Select API Migration Tutorial, this information is relevant for both RTH and DSS migration requirements.

rahul.deshmukh

@zoya.farberov Thanks now i modified as per the migration, there is no error now but i am not getting the correct response for fields:

Below is the response:

@{"@odata.context":"https://selectapi.datascope.refinitiv.com/RestApi/v1/$metadata#RawExtractionResults/$entity","JobId":"0x07ae62f53ead0f40","Notes":["Extraction Services Version 15.0.42358 (01a7f7ea050d), Built May 20 2021 18:20:45\nUser ID: 9001\nExtraction ID: 2000000296068231\nSchedule: 0x07ae62f53ead0f40 (ID = 0x0000000000000000)\nInput List (2 items):  (ID = 0x07ae62f53ead0f40) Created: 23.08.2021 17:08:43 Last Modified: 23.08.2021 17:08:43\nReport Template: _OnD_0x07ae62f53ead0f40 (ID = 0x07ae62f53ecd0f40) Created: 23.08.2021 17:07:40 Last Modified: 23.08.2021 17:07:40\nSchedule dispatched via message queue (0x07ae62f53ead0f40), Data source identifier (DED82B93BB234FF882B35BEE6E2510B3)\nSchedule Time: 23.08.2021 17:07:40\nProcessing started at 23.08.2021 17:07:40\nProcessing completed successfully at 23.08.2021 17:08:43\nExtraction finished at 23.08.2021 15:08:43 UTC, with servers: tm03n03, TRTH (49.021 secs)\nInstrument <RIC,IBM.N> expanded to 1 RIC: IBM.N.\nInstrument <RIC,CARR.PA> expanded to 1 RIC: CARR.PA.\nTotal instruments after instrument expansion = 2\n\nManifest: #RIC,Domain,Start,End,Status,Count\nManifest: CARR.PA,Market Price,,,Inactive,0\nManifest: IBM.N,Market Price,,,Inactive,0\n"]}

rahul.deshmukh

I am just doing some tests initailly with some Rics and fids to check if i am getting response from this RAW request. Is there anything wrong with the fids or Rics or do i need to modify condition or any parameters ? I just need to see response with the values correctly...Why its showing inactive in response...i have edited my question and updated the extraction request which i am using..

zoya faberov

Hello @rahul.deshmukh ,

Let me just confirm, you are doing all steps via curl as per

REST API Tutorial 8: On Demand raw data extraction

1. Authenticating and obtaining a valid token

2. Issuing request with valid token => getting back a submitted jobId/ExtractionID

3. Checking request status on the job ID => confirming it is completed = 200 ( not accepted = 202). And if status=202, checking until status is completed = 200.

4. Once the status becomes 200, extracting result via {{protocol}}{{host}}{{api}}Extractions/RawExtractionResults('{{jobId}}')/$value

This sequence of steps, as described in detail in the tutorial for Postman, will need to be done via Postman, or via curl, or implemented in any language supporting HTTP REST.

rahul.deshmukh

@zoya faberov i did step 1 and2 but not have done step 3 and directly proceed for step 4 where i provide token and my extraction request...all this steps i have done using curl...i already used curl with the same steps for FundAllocationExtractionRequest rest api and it works fine...i am doing the tick history raw request first time so not sure if i miss anything during extraction request...

zoya faberov

Hello @rahul.deshmukh ,

I have just tested the following step 4 curl, on a completed TickHistoryRaw result with status = 200

"curl-7.70.0-win64-mingw\bin\curl" -X GET "https://selectapi.datascope.refinitiv.com/RestApi/v1/Extractions/RawExtractionResults(';MYJOBID')/$value" -H "Authorization: Token MYVALIDTOKEN" -H 'Content-Type: application/gzip' -H 'X-Direct-Download: True' -x MyProxy:8080 -v -N -o "tst.csv.gz"

I got the result downloaded, as expected.

Hope this information helps

rahul.deshmukh

@zoya faberov can you please tell how do i find 'MYJOBID' in my response ? And unlike other Rest API this looks like new step for Raw extraction Request for tick history ? This makes the implementaion very difficult when we try to automate our programming script...because we have to put token and now 'MYJOBID' in the extraction request ? Your GET request looks different than i used in my code: "https://selectapi.datascope.refinitiv.com/RestApi/v1/Extractions/RawExtractionResults(';MYJOBID')/$value"

Also does this JobID change everytime like token for each request?

zoya faberov

Hello @rahul.deshmukh ,

Steps that we use to extract TickHistoryRaw content with curl:

1. Authenticate and obtain a valid token

...curl" -i -X POST -k -H "Content-Type: application/json" -H "Prefer: respond-async"  https://selectapi.datascope.refinitiv.com/RestApi/v1/Authentication/RequestToken -H 'X-Direct-Download: True' -x MYVALIDPROXY:8080 -d @creds.txt

where creds.txt file is located in the same directory and contains

{
 "Credentials": { 
      "Username": "MYVALIDDSSUSERID", 
      "Password": "MYVALIDDSSPASSWORD"
 }
}

Resulting in

@{"@odata.context":"https://selectapi.datascope.refinitiv.com/RestApi/v1/$metadata#Edm.String","value":&quot;MYVALIDTOKEN"}HTTP/1.1 404 Not Found
Cache-Control: no-cache
Pragma: no-cache
Content-Type: text/html; charset=utf-8
Proxy-Connection: Keep-Alive
Connection: Keep-Alive
Content-Length: 615

2. Use the valid token to create TickHistoryRaw extraction request:

...\curl-7.70.0-win64-mingw\bin\curl" -i -X POST -k -H "Content-Type: application/json" -H "Prefer: respond-async" "https://selectapi.datascope.refinitiv.com/RestApi/v1/Extractions/ExtractRaw&quot; -H "Authorization: Token MYVALIDTOKEN" -H 'Content-Type: application/json' -H 'X-Direct-Download: True' -x MYVALIDPROXY:8080 -d @requestBody.txt
HTTP/1.1 200 Connection established

Contents of requestBody.txt file saved into the same directory

{ 
  "ExtractionRequest": { 
    "@odata.type": "#DataScope.Select.Api.Extractions.ExtractionRequests.TickHistoryRawExtractionRequest", 
    "IdentifierList": { 
      "@odata.type": "#DataScope.Select.Api.Extractions.ExtractionRequests.InstrumentIdentifierList",  
      "InstrumentIdentifiers": [{
        "Identifier": "CARR.PA",
        "IdentifierType": "Ric"
      }]
    },
    "Condition": {
      "MessageTimeStampIn": "GmtUtc",
      "ReportDateRangeType": "Range",
      "QueryStartDate": "2016-09-29T12:00:00.000Z",
      "QueryEndDate": "2016-09-29T12:10:00.000Z",
      "ExtractBy": "Ric",
      "SortBy": "SingleByRic",
      "DomainCode": "MarketPrice",
      "DisplaySourceRIC": true
    }
  }
}

obtaining Extraction ID and status:

HTTP/1.1 202 Accepted
Cache-Control: no-cache
Pragma: no-cache
Expires: -1
Location: https://selectapi.datascope.refinitiv.com/RestApi/v1/Extractions/ExtractRawResult(ExtractionId='0x07ae64718b9d0f41')
Server: Microsoft-IIS/10.0
BeginRequestTime: 21:30:25.3627291
BeginRequestDate: 2021-08-23
Status: InProgress
Progress:
X-Request-Execution-Correlation-Id: CiD/19945/AAAAAA.07ae64718bad0f41/RA
X-App-Id: Custom.RestApi
X-App-Version: 15.0.155.64
CPUUtilization: 26.47529
RequestsPerSec: 0
W3WP-PrivateBytes: 7120
Date: Mon, 23 Aug 2021 21:30:55 GMT
Content-Length: 0

3. Check status of the extraction via job ID

...curl" -i -X GET -k -H "Content-Type: application/json" -H "Prefer: respond-async" "https://selectapi.datascope.refinitiv.com/RestApi/v1/Extractions/ExtractRawResult(ExtractionId='0x07ae64718b9d0f41')" -H "Authorization: Token MYVALIDTOKEN" -H 'Content-Type: application/json' -H 'X-Direct-Download: True' -x MYPROXY:8080

until it becomes status 200=completed

HTTP/1.1 200 OK
Cache-Control: no-cache
Pragma: no-cache
Content-Type: application/json; charset=utf-8
Expires: -1
Server: Microsoft-IIS/10.0
BeginRequestTime: 21:57:10.4224282
BeginRequestDate: 2021-08-23
X-Request-Execution-Correlation-Id: CiD/19945/AAAAAA.07ae85c312ed0f67/RA
X-App-Id: Custom.RestApi
X-App-Version: 15.0.155.64
CPUUtilization: 17.59716
RequestsPerSec: 7.619745
W3WP-PrivateBytes: 1569704
Date: Mon, 23 Aug 2021 21:57:10 GMT
Content-Length: 1414

4. Pull down the extraction result and store it into a file

... curl" -X GET "https://selectapi.datascope.refinitiv.com/RestApi/v1/Extractions/RawExtractionResults(';0x07ae64718b9d0f41')/$value" -H "Authorization: Token MYVALIDTOKEN" -H 'Content-Type: application/gzip' -H 'X-Direct-Download: True' -x MYVALIDPROXY:8080 -v -N -o "tst.csv.gz"

rahul.deshmukh

thanks @zoya faberov does this works for you ? Is it working example ? I will give try today and let you know about the result...

zoya faberov

Hello @rahul.deshmukh ,

I strongly recommend against using curl for production integration with RTH REST.

Curl is intended primarily for testing, debugging, and learning APIs capabilities.

I recommend integrating, using a programming language that supports HTTP REST, of your preference or RTH .NET SDK.

Postman can be used for automation, however, it is also targeted and intended as a testing tool. Allows you to test HTTP REST quickly and efficiently.

I would like to suggest to review our Programming without SDK Tutorial and .Net SDK Tutorials (with downloadable code examples) as working examples of programmatic integration, so you can make best design choices per your requirements, so hope you will find the tutorials very helpful.

rahul.deshmukh

@zoya faberov As we are using ODI as ETL tool where we are trying to use curl RTH REST for extracting the response. Is working with curl is not stable or have any performance issue with RTH REST ? I have seen in tutorial about the Python RTH code example here RTH Python and would like to know is it working sample example and can we try this for TickHistoryraw ? Is there any working example for TickHistoryRaw REST Api for Python ?

zoya faberov

Hello @rahul.deshmukh ,

Have you had a chance to review Programming Without SDK tutorials? They go over the recommended steps and link to RTH Downloads that includes RTH Python Code Samples.

I have just used this approach, to make a python example for TickHistoryRaw on demand:

RTH_OnDemand_Raw.20210827.zip

Please find it attached, replace the credentials with yours, and I hope you find this example useful

rahul.deshmukh

@zoya faberov thanks very much ...we are here new to Python but definately if it makes the enviornment more stable without performance issue then we would definately like to try the example which you have provided and use Python for RTH REST ...i will let you know in case i face any issue while using this code for RTH Rest....

rahul.deshmukh

@zoya faberov thanks the demo example is working in python environment...just few questions...

1) As we have 3000 Identifiers list what is the best practice in terms of performance, stability and without loss of data to send this Identifier request to the RTH REST ? Should we send the Identifier request one by one to the RTH REST or bulk Identifier request for example 500 Identifier request in one RTH REST request and then another 500 Identifier request and so on.. ?

2) In demo example i see comments in some parts where it says the example is for demo purpose only and in production it will create problem...what does this actually mean and where do we need to update the code ?

3) As we are expecting huge volume of data from RTH REST from the 3000 Identifier list, we are thinking to send either one by one Identifier request or bulk Identifier(500 in one request) to the RTH REST and then get the data from this REST, store the data in the csv zip file like given in demo example and then load the data into database table....And then do another Identifier request to the REST, just delete the previous csv zip file, fetch the data from REST and load into database table and the process goes on until all the 3000 Identifier request has been sent to the REST...Is this good approach considering Volumne of data and performance ?

EXPLORE OUR SITES