TickHistoryRawExtractionRequest REST API throws an error using Curl
I have followed below steps using curl:
1) get the authentication token which is working and generated token:
OdiOSCommand "-OUT_FILE=/home/oracle/Documents/OutFile/refinitiv_file.txt" "-CAPTURE_OUT_STREAM=ON_ERROR,10,50" "-CAPTURE_ERR_STREAM=ON_ERROR,10,50"
curl -H "Content-Type: application/json" -H "Prefer: respond-async" -X POST -v -d "{\"Credentials\": {\"Username\": \"501\",\"Password\": \"xxxxxx\"}}" https://selectapi.datascope.refinitiv.com/RestApi/v1/Authentication/RequestToken
2) Use this authentication token in the below curl request:
OdiOSCommand "-OUT_FILE=/home/oracle/Documents/OutFile/refinitiv_file_extraction_response_json.txt" "-CAPTURE_OUT_STREAM=ON_ERROR,10,50" "-CAPTURE_ERR_STREAM=ON_ERROR,10,50"
curl -X POST \
https://selectapi.datascope.refinitiv.com/RestApi/v1/Extractions/ExtractRaw \
-H 'Authorization: Token __7OGZ1r1Hp4HaTam98Gp9kmovQlNv8NeKEHe6boyIRJXnzNmdLSvnfJXcHysqu9JCgX6PH9nk30671SwRb1ol_dj9oG46BY3mG6JGMMNs17J_TZ7tP9RL2Y2q-6KaqL63EjjfWt9Bm-5usuI3alHDy5Q_AwVXoSW140O8x3AwNfhvOSUpxAllzXidWNXn27yfgHrMm28ZjKEycVqJOwfztmcr75tHj9ijvHogyaXzmiKpPv2sxc45aqEW2KnrKASN1zEhzw3ICrtDBm4Xhx9TzL2TB-XpBPDDWpaUi4jFdU' \
-H 'Content-Type: application/json' \
-d '{
"ExtractionRequest": {
"@odata.type": "#DataScope.Select.Api.Extractions.ExtractionRequests.TickHistoryRawExtractionRequest",
"ContentFieldNames": [
],
"IdentifierList": {
"@odata.type": "#DataScope.Select.Api.Extractions.ExtractionRequests.InstrumentIdentifierList",
"InstrumentIdentifiers": [
{"Identifier":"IBM.N","IdentifierType":"Ric"},{"Identifier":"CARR.PA","IdentifierType":"Ric"},{"Identifier":"BEGRs.TQE","IdentifierType":"Ric"},{"Identifier":"US1156371007.GTX","IdentifierType":"Ric"},{"Identifier":"JP3249600002.GTX","IdentifierType":"Ric"}
]
},
"Condition": {
"MessageTimeStampIn": "GmtUtc",
"ReportDateRangeType": "Range",
"QueryStartDate": "2021-08-20T00:00:00.000Z",
"QueryEndDate": "2021-08-23T00:00:00.000Z",
"Fids": "3,5,12,13,15,16,19,22,23,26,27,4196,5632,5633,2130",
"DisplaySourceRIC": true
}
}
}'
The output response for the above request is as below:
{"@odata.context":"https://hosted.datascopeapi.reuters.com/RestApi/v1/$metadata#RawExtractionResults/$entity","JobId":"0x07ae207dfead0ec8","Notes":["Extraction Services Version 15.0.42358 (01a7f7ea050d), Built May 20 2021 18:20:45\nUser ID: 7701\nExtraction ID: 2000000296050738\nSchedule: 0x07ae207dfead0ec8 (ID = 0x0000000000000000)\nInput List (1 items): (ID = 0x07ae207dfead0ec8) Created: 23.08.2021 15:58:07 Last Modified: 23.08.2021 15:58:07\nReport Template: _OnD_0x07ae207dfead0ec8 (ID = 0x07ae207dfecd0ec8) Created: 23.08.2021 15:56:55 Last Modified: 23.08.2021 15:56:55\nSchedule dispatched via message queue (0x07ae207dfead0ec8), Data source identifier (8A3F4C373BA140F98836FA4F5B502FC8)\nSchedule Time: 23.08.2021 15:56:56\nProcessing started at 23.08.2021 15:56:56\nProcessing completed successfully at 23.08.2021 15:58:07\nExtraction finished at 23.08.2021 13:58:07 UTC, with servers: tm02n02, TRTH (41.83 secs)\nInstrument <RIC,IBM.N> expanded to 1 RIC: IBM.N.\nTotal instruments after instrument expansion = 1\n\nManifest: #RIC,Domain,Start,End,Status,Count\nManifest: IBM.N,Market Price,,,Inactive,0\n"]}
3) This step i am not clear yet. In this step i tried to get the JobID and also previous token i have used from step 1 and use it another curl request to get the response values for the fids:
OdiOSCommand "-OUT_FILE=/home/oracle/Documents/OutFile/refinitiv_file_extraction_response_json_values.txt" "-CAPTURE_OUT_STREAM=ON_ERROR,10,50" "-CAPTURE_ERR_STREAM=ON_ERROR,10,50"
curl -X GET \
https://selectapi.datascope.refinitiv.com/RestApi/v1/Extractions/RawExtractionResults('0x07ae2e231f1d0eeb')/$value \
-H 'Authorization: Token __7OGZ1r1Hp4HaTam98Gp9kmovQlNv8NeKEHe6boyIRJXnzNmdLSvnfJXcHysqu9JCgX6PH9nk30671SwRb1ol_dj9oG46BY3mG6JGMMNs17J_TZ7tP9RL2Y2q-6KaqL63EjjfWt9Bm-5usuI3alHDy5Q_AwVXoSW140O8x3AwNfhvOSUpxAllzXidWNXn27yfgHrMm28ZjKEycVqJOwfztmcr75tHj9ijvHogyaXzmiKpPv2sxc45aqEW2KnrKASN1zEhzw3ICrtDBm4Xhx9TzL2TB-XpBPDDWpaUi4jFdU' \
-H 'Content-Type: application/gzip' \
-H 'X-Direct-Download: True'
And this step is throwing an error as below:
ODI-1226: Step Copy of Refinitiv_RESTAPI_REQUEST fails after 1 attempt(s).
ODI-1241: Oracle Data Integrator tool execution fails.
Caused By: com.sunopsis.dwg.function.SnpsFunctionBaseException: ODI-30038: OS command returned 1. Error details are [sh: -c: line 1: syntax error near unexpected token `('
sh: -c: line 1: ` https://selectapi.datascope.refinitiv.com/RestApi/v1/Extractions/RawExtractionResults('0x07ae2e231f1d0eeb')/$value \'
].
at com.sunopsis.dwg.tools.OSCommand.actionExecute(OSCommand.java:535)
at com.sunopsis.dwg.function.SnpsFunctionBaseRepositoryConnected.execute(SnpsFunctionBaseRepositoryConnected.java:235)
at oracle.odi.runtime.agent.execution.SessionTask.execIntegratedFunction(SessionTask.java:949)
at oracle.odi.runtime.agent.execution.SessionTask.executeOdiCommand(SessionTask.java:607)
at oracle.odi.runtime.agent.execution.cmd.OdiCommandExecutor.execute(OdiCommandExecutor.java:32)
at oracle.odi.runtime.agent.execution.cmd.OdiCommandExecutor.execute(OdiCommandExecutor.java:18)
at oracle.odi.runtime.agent.execution.TaskExecutionHandler.handleTask(TaskExecutionHandler.java:52)
at oracle.odi.runtime.agent.execution.SessionTask.processTask(SessionTask.java:216)
at oracle.odi.runtime.agent.execution.SessionTask.doExecuteTask(SessionTask.java:128)
at oracle.odi.runtime.agent.execution.AbstractSessionTask.execute(AbstractSessionTask.java:886)
at oracle.odi.runtime.agent.execution.SessionExecutor$SerialTrain.runTasks(SessionExecutor.java:2225)
at oracle.odi.runtime.agent.execution.SessionExecutor.executeSession(SessionExecutor.java:610)
at oracle.odi.runtime.agent.processor.TaskExecutorAgentRequestProcessor$1.doAction(TaskExecutorAgentRequestProcessor.java:718)
at oracle.odi.runtime.agent.processor.TaskExecutorAgentRequestProcessor$1.doAction(TaskExecutorAgentRequestProcessor.java:611)
at oracle.odi.core.persistence.dwgobject.DwgObjectTemplate.execute(DwgObjectTemplate.java:203)
at oracle.odi.runtime.agent.processor.TaskExecutorAgentRequestProcessor.doProcessStartAgentTask(TaskExecutorAgentRequestProcessor.java:800)
at oracle.odi.runtime.agent.processor.impl.StartSessRequestProcessor.access$1400(StartSessRequestProcessor.java:74)
at oracle.odi.runtime.agent.processor.impl.StartSessRequestProcessor$StartSessTask.doExecute(StartSessRequestProcessor.java:702)
at oracle.odi.runtime.agent.processor.task.AgentTask.execute(AgentTask.java:180)
at oracle.odi.runtime.agent.support.DefaultAgentTaskExecutor$2.run(DefaultAgentTaskExecutor.java:108)
at java.lang.Thread.run(Thread.java:748)
Hello @rahul.deshmukh ,
Working with RTH REST via curl is stable. However, curl is a test tool, rather then a programming language. Requests via curl returns a lot of verbose output that you have indicated you would prefer not to have.
Have you had a chance to review Programming Without SDK tutorials? They go over the recommended steps and link to RTH Downloads that includes RTH Python Code Samples.
I would suggest using On-Demand examples( please see tutorials for the complete information on On-Demand and Scheduled/GUI) and replace the request in the example (IntradayBars request) with your required RTH request (TickHistoryRaw).
I have just used this approach, to make a python example for TickHistoryRaw on demand:
Please find it attached, replace the credentials with yours, and I hope you find this example useful
2) In demo example i see comments in some parts where it says the example is for demo purpose only and in production it will create problem...what does this actually mean and where do we need to update the code ?
3) As we are expecting huge volume of data from RTH REST from the 3000 Identifier list, we are thinking to send either one by one Identifier request or bulk Identifier(500 in one request) to the RTH REST and then get the data from this REST, store the data in the csv zip file like given in demo example and then load the data into database table....And then do another Identifier request to the REST, just delete the previous csv zip file, fetch the data from REST and load into database table and the process goes on until all the 3000 Identifier request has been sent to the REST...Is this good approach considering Volumne of data and performance ?
Hello @rahul.deshmukh ,
There was a mandatory migration of the endpoints, for DSS and RTH,
Please find the requirement details on DSS API front page and more in-depth from RTH API front page
The tutorial on how to migrate that should be helpful can be found at Refinitiv Datascope Select API Migration Tutorial, this information is relevant for both RTH and DSS migration requirements.
@zoya.farberov Thanks now i modified as per the migration, there is no error now but i am not getting the correct response for fields:
Below is the response:
{"@odata.context":"https://selectapi.datascope.refinitiv.com/RestApi/v1/$metadata#RawExtractionResults/$entity","JobId":"0x07ae62f53ead0f40","Notes":["Extraction Services Version 15.0.42358 (01a7f7ea050d), Built May 20 2021 18:20:45\nUser ID: 9001\nExtraction ID: 2000000296068231\nSchedule: 0x07ae62f53ead0f40 (ID = 0x0000000000000000)\nInput List (2 items): (ID = 0x07ae62f53ead0f40) Created: 23.08.2021 17:08:43 Last Modified: 23.08.2021 17:08:43\nReport Template: _OnD_0x07ae62f53ead0f40 (ID = 0x07ae62f53ecd0f40) Created: 23.08.2021 17:07:40 Last Modified: 23.08.2021 17:07:40\nSchedule dispatched via message queue (0x07ae62f53ead0f40), Data source identifier (DED82B93BB234FF882B35BEE6E2510B3)\nSchedule Time: 23.08.2021 17:07:40\nProcessing started at 23.08.2021 17:07:40\nProcessing completed successfully at 23.08.2021 17:08:43\nExtraction finished at 23.08.2021 15:08:43 UTC, with servers: tm03n03, TRTH (49.021 secs)\nInstrument <RIC,IBM.N> expanded to 1 RIC: IBM.N.\nInstrument <RIC,CARR.PA> expanded to 1 RIC: CARR.PA.\nTotal instruments after instrument expansion = 2\n\nManifest: #RIC,Domain,Start,End,Status,Count\nManifest: CARR.PA,Market Price,,,Inactive,0\nManifest: IBM.N,Market Price,,,Inactive,0\n"]}
I am just doing some tests initailly with some Rics and fids to check if i am getting response from this RAW request. Is there anything wrong with the fids or Rics or do i need to modify condition or any parameters ? I just need to see response with the values correctly...Why its showing inactive in response...i have edited my question and updated the extraction request which i am using..
Hello @rahul.deshmukh ,
Let me just confirm, you are doing all steps via curl as per
REST API Tutorial 8: On Demand raw data extraction
1. Authenticating and obtaining a valid token
2. Issuing request with valid token => getting back a submitted jobId/ExtractionID
3. Checking request status on the job ID => confirming it is completed = 200 ( not accepted = 202). And if status=202, checking until status is completed = 200.
4. Once the status becomes 200, extracting result via { {protocol}}{ {host}}{ {api}}Extractions/RawExtractionResults('{ {jobId}}')/$value
This sequence of steps, as described in detail in the tutorial for Postman, will need to be done via Postman, or via curl, or implemented in any language supporting HTTP REST.
?
@zoya.farberov i did step 1 and2 but not have done step 3 and directly proceed for step 4 where i provide token and my extraction request...all this steps i have done using curl...i already used curl with the same steps for FundAllocationExtractionRequest rest api and it works fine...i am doing the tick history raw request first time so not sure if i miss anything during extraction request...
The step 3 is required, to be sure that the job has completed, prior to extracting the result.
The processing is asynchronous, and the time it will take to process may differ based on the job size, and service load at the the time.
If you do not check status, you can not be confident that your result is ready when you extract it.
Please try to check the status, and only extract when the result is ready?
@zoya.farberov i dont understand how to do implement step 3...during implementation its going to be difficult...i think without step 3 also we can run the extraction request and get the reponse as mentioned in the tutorial this step is optional...the REST API Reference Tree is not yet updated and migrated and its still old...
Hello @rahul.deshmukh ,
I have just tested the following step 4 curl, on a completed TickHistoryRaw result with status = 200
"curl-7.70.0-win64-mingw\bin\curl" -X GET "https://selectapi.datascope.refinitiv.com/RestApi/v1/Extractions/RawExtractionResults('MYJOBID')/$value" -H "Authorization: Token MYVALIDTOKEN" -H 'Content-Type: application/gzip' -H 'X-Direct-Download: True' -x MyProxy:8080 -v -N -o "tst.csv.gz"
I got the result downloaded, as expected.
Hope this information helps
@zoya.farberov can you please tell how do i find 'MYJOBID' in my response ? And unlike other Rest API this looks like new step for Raw extraction Request for tick history ? This makes the implementaion very difficult when we try to automate our programming script...because we have to put token and now 'MYJOBID' in the extraction request ? Your GET request looks different than i used in my code: "https://selectapi.datascope.refinitiv.com/RestApi/v1/Extractions/RawExtractionResults('MYJOBID')/$value"
Also does this JobID change everytime like token for each request?
You assigned jobId comes back in the response in step 2 above. Do you have it?
I think working through tutorial REST API Tutorial 8: On Demand raw data extraction may be helpful to you.
Postman tool is free, you will be able to download it from https://www.postman.com/downloads/, also by downloading our working request collection that comes with REST tutorials, you will be able to run this and other requests included with the collection.
@zoya.farberov i have jobID.. you can also see in the reponse which i am getting back...can you please help how to use Postman and the link which you share is not working...
In the first step i generated token
In second step i used this token in my extraction request which i have mentioned in the question
In third step again i have to use curl request with the JobID like you mentioned in your curl to get the response back with fids value ?
To be honest Tutorial is bit confusing for me and not understandable and not all the steps are mentioned with the actual complete request...So i am trying to figure out the soltuion with my example..Is it possible for you just to use my request in question and provide soultion ...thanks...so that i can follow the same step..
Hello @rahul.deshmukh ,
Steps that we use to extract TickHistoryRaw content with curl:
1. Authenticate and obtain a valid token
...curl" -i -X POST -k -H "Content-Type: application/json" -H "Prefer: respond-async" https://selectapi.datascope.refinitiv.com/RestApi/v1/Authentication/RequestToken -H 'X-Direct-Download: True' -x MYVALIDPROXY:8080 -d @creds.txt
where creds.txt file is located in the same directory and contains
{
"Credentials": {
"Username": "MYVALIDDSSUSERID",
"Password": "MYVALIDDSSPASSWORD"
}
}
Resulting in
{"@odata.context":"https://selectapi.datascope.refinitiv.com/RestApi/v1/$metadata#Edm.String","value":"MYVALIDTOKEN"}HTTP/1.1 404 Not Found
Cache-Control: no-cache
Pragma: no-cache
Content-Type: text/html; charset=utf-8
Proxy-Connection: Keep-Alive
Connection: Keep-Alive
Content-Length: 615
2. Use the valid token to create TickHistoryRaw extraction request:
...\curl-7.70.0-win64-mingw\bin\curl" -i -X POST -k -H "Content-Type: application/json" -H "Prefer: respond-async" "https://selectapi.datascope.refinitiv.com/RestApi/v1/Extractions/ExtractRaw" -H "Authorization: Token MYVALIDTOKEN" -H 'Content-Type: application/json' -H 'X-Direct-Download: True' -x MYVALIDPROXY:8080 -d @requestBody.txt HTTP/1.1 200 Connection established
Contents of requestBody.txt file saved into the same directory
{
"ExtractionRequest": {
"@odata.type": "#DataScope.Select.Api.Extractions.ExtractionRequests.TickHistoryRawExtractionRequest",
"IdentifierList": {
"@odata.type": "#DataScope.Select.Api.Extractions.ExtractionRequests.InstrumentIdentifierList",
"InstrumentIdentifiers": [{
"Identifier": "CARR.PA",
"IdentifierType": "Ric"
}]
},
"Condition": {
"MessageTimeStampIn": "GmtUtc",
"ReportDateRangeType": "Range",
"QueryStartDate": "2016-09-29T12:00:00.000Z",
"QueryEndDate": "2016-09-29T12:10:00.000Z",
"ExtractBy": "Ric",
"SortBy": "SingleByRic",
"DomainCode": "MarketPrice",
"DisplaySourceRIC": true
}
}
}
obtaining Extraction ID and status:
HTTP/1.1 202 Accepted Cache-Control: no-cache Pragma: no-cache Expires: -1 Location: https://selectapi.datascope.refinitiv.com/RestApi/v1/Extractions/ExtractRawResult(ExtractionId='0x07ae64718b9d0f41') Server: Microsoft-IIS/10.0 BeginRequestTime: 21:30:25.3627291 BeginRequestDate: 2021-08-23 Status: InProgress Progress: X-Request-Execution-Correlation-Id: CiD/19945/AAAAAA.07ae64718bad0f41/RA X-App-Id: Custom.RestApi X-App-Version: 15.0.155.64 CPUUtilization: 26.47529 RequestsPerSec: 0 W3WP-PrivateBytes: 7120 Date: Mon, 23 Aug 2021 21:30:55 GMT Content-Length: 0
3. Check status of the extraction via job ID
...curl" -i -X GET -k -H "Content-Type: application/json" -H "Prefer: respond-async" "https://selectapi.datascope.refinitiv.com/RestApi/v1/Extractions/ExtractRawResult(ExtractionId='0x07ae64718b9d0f41')" -H "Authorization: Token MYVALIDTOKEN" -H 'Content-Type: application/json' -H 'X-Direct-Download: True' -x MYPROXY:8080
until it becomes status 200=completed
HTTP/1.1 200 OK Cache-Control: no-cache Pragma: no-cache Content-Type: application/json; charset=utf-8 Expires: -1 Server: Microsoft-IIS/10.0 BeginRequestTime: 21:57:10.4224282 BeginRequestDate: 2021-08-23 X-Request-Execution-Correlation-Id: CiD/19945/AAAAAA.07ae85c312ed0f67/RA X-App-Id: Custom.RestApi X-App-Version: 15.0.155.64 CPUUtilization: 17.59716 RequestsPerSec: 7.619745 W3WP-PrivateBytes: 1569704 Date: Mon, 23 Aug 2021 21:57:10 GMT Content-Length: 1414
4. Pull down the extraction result and store it into a file
... curl" -X GET "https://selectapi.datascope.refinitiv.com/RestApi/v1/Extractions/RawExtractionResults('0x07ae64718b9d0f41')/$value" -H "Authorization: Token MYVALIDTOKEN" -H 'Content-Type: application/gzip' -H 'X-Direct-Download: True' -x MYVALIDPROXY:8080 -v -N -o "tst.csv.gz"
This is a fully working example
@zoya.farberov i am able to generate token in step1. But when i use your second step i am getting error now:
Below is my script:
curl -i -X POST -k -H "Content-Type: application/json" -H "Prefer: respond-async" "https://selectapi.datascope.refinitiv.com/RestApi/v1/Extractions/ExtractRaw" -H "Authorization: Token MYTOKEN" -H 'Content-Type: application/json' -H 'X-Direct-Download: True' -x MYPROXY:8080 -d {
"ExtractionRequest": {
"@odata.type": "#DataScope.Select.Api.Extractions.ExtractionRequests.TickHistoryRawExtractionRequest",
"IdentifierList": {
"@odata.type": "#DataScope.Select.Api.Extractions.ExtractionRequests.InstrumentIdentifierList",
"InstrumentIdentifiers": [{
"Identifier": "CARR.PA",
"IdentifierType": "Ric"
}]
},
"Condition": {
"MessageTimeStampIn": "GmtUtc",
"ReportDateRangeType": "Range",
"QueryStartDate": "2016-09-29T12:00:00.000Z",
"QueryEndDate": "2016-09-29T12:10:00.000Z",
"ExtractBy": "Ric",
"SortBy": "SingleByRic",
"DomainCode": "MarketPrice",
"DisplaySourceRIC": true
}
}
}
What mistake i did ?
Hello @rahul.deshmukh ,
I strongly recommend against using curl for production integration with RTH REST.
Curl is intended primarily for testing, debugging, and learning APIs capabilities.
I recommend integrating, using a programming language that supports HTTP REST, of your preference or RTH .NET SDK.
Postman can be used for automation, however, it is also targeted and intended as a testing tool. Allows you to test HTTP REST quickly and efficiently.
I would like to suggest to review our Programming without SDK Tutorial and .Net SDK Tutorials (with downloadable code examples) as working examples of programmatic integration, so you can make best design choices per your requirements, so hope you will find the tutorials very helpful.
@zoya.farberov As we are using ODI as ETL tool where we are trying to use curl RTH REST for extracting the response. Is working with curl is not stable or have any performance issue with RTH REST ? I have seen in tutorial about the Python RTH code example here RTH Python and would like to know is it working sample example and can we try this for TickHistoryraw ? Is there any working example for TickHistoryRaw REST Api for Python ?