question

Upvotes
Accepted
5 0 0 3

RDP news with no payload

We have a python application using RDP API receiving news in json format. Most of the json objects received contain payload. For example

{
  "attributes": [
    {
      "domain": {
        "type": "string",
        "value": "story"
      }
    }
  ],
  "envelopeVersion": "1.0",
  "ecpMessageID": "urn:newsml:reuters.com:20230721:nBw70J7x7a:1",
  "sourceSeqNo": 1153992,
  "distributionSeqNo": 478052,
  "sourceTimestamp": "2023-07-21T10:50:00.131Z",
  "distributionTimestamp": "2023-07-21T10:50:03.243Z",
  "payloadVersion": "2.0",
  "subscriptionID": "535c1fcb-6756-439b-8fb9-f453ca89acc3",
  "payload": {
    "newsItem": {
      "_guid": "tag:reuters.com,2023-07-21:newsml_Bw70J7x7a",
      "itemMeta": {
...

But some json objects received do not contain payload. For example

{
  "attributes": [
    {
      "domain": {
        "type": "string",
        "value": "story"
      }
    }
  ],
  "envelopeVersion": "1.0",
  "ecpMessageID": "urn:newsml:reuters.com:20230721:nRSU8251Ga:1",
  "sourceSeqNo": 1146276,
  "distributionSeqNo": 477496,
  "sourceTimestamp": "2023-07-21T10:15:15.352Z",
  "distributionTimestamp": "2023-07-21T10:15:16.888Z",
  "payloadVersion": "2.0",
  "subscriptionID": "535c1fcb-6756-439b-8fb9-f453ca89acc3",
  "href": "https://api.refinitiv.com/file-store/v1/files/4bef-dcc1-89bf39f4-bf4f-79d00c6565d5/stream",
  "internalS3Url": "a204892-content-eds-alt-prod-use1/news-stories/024c/2023-07-21/acf01783-38e4-4ef2-b537-b339a94dcf93.json"
}


How to interpret such news items withput payload? Is there an API to get expanded news story?


#technology#contentnewsrdp
icon clock
10 |1500

Up to 2 attachments (including images) can be used with a maximum of 512.0 KiB each and 1.0 MiB total.

Upvotes
Accepted
14.5k 30 5 10

Hi @OneTick ,

Is /alerts/v1/news-stories-subscriptions the endpoint you're using?

If so, in it's reference page in the API doc, there's a paragraph explaining this

Downloading with Messages without "payload" field and it contains href link

Actually, we have maximum size limitation to deliver messages to user SQS queue that it's 180 KB and if message size is larger than this limit, we'll remove "payload" field and put the link into "href" fields as shown below

1. Here is example messages

   {
       "attributes": [{
               "attribute1": ["value1"]
           }, {
               "attribute2": ["value2"]
           }
       ],
       "envelopeVersion": "1.0",
       "ecpMessageID": "99d512f7-d4b3-4eb9-8ad3-01b0a0cded33",
       "sourceSeqNo": 123,
       "distributionSeqNo": 1,
       "distributionTimestamp": "2021-02-06T18:52:11.673Z",
       "payloadVersion": "1.0",
       "subscriptionID": "48e4f082-6993-445a-85fc-bbc2e7eaf553",
       "href": "https://api.ppe.refinitiv.com/file-store/beta1/files/42e5-0b2d-eb853c37-abd4-8c8ec6a60e56/stream",
       "internalS3Url": "a204892-content-eds-alt-dev-use1/news-stories/test/temp_large_msg_5xMB.json",
       "ecpID": "23644"
   }

2) Once you receive this message, you have 2 method to download this file as following below

2.1 Download this file from RDP API Playground via web-interface

2.1.1) APIDOCS GET File Stream API

2.1.2) From href, fileId will be located in the url. In this case, your fileId is 42e5-0b2d-eb853c37-abd4-8c8ec6a60e56

2.1.3) From No.1, please replace {fileId} in this url https://api.ppe.refinitiv.com/file-store/v1/files/{fileId}/stream and you'll get actual payload from the response as example below

{
    {
        "newsMessage": {
            "header": {
                "channel": [
                    {
                        "$": "TXT"
                    }
                ],
                "destination": [
                    {
                        "_qcode": "NP:BSW"
                    },
                    {
                        "_qcode": "NP:CNR"
                    }
                ],
                "priority": {
                    "$": 3
                },
                "sender": {
                    "$": "reuters.com"
                },
    .....................................................
    }
}

2.1.4) There are another option, if you would like to download file later you can specify with this url https://api.refinitiv.com/file-store/v1/files/{fileId}/stream?doNotRedirect=true and you'll receive s3 url and the link will be expired within 6 hours. You have to call this fileStream API again once it expire for getting new s3 url and example response is shown as following below

    {
        "url": "https://a204892-content-eds-alt-prod-use1.s3.amazonaws.com/news-stories/0025/2021-01-21/5e6247d7-e53d-41fb-a59a-81a82990c8f0.json?x-request-Id=2e7d4b2c-0aba-4d08-8750-3dec9d2570b2&x-package-id=4b74-070f-1558af6d-b1ec-3ade7455b05b&x-client-app-id=6bc155331c3f4c78bae8b0389c262ec17f2a9191&x-file-name=filesets-4bdbe568-052e-4c45-a73d-40dcfb4473c5-1612980871&x-fileset-id=45ce-0e39-1e51d321-877c-217289179fdc&x-bucket-name=protocol-alarm&x-uuid=GE-203&x-file-Id=4411-c086-cba648a7-8c0d-6c06529e8588&x-fileset-name=bus&x-event-external-name=cfs-claimCheck-download&X-Amz-Security-Token=IQoJb3JpZ2luX2VjELL%2F%2F%2F%2F%2F%2F%2F%2F%2F%2FwEaCXVzLWVhc3QtMSJHMEUCIQCA4Efh5JfLsAif5vTxhVW7fAzT%2BLKkf6f9Twxw0hutUwIgeQcYDvbwDk9O9jFErTTs30n2eHUGA5cZTLF40hs7We8q2AMIq%2F%2F%2F%2F%2F%2F%2F%2F%2F%2F%2FARACGgw2MDgwMTQ1MTUyODciDAZrUt4WYi%2FX1RH9QCqsAxtw%2ByGIsYzVsLSrnh%2BL3zFVP8VBVNXYHUwmo7LLhk6Hpi8oZIL38S5rWsbnTO%2BHdG9M5V%2FMBIYqyaRpY8Lt9iThpEEQxTa6Yfsgyc0gER4l2Bhh%2B4MJH8Z%2FrigL7z3us49r5dxgd5aepEWgixbutqFL5pQPUcrHcpukbVRbM9mk%2F9d82VkHnXX62gBFoNJk6fQM5Io7ycvdcDw%2Fqz70DXYpzOMvlx8NF9nSgrFklBw9L%2F9tKMOrnwioKqmW5FUdlgaH3COkAQBabxFVg2rMiaEOpVaOH9Fn%2F2qdZ%2B4q4OQhzEkYo2UaRN0s1Y26TotdGhk2zAkVHSIghvUBfWjzdBnAu7rV9xptWNJ7AyJbDjw0G3yQxU1nznHRqjZpMA7jv8KBU5b6j89IlV7arfzH6mw0c990ziO5tWNPCxIHn6cY%2BPgBOtcqUK%2BisPDTDaWlAxLvIVWYc1%2Bi0IPd7OuT6h5cG5sodyQsB%2FA%2B3kYHuaHBGZ%2FwBKwiLoVQ7f6qGgF8wkBuWXCk5w21Ety1DCPRSWtLkK94gvP4hMgNIc9jXZg4GyauDNGQZ4Em2%2FOwMMewkIEGOuwB7ScuAYjKxmcztZU2heFRheWByvVm20XkycUKNM%2B1mC5GpVXnP75YiNjXcngDedzeo%2Bv6yfmNo3upsTzqsdvNu%2Fl8bAK68xUJC6oiq3gmhJ6d3CYdkAUGcUElQ2VX7cVCrHAMn41lTIjx5l0iCPNeR6firb5m%2Bl6%2FZYFs1VX5mfHrxMjv3qvHoAv1h6HIoN4YJQKipq%2Fyh4auwW5ESA135i1ZRPpZzmKfjTcDbj0ha2jCNw9x5Wmg35u5x7cPZaVI4%2F%2FngCoqLVG9H3q3CRJSS%2FsD4NhoiXtmUTRk3kVDTrJ%2BA4cBebhtXegWkC0%3D&X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Date=20210210T191613Z&X-Amz-SignedHeaders=host&X-Amz-Expires=21599&X-Amz-Credential=ASIAY3ED5HRL2GZ4Y354%2F20210210%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Signature=1d535ea691fbff82121a56f7f7b4525d0cd247cc44f2841e8fd6b0199318268d"
    }

2.1.5) You can download file from "url" field directly

2.2 Download this file from RDP API Gateway

2.2.1) Login with your username and password to retrieve request token from RDP API Gateway via https://api.refinitiv.com/auth/oauth2/v1/token and please make sure you specify "trapi.cfs.subscriber.read" scope in your request as example below.

    - Headers --> Key: Content-Type  Value: application/x-www-form-urlencoded
    - requestBody --> grant_type=password&username=<USERNAME>&password=<PASSWORD>&scope=trapi.cfs.claimcheck.read trapi.cfs.subscriber.read&client_id=<CLIENT_ID>&takeExclusiveSignOnControl=true
    Note: If you don't have CLIENT_ID please generate via [AppKeyGenerator](https://apps.cp.thomsonreuters.com/apps/AppkeyGenerator).

2.2.2) Please make sure you found scope name "trapi.cfs.claimcheck.read" or "trapi.cfs.subscriber.read" as example response below

    {
       "access_token": "eyJ0eXAiOiJhdCtqd3QiLCJhbGciOiJSUzI1NiIsImtpZCI6IlNMVWxGQVlkZUhCb1dmMnplTG9lMTZKY2JrcVpXLU9hRVh2NTZsVkZ3MzAifQ.eyJkYXRhIjoie1wiY2lwaGVydGV4dFwiOlwiUlpvc1BKX0F2MnlWWUNKamtGTGhxR2liR1owZDRqVUNTU0VFamYxX2Y5cU1CcFh6NFphMWp3TFA3NzFraHdBOGZBdFh4WnZESVo5RXg5M0s5VTFsTmxfWjNMSi0tQTRCRjhwYm8tZjhHLWZBYklnS2NJR3JHSGdZbklEeWxWcGpJbnlIQm5jYnR1Y2pydWV3M1RBY0xPUlhaZmVxeWNSVkRPOHMzcHRIS01XbXRzdzk3WFBjd2hFbVF4YWR1UzZWcEV2ak1KMDNkT1dlV18zQWJDZkJqZmw0QkZQRjhpOHhnRGtKUTR3Wmo2WVVrdGNBWEY1SmZsbGtUZVQ3RVhUS09ZR0hKSlhILU9HVUdRYThDaEk0cy14ZUVsbDRtaHBpYm5xWjE1d3BubXNFQktqNS1ERENrUk1tZjVqMGJVUDZUYTJCdngwMjJiRjczQTh0VzhUYy0tZ0ZIdnlRbE0zMnk5ajJyQWFNS3p3b04xTVU5SmxPQnRVWWpKWmVrSUpWQlp1Tm5lYlc4RTNTUE5fMnlHVjFKUVljZmQ5aXJ5cXNWTmdzWDY5NDBIbjJnZ3dwZzRCTXA2N2ZEajNDUmdfT3hWZTc0c2xIYmkzWnNpS0djbDUzUUNtdm9oVm92c1U1bUdBXCIsXCJpdlwiOlwiWFB2NDAyeEswYWdKU1M3TFwiLFwicHJvdGVjdGVkXCI6XCJleUpoYkdjaU9pSkJWMU5mUlU1RFgxTkVTMTlCTWpVMklpd2laVzVqSWpvaVFUSTFOa2REVFNJc0lucHBjQ0k2SWtSRlJpSjlcIixcInJlY2lwaWVudHNcIjpbe1wiZW5jcnlwdGVkX2tleVwiOlwiQVFJQkFIamxWM2dlbGtxMGxibUhEckpsU3BQTGlLZTRqM2NaeDJLbXR3djBUdnFZTGdFbUxFamU3WGZ3a1pTd3N0Vk5nVndaQUFBQWZqQjhCZ2txaGtpRzl3MEJCd2FnYnpCdEFnRUFNR2dHQ1NxR1NJYjNEUUVIQVRBZUJnbGdoa2dCWlFNRUFTNHdFUVFNOEs3UzFBYUU2V00wWnQ4UUFnRVFnRHNHZjNVQy02SGpHazFFZ3J4dVhPbThVMm9VdzRVMTkwOFdibzZRUTdnQ3NRSUhTNWstbkpsajBXcG8yY0M4VFREQWZ5anhwNmgyQUxDVjhBXCIsXCJoZWFkZXJcIjp7XCJraWRcIjpcImFybjphd3M6a21zOmFwLXNvdXRoZWFzdC0xOjY1MzU1MTk3MDIxMDprZXkvNjJhMzY4MzUtYmNiNS00N2Y5LWJiMDUtNzk5MjA1N2MxNWM3XCJ9fSx7XCJlbmNyeXB0ZWRfa2V5XCI6XCJBUUlDQUhqOW9sZDI5RXB3VU1hZTZCSjRRVWVrOGZMQTlxQ2t3YlBKUFF1QV9tMEdwUUVMekthMWlZUXFvRy0zeGZ2aGJKTDJBQUFBZmpCOEJna3Foa2lHOXcwQkJ3YWdiekJ0QWdFQU1HZ0dDU3FHU0liM0RRRUhBVEFlQmdsZ2hrZ0JaUU1FQVM0d0VRUU1EM1dPT0NZdUxrcmZKcGxNQWdFUWdEc2gwWHVfb2kzNlptUFR6VFQ4X0loNTR2Y0dGS3J4UnVfTjNVS01zQ2w1a3Z0Yk1Vanl2c2ZxZTdzeTAyUlI0M2Q3ZWJlV1JfVmtEeDMwMXdcIixcImhlYWRlclwiOntcImtpZFwiOlwiYXJuOmF3czprbXM6dXMtZWFzdC0xOjY1MzU1MTk3MDIxMDprZXkvNjUwOGQzM2YtMjU4My00YjNmLTlkZjItYmQwZTQ0YzUxZTQzXCJ9fSx7XCJlbmNyeXB0ZWRfa2V5XCI6XCJBUUlDQUhoWEtRM2g5YU9NVThPX24tZy1ISV93SmYwTmRCRzc0RHpLSjAzVDk0T2d3UUcxZURVVmQ4TmhGS0c4aUdBM2RHbjRBQUFBZmpCOEJna3Foa2lHOXcwQkJ3YWdiekJ0QWdFQU1HZ0dDU3FHU0liM0RRRUhBVEFlQmdsZ2hrZ0JaUU1FQVM0d0VRUU1BT1NHV3RiUEEzOHNYQ2FDQWdFUWdEdEJBU1RSenpxd00wYUxQWnRTeEVoellkb3RZYWlxYWVDczlkSVRwV0hJTFdBbDBjOEZPNzhLVlVsZENzYmFqVmNvN0toWVVxeWZyZW1FbWdcIixcImhlYWRlclwiOntcImtpZFwiOlwiYXJuOmF3czprbXM6ZXUtd2VzdC0xOjY1MzU1MTk3MDIxMDprZXkvOGQwOTQzM2YtYWNmZS00MjQ0LWExYjktZTcwOTVjOGFkNTZlXCJ9fV0sXCJ0YWdcIjpcImFYeFA2d3RydlhQalFzcnNBZVdOWUFcIn0iLCJyczEiOiIwYmM3ZWE2NTUyM2UxZDY5MGIzODFjYWE3Y2Y2NTJlOWUxYjc5MjA3IiwiYXVkIjoiNmJjMTU1MzMxYzNmNGM3OGJhZThiMDM4OWMyNjJlYzE3ZjJhOTE5MSIsImlzcyI6Imh0dHBzOi8vZmluYW5jaWFsLnRob21zb25yZXV0ZXJzLmNvbS9hcGkvaWRlbnRpdHkvc3RzX3ByZXByb2QiLCJleHAiOjE2MTI5ODQ4NzAsImlhdCI6MTYxMjk4NDU3MH0.hJLO255l2Im7vwIzf2H2Bt8iToeJ8wIWpc7DavIkWNLTzQFsJEHNQOsTlnC5dUtvMFzKBd2DoXGTJO1Oll42pQnf5knD7HuQLK-9pCND1jtt1_t_0q0qnVjIce_050pm-wn8giLkGZsdE1ZNXBIEU2rPgQqpg2yjF7ADXKRIMcXfqDBkuHzMvvx_X_cGDaroTUcBvT3iC1hxFMmgBNK56rV_uaHTuPM9LbP1mpdn_27z7dHmCVB-tAaBsf_OBSYDvxfIIdikYVf69a_1t4QN6rlbvH9OstTXVrkGtUYrGvDb9kUAzzxQbIhjn5zlFkJGC29-dcWSMug5tUDAppTC8w",
       "refresh_token": "aa02622a-dd18-431d-937e-01f63ced29df",
       "expires_in": "300",
       "scope": "trapi.cfs.claimcheck.read trapi.cfs.subscriber.read ....",
       "token_type": "Bearer"
    }
 Note: Please keep refresh_token to renew expired token and access_token for downloading actual payload via file stream API

2.2.3) When token is expired please retrieve token from RDP API Gateway as same as 2.2.1 but it's different request body as shown below

- Headers --> Key: Content-Type  Value: application/x-www-form-urlencoded
- requestBody --> grant_type=refresh_token&username=<USERNAME>&refresh_token=<REFRESH_TOKEN>&client_id=<CLIENT_ID>
Note: it's more secure and more comfortable that you don't have to supply password and scope every time you retrieve token. However, if you could not use REFRESH_TOKEN to retrieve a new one please use ACCESS_TOKEN by follow 2.2.1 instead

2.2.4) Result should be similar with 2.2.2

2.2.5) Please check "href" field from messages that you receive and you can choose whether you want it redirects to download actual file or download it later by keeping s3 url from 2 url as following below

- Redirect --> https://api.ppe.refinitiv.com/file-store/v1/files/{fileId}/stream
- Do not redirect --> https://api.ppe.refinitiv.com/file-store/v1/files/{fileId}/stream?doNotRedirect=true
For example results please check 2.1.3 and 2.1.4. Here's request information below 
- Headers --> Authorization: "Bearer <ACCESS_TOKEN>"  
icon clock
10 |1500

Up to 2 attachments (including images) can be used with a maximum of 512.0 KiB each and 1.0 MiB total.

Upvotes
5 0 0 2

Hi Raksina,

Thanks for the detailed response!
We are using endpoint https://api.refinitiv.com/message-services/v1/news-stories/subscriptions

Is the sqs limit counted on actual json?

For example, if full story json containing inlineData fields is over 180K, I understand we will get json without payload. But if headline json (without inlineData fields) for same news story is less than 180K, will we get the headline json with headline subscription? ie. with endpoint https://api.refinitiv.com/message-services/v1/news-headlines/subscriptions

Regards
-Makarand Vaidya

icon clock
10 |1500

Up to 2 attachments (including images) can be used with a maximum of 512.0 KiB each and 1.0 MiB total.

Upvotes
14.5k 30 5 10

Hi @makarand.vaidya ,

The moderatorson this forum are expertise on Refinitiv APIs usage. However, they do not havedeep expertise in every type of content available through Refinitiv products.Such expertise is available through Refinitiv Helpdesk, which can be reachedvia MyRefinitiv. To be of help, ticket number 12819914 was raised on yourbehalf and the support team is going to contact you soon to assist with this.

Hope this helpsand please let me know in case you have any further questions

icon clock
10 |1500

Up to 2 attachments (including images) can be used with a maximum of 512.0 KiB each and 1.0 MiB total.

Write an Answer

Hint: Notify or tag a user in this post by typing @username.

Up to 2 attachments (including images) can be used with a maximum of 512.0 KiB each and 1.0 MiB total.