Upgrade from Eikon -> Workspace. Learn about programming differences.

For a deeper look into our Eikon Data API, look into:

Overview |  Quickstart |  Documentation |  Downloads |  Tutorials |  Articles

question

Upvotes
Accepted
27 2 2 6

Performance Issues With Fetching News Story in Real-Time

This is related to my previous issue found here:
https://community.developers.refinitiv.com/questions/106792/fetching-newsstory-via-api-not-working-cnet-framew.html. After 14 days of going back and forth with the Refinitiv Helpdesk to no avail, they ultimately just pointed me back to the Developer's Forum.


I'm trying to fetch the news story of an incoming news item in real-time using the Pricing Library's streaming mechanism. First, I'm filtering for the main newswires such as Accesswire, Business Wire, Globe Newswire, and PR Newswire, and I've reversed engineered the storyID from the PNAC field as such:

var stream = Pricing.Definition().Universe("NFCP_UBMS").GetStream()
                              .OnUpdate((item, update, s) => DisplayUpdate(item, update))
                              .OnError((item, err, s) => Console.WriteLine(err));

private void DisplayUpdate(string item, JObject update)
{
   var fields = update["Fields"];
   string newsSource = (string)fields["ATTRIBTN"];
   string PNAC = (string)fields["PNAC"];
      if ((newsSource == "ACSWIR" || newsSource == "BSW" || newsSource == "GNW" || newsSource == "PRN"))
      {
         string storyID = "urn:newsml:reuters.com:" + DateTime.Now.ToString("yyyyMMdd") + ":" + PNAC + ":1";
        getNewsStoryBody(storyID);
      }
}


I then use the storyID to fetch the news body as such:

        public async void getNewsStoryBody(string storyID)
        {
            if (DateTime.Now.AddMilliseconds(-5000) > newsRequestStartTime) // Took too long due to multiple requests
            {
                Console.WriteLine("Could not fetch news body from server");
                return;
            }

            await Task.Run(() =>
            {
                    var story = Story.Definition(storyID).GetData();

                    if (story.IsSuccess)
                    {
                        Console.WriteLine("Retrieved news body: " + story.Data.NewsStory);
                    }
                    else
                    {
                        if ((string)story.HttpStatus["HTTPReason"] == "Not Found")
                        {
                            getNewsStoryBody(storyID);
                            //Have to recursively call this function, because the news body often cannot be found for multiple seconds after the headline is picked up in the live news stream
                        }
                    }
            });
        }

The problem is that the story will not be retrievable from the server ("Story Not Found" error) for at least 2500ms after the headline is available, which is why in getNewsStoryBody(), I need to recursively fetch the story. This issue is present only in real-time news items (the server response time is much faster for existing historical news with a response time of ~500ms or less), which leads me to the conclusion that the Refinitiv servers has performance issues in making news items available as they come in. Based on empirical observations, real-time news items with a larger body take longer to be available for retrieval by the server than ones with shorter bodies. Historical news items have no observable differences in response time based on body length, which again points to performance issues of Refinitiv's backend in processing news items for retrieval in real-time.

eikon-data-apirefinitiv-dataplatform-eikon#product#contentnews
icon clock
10 |1500

Up to 2 attachments (including images) can be used with a maximum of 512.0 KiB each and 1.0 MiB total.

Upvotes
Accepted
82.9k 281 53 77

@cmendolsohn

I will call the source of NFCP_UBMS as real-time streaming news and call the source of get_headlines and get_story methods as news feed. I would to say that they are separated systems.

I don't have informaion about how news feed handles or processes news. You may need to contact the news support team to verify if there is a delay in news feed used by Eikon comparing to real-time streaming news. As I know, Real-time streaming news doesn't have database so it can't provide historical news. On the other hand, news feed can provide historical news.

In the current implementation, you are using the old method to consume real-time news. Previously, we supported the N2_UBMS RIC that provide real-time news headlines (similarly to NFCP_UBMS). Each headline had a PNAC and we could use a PNAC to get news story by subscribing it through the real-time feed. However, currently, we can't get N2_UBMS or PNAC from the real-time feed. This method has been replaced with MRN (Machine Readable News).

Yes, you can retreive both headlines and stories from MRN. The following is the raw sample story retrieved from MRN.

{
  "altId": "nNDL1qs8mx",
  "audiences": [
    "NP:PUBTA",
    "NP:PUBTF"
  ],
  "body": "\nBisnis.com, JAKARTA - The banking sector has begun to increase its capital\nexpenditure (capex) d
...
...",
  "firstCreated": "2023-07-27T04:28:09Z",
  "headline": "PERBANAS - Prone to Cyber Attacks, Banking Sector Increases Capex for IT Investment",
  "id": "NDL1qs8mx_2307272B9MWFN2CohcWde8B9qr53EH5Ttyi+X8LQiocF",
  "instancesOf": [],
  "language": "en",
  "messageType": 2,
  "mimeType": "text/plain",
  "provider": "NS:PUBT",
  "pubStatus": "stat:usable",
  "subjects": [
    "B:125",
    "B:126",
    "B:127",
    "B:161",
    "B:172",
    "B:261",
    "B:262",
    "B:278",
    "E:5",
    "G:1",
    "G:25",
    "G:2EK",
    "G:BN",
    "G:CI",
    "G:K",
    "G:S",
    "M:1P2",
    "M:1QD",
    "M:1WN",
    "M:28T",
    "M:2CM",
    "M:2CN",
...
  ],
  "takeSequence": 1,
  "urgency": 3,
  "versionCreated": "2023-07-27T04:28:09Z"
}

With the RD .NET libraray, you will get the stream of news stories via the callback.

1690432317387.png

You can contact your Refinitiv account team or sales team to discuss the requirement. They can recommend the product that can suite the requirement.





1690432317387.png (32.3 KiB)
icon clock
10 |1500

Up to 2 attachments (including images) can be used with a maximum of 512.0 KiB each and 1.0 MiB total.

Upvote
82.9k 281 53 77

@cmendolsohn

Thank you for reaching out to us.

It looks like the application is fetching news headlines and news stories from two different systems.

Headlines and PNACs are from the real-time feed while stories are from the news story endpoint (via Eikon) by using story IDs created from PNACs. This method works but I assume that it may take sometimes for the news service used by Eikon to convert PNACs to storyIDs and create news stories in HTML format. Moreover, typically, the application gets storyIDs from the news headline endpoint, not from the real-time feed.

To get new stories from the real-time feed, you need to use Machine Readable News (MRN). However, you need to have a Refinitiv Data Platform's account. You can refer to the 2.3.06-News-MRN example.

icon clock
10 |1500

Up to 2 attachments (including images) can be used with a maximum of 512.0 KiB each and 1.0 MiB total.

Thanks for the response @Jirapongse . Just to confirm: you are saying that without MRN, ~2500ms response time in fetching a news story is as good as the Eikon system can perform?

Also, what is the difference between the Pricing library's real-time feed (my current implementation) vs the MRN real-time feed, when it comes to headline-only delivery? Any workflow/performance differences between the 2 for headline-only delivery?

Write an Answer

Hint: Notify or tag a user in this post by typing @username.

Up to 2 attachments (including images) can be used with a maximum of 512.0 KiB each and 1.0 MiB total.