Replicate GOVSRCH search results using Search API

For an internal tool, I want to copy the results of arbitrary GOVSRCH searches by copying the identifiers (e.g. RIC/ISIN) of each search result and retrieving each result using the Search Lookup API (i.e. `/api/rdp/discovery/search/v1/lookup`). Exporting the search results as Excel is not an option. I would use the JSON-based query export, but that's impossible due to this bug https://community.developers.refinitiv.com/questions/101947/govsrch-produces-malformed-json-queries-when-using.html">https://community.developers.refinitiv.com/questions/101947/govsrch-produces-malformed-json-queries-when-using.html

I would like to retrieve the bonds in the search results using their primary key. However, I noticed that some bonds in the list don't have an ISIN, and others don't have a Preferred RIC. This means that there is no primary key available for me to retrieve using the Search Lookup API (i.e. `/api/rdp/discovery/search/v1/lookup`). See example in screenshot.

Is there some sort of 'primary key' field in the Search API that I can use to retrieve bonds that is actually unique and always available for each bond?

Find more posts tagged with

python api

rdp-api

#product

Accepted answers

nick.zincone

Hi @tr105

Regarding the bug you are finding, if you haven't already, I would suggest you report it, via F1- Help & Support. In my example, I did not come across any bug. If there was a syntax error, I would have likely been able to manually correct it once I ingested the output within the rd.search() call.

To clearly understand which identifier should be used to reference each bond, I would suggest you bring in a content specialist to confirm the specifics of the data and which one is more suitable. I believe it is RIC based on my experience working with the data, but this site is only focused on the usage of APIs and less on the specifics of content. The Helpdesk (F1 within Workspace) should be your preferred channel.

Just to be clear on your workflow, you want to:

Create a search within GOVSRCH to generate output that looks accurate
Export a query so you can use the rd.search() API. This will generate the list of bonds.
Use an identifier (ISIN or RIC) to perform a 'lookup'.

What I was focused on with my answer was to create a more precise list of bonds as I believe the list you generated may include unwanted bonds - given they have empty ISIN or RIC values. There may be a legitimate reason why some of the fields are empty. In addition, the export query tool looks to be a one-time step - and you should be able to workaround the bug if it appears to be a syntax error. If not, please post the export here so I can try it out. I'm aware you are using 'lookup', but that is step 3 of your workflow, I was focused solely on generating an accurate list of bonds, i.e. step 2 above. I usually only use GOVSRCH as a good way to generate an initial result, then fine-tune using specific filter properties within the API. Once you do confirm with the helpdesk the specific identifier, then you can build your filter around that.

All comments

nick.zincone

Hi @tr105

If you want to programmatically access this data via our APIs, you will have to go through an iterative process to experiment with the results to determine a suitable primary key value to retrieve the list of data. Search is extremely rich in what it provides and will require some fine tuning in order to filter out and request for the data of interest.

Given that, I would start off the journey as you did - retrieve a basic set of results and build from there. Here is the steps I would likely take:

1. As you did, derive a basic query within GOV SRCH then export the query:

For example, the above export looks like this:

[
    {
        "View": "GovCorpInstruments",
        "Top": 10,
        "Filter": "((DbType eq 'GOVT' or DbType eq 'CORP' or DbType eq 'AGNC' or DbType eq 'OMUN' or DbType eq 'OTHR') and IsActive eq true and (DbTypeDescription xeq 'Corporate'))",
        "Select": "RIC,EJVAssetID,DTSubjectName,BusinessEntity,PI,SearchAllCategoryv3,SearchAllCategoryv2,SearchAllCategory,DBSTicker,CouponRate,MaturityDate,IssueDate,ISIN,RCSCurrencyLeaf,RCSCountryLeaf,DbTypeDescription,InstrumentTypeDescription,RCSCouponTypeGenealogy,FaceIssuedUSD,RCSBondGradeLeaf"
    }
]

2. Generate an API call within Python (CodeBook in my case) to use the above results:

import refinitiv.data as rd
from refinitiv.data.content import search

rd.open_session()

filter = "((DbType eq 'GOVT' or DbType eq 'CORP' or DbType eq " \
         "'AGNC' or DbType eq 'OMUN' or DbType eq 'OTHR') and " \
        "IsActive eq true and (DbTypeDescription xeq 'Corporate'))"

select = "RIC,EJVAssetID,DTSubjectName,BusinessEntity,PI,SearchAllCategoryv3," \
         "SearchAllCategoryv2,SearchAllCategory,DBSTicker,CouponRate,MaturityDate," \
         "IssueDate,ISIN,RCSCurrencyLeaf,RCSCountryLeaf,DbTypeDescription," \
         "InstrumentTypeDescription,RCSCouponTypeGenealogy,FaceIssuedUSD," \
         "RCSBondGradeLeaf,ParentIssuerName,AssetStatusDescription," \
         "IsPerpetualSecurity,IssuerOrgid,AssetStatus,RCSCountryGenealogy"

rd.discovery.search(
    view = search.Views.GOV_CORP_INSTRUMENTS,
    top = 10,
    filter = filter,
    select = select
)

At this stage, I now have the data within my coding environment and can begin analyzing. As you can see, I only pulled down the top 10 results (top 10). However, if I specify the maximum rows (10000), I get back 10000 so I know there is more. When I see these many results, this is more likely the case that I may need to further filter down the request. This is the experimentation that I was referring to. What you can do is look at the results within GOV SRCH and determine if there are bonds that should not be there, for example, do you want perpetual bonds, do you want bonds that have matured - this is the how rich the Search environment is.

What I've done is utilized a programmatic discovery tool that I built that helped me figure out what properties and what specific elements I may need to filter out. The tool can also help to identify what would be a primary key. For example, you may see missing ISIN or Preferred RICs - there may be a reason for this, such as a very specific bond that you may not be interested in. When it gets down to this level of granularity of data, you may need to reach out to the Refinitiv Helpdesk.

The discovery tool that helps me with Search can be better explained in this article: Build queries easily using Refinitiv's Search API with Python and Jupyter | Devportal

tr105

Dear @nick.zincone ,

Thank you for your reply! I am aware of how the Python libraries and Search API work, but thank you for your walkthrough nonetheless.

My problem is that the bug I referenced (a bug in GOVSRCH, not the API), prevents me from exporting the GOVSRCH query as JSON. Hence, I am trying to import the results of my GOVSRCH searches using the primary keys of the bonds through the Search Lookup API (i.e. `/api/rdp/discovery/search/v1/lookup`). Note that this API is distinct from the one you use in your example.

My question is: which bond field is the primary key of each bond? ISIN and RIC won't do, for the aforementioned reasons.

nick.zincone

Hi @tr105

Just to be clear on your workflow, you want to:

Create a search within GOVSRCH to generate output that looks accurate
Export a query so you can use the rd.search() API. This will generate the list of bonds.
Use an identifier (ISIN or RIC) to perform a 'lookup'.

EXPLORE OUR SITES