How to post bulk entity search requests via API using Python

I am trying to go from ticker + exchange mic code to Refinitive RIC and permID. I am using the permID API entity search functionality. I can successfully use the requests.get method to obtain such information for one ticker but when I try to use requests.post to send bulk requests I receive a 401 message back. The piece of code that is succesfully getting back an individual entry info is the following:

import requests     
import json        
import pandas as pd
import pprint

# Your own access token
access_token = 'XXXX' # XXXX has been replaced with my private API token
q_1 = 'ticker:SAP AND mic:XETR'

# API endpoint
request_url = " https://api-eit.refinitiv.com/permid/search&quot;
headers = {
    'q': q_1,
    'access-token': access_token
}
response = requests.get(request_url, params=headers)
print(response)
r = response.json()

The piece of code which is howerver returning an authentication error is the following:

import requests
import json
import pandas as pd
import pprint

# Your own access token
access_token = 'XXXX' # XXXX has been replaced with my private API token
q_1 = 'ticker:SAP AND mic:XETR'
q_2 = 'ticker:ADS AND mic:XETR'

# API endpoint
request_url = " https://api-eit.refinitiv.com/permid/search&quot;
data = {
    'q': [q_1, q_2],
    'access-token': access_token
}
response_post = requests.post(request_url, data=data)
print(response_post)
r = response_post.json()

Can you please confirm whether:

a) it is possible to request.posts in bulk as I am attempting to do?

b) why I am getting this error message back?

c) how I can fix my code to obtain a positive output?

I have ±500 tickers to request programmatically and I would like to do that by splitting my HTTPS requests in bulks of 50 each if possible.

Thanks you.

Find more posts tagged with

permid-api

intelligent-tagging-api

open-permid-api

Accepted answers

zoya faberov

Hello @Linda,

I do not believe that entity search service spec supports array-based query as your example suggests, please see PermID API User Guide (Online version). The best you will be able to do with this endpoint, is loop the requests one by one.

I think Record Match service would be more suitable for your requirement. See Record Matching - Match Post Tutorial in Java for explanation, and included example of input file:

LocalID,RIC,Ticker,Name,Country,Street,City,PostalCode,State,Websi
1,AAPL.O,,Apple,US,"Apple Campus, 1 Infinite Loop",Cupertino,95014,Californi
2,AAPL.O,,Apple,,,,
3,,TEVA,Teva Pharmaceutical Industries Ltd,IL,,Petah Tikva,
4,TATA,,Tata Sky,IN,,,
5,IBM.N,,,,,,
6,,msft,,,,,
7,,GPRO,GoPro Inc,,,,,,

You may wish to Try It Out with included Swagger doc, by pasting your token, this input and selecting Execute.

It supports tickets, and this way you should be able to send all your records in a single request, formatted this way. The description is in PermID API User Guide (Online version).

If your required entity list grows to be very large at some point, in multiple thousands, you may consider an alternative solution, using the overall universe and parsing what you need. Seeding, by downloading bulk file from PermId org, and using PermId atom feed for updates, once you have the complete record, you would parse out the orgs that you require. See "Atom Feeds" section in PermID API User Guide (Online version).

All comments

zoya faberov

Hello @Linda,

I think Record Match service would be more suitable for your requirement. See Record Matching - Match Post Tutorial in Java for explanation, and included example of input file:

LocalID,RIC,Ticker,Name,Country,Street,City,PostalCode,State,Websi
1,AAPL.O,,Apple,US,"Apple Campus, 1 Infinite Loop",Cupertino,95014,Californi
2,AAPL.O,,Apple,,,,
3,,TEVA,Teva Pharmaceutical Industries Ltd,IL,,Petah Tikva,
4,TATA,,Tata Sky,IN,,,
5,IBM.N,,,,,,
6,,msft,,,,,
7,,GPRO,GoPro Inc,,,,,,

You may wish to Try It Out with included Swagger doc, by pasting your token, this input and selecting Execute.

It supports tickets, and this way you should be able to send all your records in a single request, formatted this way. The description is in PermID API User Guide (Online version).

Linda

Hello @zoya faberov this is very helpful thanks. I am still familiarising with the API and its capabilities. Would you be so kind to provide me with a python example on how I can make use of the API (whichever one of its functionalities) to send a request and get back a list of primaryRICs for the given companies. The problem I am facing is the following:

1. I have a list of tickers and relevant mic codes (i.e. ticker:SAP AND mic:XETR, ticker:AAPL AND mic:XNGS, ticker:FRA AND mic:XETR, ticker:ADS AND mic:XETR)

2. I would like to get back a list of RICs for those combinations (i.e. SAPG.DE, AAPL.OQ, FRAG.DE, ADSG.DE)

At the moment I am using the permID API sending individual requests.get but the overheads infra cost for my system is a bit too much given I have a few hundreds ticker-mic combinations to solve for. I would like to be able to send this request in bulks somehow if possible but my familiarity with the API to do so does not allow me to make any remarkable progress as of yet.

I would really appreciate some help on this task i possible.

zoya faberov

Hi @Linda,

Please refer to Introduction to the Open PermID Python library article. See Record Matching and Record Matching File sections for multiple instrument submission discussion. The article comes with downloadable accompanying Python code hosted on GitHub.

You have an option to create the input as described in the article, using the list of tickers (see PermID API User Guide for the complete list of the supported identifiers, mics are not included) and run the companion code, to submit, in the described format, in a single request, and the output should contain the RICs, you are looking to map, in a Dataframe.

Linda

Thank you @zoya faberov - I have tried the following:

from OpenPermID import OpenPermID
import pandas as pd
opid = OpenPermID()
opid.set_access_token('XXX')

organization = """
LocalID,Standard Identifier,Name,Country,Street,City,PostalCode,State,Website
1,Ticker:AAPL&&MIC:XNYS,,,,,,,
2,Ticker:AAPL&&MIC:XNGS,,,,,,,
3,Ticker:FB&&MIC:XNGS,,,,,,,
4,Ticker:SAP&&MIC:XETR,,,,,,,
3,Ticker:SAP&&MIC:XNGS,,,,,,,
"""
output, err = opid.match(organization, dataType='Quotation')

This is the same script suggested in the example on there tutorial, however the script return an error:

'Bad Request: {"errorCode":3,"errorCodeMessage":"Invalid Content - Required fields are missing. matchDatatype value: Quotation is not supported."}'

It seems that 'Quotation" is not supported although it is listed as a possible input under dataType. Am I not seeing anything obvious here?

zoya faberov

Hello @Linda,

The library can be used to match, consistently, to what you submit as input. For instance, if you submit organizations:

organization="""
LocalID,Standard Identifier,Name,Country,Street,City,PostalCode,State,Website
1,,Apple,US,"Apple Campus, 1 Infinite Loop",Cupertino,95014,California,
2,,Apple,,,,,,
3,,Teva Pharmaceutical Industries Ltd,IL,,Petah Tikva,,,
4,,Tata Sky,IN,,,,,
5,RIC:IBM.N|Ticker:IBM,,,,,,,
6,Ticker:MSFT,,,,,,,
7,LEI:INR2EJN1ERAN0W5ZP974,,,,,,,
8,Ticker:FB&&Exchange:NSM,,,,,,,
9,Ticker:AAPL&&MIC:XNGS,,,,,,,
"""
output,err = opid.match(organization)

You can match organizations:

output,err = opid.match(data,dataType='Organization',numberOfMatchesPerRecord=1,raw_output=False)

If you submit Quotations, you should be able to match quotations

quotation="""
LocalID,Standard Identifier
1,RIC:IBM.N|Ticker:IBM
2,Ticker:MSFT
3,RIC:IBM.N&&Ticker:IBM
"""
output,err = opid.match(quotation, dataType='Quotation',raw_output=True)

Hope this helps.

EXPLORE OUR SITES