question

Upvotes
Accepted
atul.arya avatar image
3 ●0 ●1 ●4

Refinitiv API errors

I tried running this code (attached below). The code is inconsistent in giving results. It worked in the very first attempt for 12 companies but then it throws an error when I ran it for more than 2 companies and sometimes it runs in 3rd/4th attempts. In this inconsistency problem, we can not just say that this code will run for x companies in y attempts (y<=5). Is this a server overload problem or inherent issues with this API?


Code

rd.open_session()

# Define a function to chunk a list into smaller lists of a specific chunk size

def chunk_list(first, chunk_size):

for i in range(0, len(first), chunk_size):

yield first[i:i + chunk_size]

# Set the chunk size

chunk_size = 1000

# Define a list of RICs

RIC_list = ['GE', 'ROK','TT', 'EMR', 'DHR'] #, 'SCHN.PA', 'ABBN.S', 'SIEGn.DE', 'DHR','HON.O','JCI'] #, 'ZOMT.NS', 'INAR.NS', 'TATE.NS', 'COMU.NS', 'CENA.NS',

fields1 = [

'TR.FundClassId','TR.FundPortfolioName','TR.FundParentNameHistory', 'TR.FundPortfolioName.investorid',

'TR.FundOwnershipTurnover(Sdate=2024-06-30)',

'TR.FundAdjShrsHeld(Scale=6, Sdate=2024-06-30)','TR.FundAdjShrsHeld(Scale=6, Sdate=2024-03-31)','TR.FundAdjShrsHeld(Scale=6, Sdate=2023-12-31)','TR.FundAdjShrsHeld(Scale=6, Sdate=2023-09-30)',

'TR.FundAdjShrsHeld(Scale=6, Sdate=2023-06-30)','TR.FundAdjShrsHeld(Scale=6, Sdate=2023-03-31)', 'TR.FundAdjShrsHeld(Scale=6, Sdate=2022-12-31)','TR.FundAdjShrsHeld(Scale=6, Sdate=2022-09-30)',

'TR.FundAdjShrsHeld(Scale=6, Sdate=2022-06-30)','TR.FundAdjShrsHeld(Scale=6, Sdate=2022-03-31)', 'TR.FundAdjShrsHeld(Scale=6, Sdate=2021-12-31)','TR.FundAdjShrsHeld(Scale=6, Sdate=2021-09-30)',

'TR.FundHoldingsDate(Sdate=2024-06-30)','TR.FundHoldingsDate(Sdate=2024-03-31)','TR.FundHoldingsDate(Sdate=2023-12-31)','TR.FundHoldingsDate(Sdate=2023-09-30)',

'TR.FundHoldingsDate(Sdate=2023-06-30)','TR.FundHoldingsDate(Sdate=2023-03-31)','TR.FundHoldingsDate(Sdate=2022-12-31)','TR.FundHoldingsDate(Sdate=2022-09-30)',

'TR.FundHoldingsDate(Sdate=2022-06-30)','TR.FundHoldingsDate(Sdate=2022-03-31)','TR.FundHoldingsDate(Sdate=2022-12-31)','TR.FundHoldingsDate(Sdate=2022-09-30)',

'TR.FundHoldingsDate(Sdate=2022-06-31)', 'TR.FundTotalEquityAssets','TR.FundInvtStyleCode','TR.FundInvestorType','TR.FundAddrCountry','TR.FundParentType','TR.FdShrhldrActivistType', 'TR.FundInvtOrientation','TR.FundEarliestHoldDate','TR.FundAggPE12MonthsFwd','TR.FundContactFirstName','TR.FundContactMiddleInit','TR.FundContactLastName','TR.FundestorContactTitle',

'TR.FdConPrimaryFunction','TR.FundContactEmailAddr','TR.FundContactTelCntry','TR.FundContactTelAreaCode','TR.FundContactTelExt','TR.FundContactTelNumber','TR.FundAggRevenue','TR.FundAggReturnOnEquity','TR.FundAggReturnOnAssets'

]

parameters = {}

df_combined = None

retry_max = 5

retry_count = 1

# Define a function to get data with retry logic

def get_data_with_retry(chunk, fields, parameters):

global retry_count

while retry_count <= retry_max:

try:

df = rd.get_data(chunk, fields=fields, parameters=parameters)

return df

except RDError as e:

print(f"RDError code: {e.code}")

print(f"RDError message: {e.message}")

print(f"Attempt {retry_count} failed. Retrying...")

time.sleep(2**retry_count) # Exponential backoff

retry_count += 1

raise Exception("Maximum retry attempts reached.")

# Try to retrieve data in chunks and handle errors

try:

for idx, chunk in enumerate(chunk_list(RIC_list, chunk_size)):

print('i:', idx , 'c:', chunk)

df = get_data_with_retry(chunk, fields1, parameters)

if df_combined is None:

df_combined = df

else:

df_combined = df_combined.append(df)

print(f'{len(chunk)} chunks completed')

retry_count = 1 # Reset retry count for the next chunk

except Exception as e:

print(f"An error occurred: {e}")


Error:

1720072012110.png


Any help to understand this error and make code consistently give results would be much appreciated.

#technologyrefinitiv-data-platform
1720072012110.png (139.9 KiB)
icon clock
10 |1500

Up to 2 attachments (including images) can be used with a maximum of 512.0 KiB each and 1.0 MiB total.

2 Answers

· Write an Answer
Upvote
Accepted
Jirapongse avatar image
84.6k ●287 ●53 ●77

@atul.arya

Thank you for reaching out to us.

There are two types of timed out.

1. The client timed out

The client timed out can be configured via the configuration file (refinitiv-data.config.json) or the code.

{ 
  "http": {
    "request-timeout": 60
  },
...
}

The code is:

config = rd.get_config()
config.set_param("http.request-timeout", 60)
rd.open_session()

2. The server timed out

The server timed out can't be modified. Therefore, the number of items, fields, or date range should be reduced.

According to the error, it should be the client timed out. Please try to increase the request-timeout value in the RD configurations.

icon clock
10 |1500

Up to 2 attachments (including images) can be used with a maximum of 512.0 KiB each and 1.0 MiB total.

Upvotes
atul.arya avatar image
3 ●0 ●1 ●4

Hi @Jirapongse

Thank you for your response.

I have tested the code with your above-mentioned suggestion. This helped resolve most of the errors we were facing earlier. However, we still encounter some errors infrequently. While the solution works well for smaller lists of companies, it does not scale to the level we were aiming for.

We appreciate your assistance.

icon clock
10 |1500

Up to 2 attachments (including images) can be used with a maximum of 512.0 KiB each and 1.0 MiB total.

Jirapongse avatar image Jirapongse
Yes, it wil help the client timed out. However, it will not solve the server timed out if the application is still requesting a lot of data. Please try to reduce the number of items, fields, or requested date range in each request.


As far as I know, the API is not designed for applications to request a lot of data in one request. You may contact your LSEG account team or sales directly to discuss alternative methods.

atul.arya avatar image atul.arya Jirapongse
Sure, Thank you for the clarification.

Write an Answer

Hint: Notify or tag a user in this post by typing @username.

Up to 2 attachments (including images) can be used with a maximum of 512.0 KiB each and 1.0 MiB total.