Loop by chunks (RDP)

Find more posts tagged with

python

eikon

refinitiv-dataplatform-eikon

rdp-api

Accepted answers

I am not entirely sure what you are after - but from my understanding, I think something like the following might deliver what you are after?

import refinitiv.data as rd
from refinitiv.data.content import historical_pricing
from refinitiv.data.content.historical_pricing import Intervals
import pandas as pd
rd.open_session()
import csv
csvReader = csv.reader(open("Exp.csv"))
instruments=[]
for row in csvReader:
    instruments.append(row[0])
universe=instruments
content_df = []
chunklist=200
for i in range(0, len(instruments), chunklist):
    stocks = instruments[0:0+chunklist]
    response = historical_pricing.summaries.Definition(universe=stocks, 
                                                   interval = Intervals.MONTHLY, 
                                                   start="2016-01-01", end="2022-01-01"
                                                  ).get_data()
    df=response.data.df
    content_df.append(df.transpose())

final_df = pd.concat(content_df)
final_df.to_csv("new-rdp.csv")

NOTE: I was NOT able to test the above to conclusion - as I have been testing this and other scripts and I seem to have hit some Eikon Data limit - so the call fails with 'Too many requests, please try again later'

If the above does not deliver what you are after, I recommend you work through the tutorials etc and increase your Python knowledge - so that you are able to fulfil your requirement. I had never programmed Python until a few years ago - and once I spend some time working through some of the excellent Python tutorials online, I was quickly able to learn enough to create basic scripts such as those above.

All comments

Could you please help me out? @chavalit-jintamalit @jason.ramchandani01 @Alex Putkov.1

I am not an Eikon Content expert, so cannot help identify the correct Bond fields etc.

However, looking at your previous post that used the Eikon Data API, I have attached the equivalent python script that uses the RD Python Library. I have named the file George.txt - please rename it to George.py or copy and paste the code into your Jupyter Notebook etc.

George.txt

If the choice of fields etc is not correct, I recommend you raise a Content-Type ticket with the Eikon team at My.Refinitiv - who should be able to help you identify the correct Fields to use.

You will of course need to edit the path specified by RD_LIB_CONFIG_PATH to the location of your RD Python Library config file.

Note that I reduced the chunking size to 200 - as that worked for me, whereas 2000 did not.

You may be able to use a larger chunk size to improve the completion time of the script.

Hello @umer.nalla ,

First of all, thank you for your response.

Could you please tell me how can I edit the following two fields in order to run the code? I have zero background in coding so I apologise for asking (probably) very basic questions.

os.environ["RD_LIB_CONFIG_PATH"] = r"C:\OneDriveR\OneDrive - London Stock Exchange Group\Refinitiv\RD.Python\Example.DataLibrary.Python\Configuration"

# I tried putting the following above but it did not work (/Users/macintosh/opt/anaconda3/lib/python3.9/site-packages/python_configuration-0.8.2.dist-info)

session = rd.open_session('desktop.workspace') # config_name="../Configuration/refinitiv-data.custom.config.json")

#do I need to change 'desktop.workspace' above?

I get a lot of errors like

{"code":400,"message":"Application key is not valid
This session is not defined in the default configuration file

nick.zincone

There are a number of ways you can work with a configuration file. I would suggest you refer to the Session example which outlines the different ways you can open a session. Among the examples, it discusses the different ways you can work with a configuration file.

What I normally do, is copy the example configuration file into the same directory where your notebook(s) are running. When you do that, all you really need is the following line:

# Edit the refinitiv-data.config.json file (set the default)
rd.open_session()

You don't need to set the os.environ. It will find the file in your current directory. Secondly, find the session (platform or desktop), provide the required credentials then simply change the default line in the file to point to the session you want to open.

You will require some level of basic understanding of how the RD library is configured in order to use it and I recommend you work through the Quick Start and tutorials as a minimum.

However, for the particular issue, you are facing - the default behaviour of the RD Python library can come to your rescue.

If you don't specify a config file path - the RD Python Library will use a default in-memory configuration. And in that default configuration, it so happens that 'desktop.workspace' is the default session type.

So, for your scenario, you can comment out the following lines of code and add the extra line of code below

#os.environ["RD_LIB_CONFIG_PATH"] = r"C:\OneDriveR\OneDrive - London Stock Exchange Group\Refinitiv\RD.Python\Example.DataLibrary.Python\Configuration"

import refinitiv.data as rd

#session = rd.open_session('desktop.workspace')# config_name="../Configuration/refinitiv-data.custom.config.json")

rd.open_session()

i.e.

comment out or delete the os.environ line and the session= rd.open_session line
add the rd.open_session() line

Hello @umer.nalla , @nick.zincone

After considering your suggestions here is new error I receive. What is wrong with my code?

nick.zincone

To help this forum narrow down the issue, I would suggest you strip out all the extra code you've included and only focus on a small segment to demonstrate so we can attempt to pass on the code to the development team. In addition, you cut off the full error message so it's unclear what the error is.

For example, here is what I'm referring to. I took your code and stripped it down to attempt to replicate the issue.

Hello @nick.zincone, @umer.nalla

Please find below the error code I receive when I run the code for 10,000 RICs. What can be the root cause of this? Could you please run the code for the 10,000 RICs (I attached the file previously) and let me know how to proceed?

Thank you.ezyzip.zip

Did you try what @nick.zincone suggested - i.e. strip the code down to the basics? Do you still see an error with the stripped-down script with 10,000RIC? What is the error you see?

As mentioned yesterday, I was able to run the George.py script I previously attached with the DataNew.csv you supplied earlier (containing 10,000 RICs) - with the chunking set to 200.

Dear @umer.nalla and @nick.zincone, thank you for your help so far.

Please note that I have found out many RICs whose information is present in historical_pricing.summaries but not in rd.get_data. Therefore, I am struggling to create a code that performs identical operation with historical_pricing.summaries. Could you please provide a sample code for me?

Below is what I have for 1 single RIC, however, there are two problems with it:

output is not properly formatted (you may see that RICs are somewhere above - I want RICs to be next to the bond information). I am struggling to format it well and would appreciate your help.
I do not know how to integrate Loop so that it again opens the .csv file and returns output for all RICs, not just one...

response = historical_pricing.summaries.Definition(["00033AAA6="], interval = Intervals.MONTHLY, start="2016-01-01", end="2022-01-01", fields = ["BID", "ASK", "MID_PRICE", "MID_YLD_1"]).get_data()

df=response.data.df

I have read many online resources (https://github.com/Refinitiv-API-Samples/Example.DataLibrary.Python/blob/main/Examples/2-Content/2.01-HistoricalPricing/EX-2.01.01-HistoricalPricing.ipynb) but because I do not have any coding background I simply cannot complete this project. If you could help me out with this you will make my life a lot easier.

nick.zincone

If you would like to explore the time series API calls instead of the rd.get_data, we can focus on that instead.

For example, if you want the output formatted so the RIC is organized beside the columns of data, you can use the dataframe transpose method. For example:

df = rd.get_history(
    universe=["LSEG.L", "VOD.L"], 
    fields=["BID", "ASK", "MID_PRICE", "MID_YLD_1"], 
    interval='monthly', 
    start="2016-01-01", 
    end="2022-01-01"
)
df.transpose()

The output from the time series is organized differently, but you should still be able to process the data.

Regarding the looping, you can use a similar example above where the code included was bucketed.

My general advice if you are new to programming, is to start with the basics to get and process the data you need in the format you need - that is, with hardcoding only a few RICs. This will make life significantly easier than trying to go full out and trying to create algorithms to bucket and process 10,000 RICs. I know you need to eventually get there, but starting with basics will make it so much easier.

Hope this will get you started

Hello @nick.zincone, @umer.nalla

Please note that as I told you before I want to build a code with historical_pricing.summaries.Definition(. The code you supplied in the latest comment does not retrieve data for some of the RICs. How can I get started with this? It will probably take me months to build a code I need, while you might need only several minutes. Will you be so kind as to provide a sample code for me? What may I do in return for you? I simply cannot do this because I am a history major and you are probably the only people around the world who can help me out.

Thanks in advance.

Hello @nick.zincone @umer.nalla

New RDP.txt

This is what I have for now but it fails to work. Could you please optimise this code so it can run through thousands of RICsRICs.zip? I am also attaching a sample of non-duplicated RICs for you to test the code in the environment.