How do I clean the dataset?

Question

question

allatuw

3 ●0 ●0 ●5

How do I clean the dataset?

Hi Dev Community,

Just a little background to my coding experience, I'm fairly green with regards to coding in Python. Picking up things and learning as I go along. I've only really got experience with VBA and am self taught. Apologies in advance if the solution is pretty obvious.

I'm trying to pull some historical FX data to determine the average daily % change between close dates.

df = ek.get_timeseries(rics, 
                        start_date='2021-03-01',  
                        end_date='2022-06-30',
                        fields='Open Close',
                        interval='daily

*** On a separate note, is there a limit to the amount of data I can pull on these queries?

I've printed out the dataframe to excel and have noticed that some currencies include prices on weekends. I've managed to handle these by converting the index to include the day name and then to exclude the weekends from the dataframe.

df.index = df.index.strftime('%a, %d-%b-%y')
df = df[df.index.str.contains('Sat', 'Sun') == False]

I've also noticed that some currencies have missing opening or closing prices. I'm struggling to figure out how to cycle through the dataframe (array?) to account for these missing prices.

If there is a missing OPEN price, I want to populate the field with the prior day's CLOSE price
If there is a missing CLOSE price, I want to populate the field the the next day's OPEN price.

Once this is all sorted I'll use the following calculate the daily % change and the average.

df_delta = df.pct_change()
df_mean = df_delta.mean()

Thanks!

eikon-data-api python time-series encoding

weekend.png (37.4 KiB)

Sep 14, 2022 at 10:57 AM

10 |1500

Attachments: Up to 2 attachments (including images) can be used with a maximum of 512.0 KiB each and 1.0 MiB total.

wasin.w Sep 23, 2022 at 04:36 AM

Hello @allatuw

Thank you for your participation in the forum. Is the reply below satisfactory in resolving your query?

If so please can you click the 'Accept' text next to the appropriate reply? This will guide all community members who have a similar question.

Thanks,

AHS

Answer 1 · 2022-09-15T07:10:25Z

raksina.samasiri

14.2k ●30 ●5 ●10

hi @allatuw ,

First, the limit can be checked at Eikon Data API Usage and Limits Guideline

Then about the missing prices, let's say the RICs list is the below

rics = ['AUD=','EUR=','GBP=','CNY=','CHF=','NZD=']
df = ek.get_timeseries(rics, 
                        start_date='2021-03-01',  
                        end_date='2022-06-30',
                        fields='Open Close',
                        interval='daily')
df.index = df.index.strftime('%a, %d-%b-%y')
df.loc[~df.index.str.contains('Sat', 'Sun')]

(I've adjusted the code to filter out Sat and Sun rows a bit)

To replace the missing prices, the code below can be used

for ric in rics:
    df[(ric, 'OPEN')].fillna(df[(ric, 'CLOSE')].shift(1), inplace=True)
    df[(ric, 'CLOSE')].fillna(df[(ric, 'OPEN')].shift(-1), inplace=True)

Here's an output example of the case when there is a missing OPEN price, I want to populate the field with the prior day's CLOSE price

Hope this helps and please let me know in case you have further questions

1663225522944.png (74.4 KiB)

Sep 15, 2022 at 07:10 AM

10 |1500

Attachments: Up to 2 attachments (including images) can be used with a maximum of 512.0 KiB each and 1.0 MiB total.

raksina.samasiri Sep 15, 2022 at 07:23 AM

Testing with the mock up data

1663226632363.png (62.1 KiB)

Answer 2 · 2022-09-15T23:08:06Z

allatuw

3 ●0 ●0 ●5

Brilliant, thanks for your help @raksina.samasiri !

Sep 15, 2022 at 11:08 PM

10 |1500

Attachments: Up to 2 attachments (including images) can be used with a maximum of 512.0 KiB each and 1.0 MiB total.

Q&A Forum

question

How do I clean the dataset?

2 Answers

Write an Answer

question

How do I clean the dataset?

2 Answers

Write an Answer

Related Questions