How to append daily data?

bill39

I have the previous 365 days of data stored in 'Daily API Download Master.csv'. I wish to daily append this file with new data. However no data is being appended with the below code. How might I get this to work?

#import packages

import eikon as ek # the Eikon Python wrapper package

import pandas as pd

import numpy as np

import datetime

from datetime import timedelta, date, datetime

from pandas.tseries.offsets import BDay

#connects to Bill's Eikon terminal

ek.set_app_key('x')

#retreive RICs

df_rics,e = ek.get_data("lists('Inv Trust List')","TR.RIC")

ric_list = df_rics['Instrument'].tolist()

#Slice, loop and concatenate the retrieval request

n = 50

df = pd.DataFrame()

for ric_chunk in [ric_list[i:i + n]

for i in range(0, len(ric_list), n)]:

tmp_df, e = ek.get_data(ric_chunk,

['TR.RNSFilerName',

'TR.RNSAnnouncedDate',

'TR.RNSTransactionType',

'TR.RNSARNumShrsTransacted',

'TR.RNSARPctOSTransacted',

'TR.RNSARTransactionPrice',

'TR.RNSARMktValTransaction',

'TR.RNSARTotShrsPostTrans',

'TR.RNSARPctOSPostTrans'])

df = tmp_df.append(df)

#set boundary dates - previous business day

end_date = date.today()

start_date = end_date - BDay(1)

#string format dates

end_date_str = datetime.strftime(end_date, "%Y-%m-%d")

start_date_str = datetime.strftime(start_date, "%Y-%m-%d")

#datetime formatting

df['RNS Announced Date'] = pd.to_datetime(df['RNS Announced Date'])

#set date constraints as object

mask = (df['RNS Announced Date'] > start_date_str) & (df['RNS Announced Date'] <= end_date_str)

#reset dataframe for boundary dates

df = df.loc[mask]

#rename '£' symbol

df.rename(columns={'RNS AR Price (at Transaction) - £': 'RNS AR Price (at Transaction) GBP',

'RNS AR Market Value of Transaction - £': 'RNS AR Market Value of Transaction - GBP'},

inplace=True)

#append new data to the the master csv

df.to_csv('Daily API Download Master.csv', mode='a', header=False)

Alex Putkov.1

@bill39

I think the problem is in your filtering logic by announcement date. The data you retrieve is not real-time. The most recent announcement date in the result set will be yesterday's date. Yet your logic filters out all records with announcement date less than today. I think you need to include the records with announcement date of yesterday, e.g. as follows

mask = ((df['RNS Announced Date'] >= start_date) & 
        (df['RNS Announced Date'] <= end_date))

Note that compared to your code I'm including records where the announcement date equals start_date.