Concatenate pandas DataFrames generated with a loop

Question

I am creating a new DataFrame named data_day, containing new features, for each day extrapolated from the day-timestamp of a previous DataFrame df.

My new dataframes data_day are 30 independent DataFrames that I need to concatenate/append at the end in a unic dataframe (final_data_day).

The for loop for each day is defined as follow:

num_days=len(list_day)

#list_day= random.sample(list_day,num_days_to_simulate)
data_frame = pd.DataFrame()

for i, day in enumerate(list_day):

    print('*** ',day,' ***')

    data_day=df[df.day==day]
    .....................
    final_data_day = pd.concat()

Hope I was clear. Mine is basically a problem of append/concatenation of data-frames generated in a non-trivial for loop

Answer 1

Pandas concat takes a list of dataframes. If you can generate a list of dataframes with your looping function, once you are finished you can concatenate the list together:

data_day_list = []
for i, day in enumerate(list_day):
    data_day = df[df.day==day]
    data_day_list.append(data_day)
final_data_day = pd.concat(data_day_list)

Answer 2

Exhausting a generator is more elegant (if not more efficient) than appending to a list. For example:

def yielder(df, list_day):
    for i, day in enumerate(list_day):
        yield df[df['day'] == day]

final_data_day = pd.concat(list(yielder(df, list_day))