Reading CSV files in a loop using pandas, then concatenating them

Question

I have 10 csv files, named data_run1_all.csv, data_run2_all.csv, ..., data_run10_all.csv. CSV files have same columns, but different rows.

Now I am importing them one by one to df_run1, df_run2, ..., df_run10.

Can I use a loop to import them? Something like: i=1 to 10, df_runi=pandas.read_csv('data_runi_all.csv').

I am asking because the data analysis, plotting, etc. for each data frame are same, too. All the code for each data frame is repeated 10 times. If I can use a loop to do 10 times, the code will be much shorter and readable.

Answer 1

Read your CSVs in a loop and call pd.concat:

file_name = 'data_run{}_all.csv'
df_list = []
for i in range(1, 11):
    df_list.append(pd.read_csv(file_name.format(i))

df = pd.concat(df_list)

---

Alternatively, you could build the list inside a comprehension:

file_name = 'data_run{}_all.csv'
df = pd.concat([pd.read_csv(file_name.format(i)) for i in range(1, 11)])

Answer 2

You need to make df_run a list. You could do something like this:

df_run = []
for i in range(1,10):
  df_run.append(pandas.read_csv('data_run{0}_all.csv'.format(i))
for df in df_run:
  // Do your processing

Or do everything in a single loop, and avoid having the list.