Python - List comprehension expression for assigning values from list to list of DataFrames

Question

I have a list of DataFrame (one per file I have read in) and a list of strings (the filenames of the files). I want to create a new column filename in each DataFrame and assign it the corresponding value in the file name list. The goal is to identify the source of the data once I have concatenated the DataFrame list.

df = pd.DataFrame({ 'A' : pd.Series(1,index=list(range(4)),dtype='float32'),
                     'B' : 'bar',
                     'C' : 'foo' })

list_df = [df,df,df]
file_names = ['source1','source2','source3']

I am trying to do something like so:

[x.loc[:,'filename']  = file for (x,file) in (list_df,file_names)]

This is obviously not working as iterating over a tuple is not allowed within a list comprehension expression(?). For loop would be an option (but sub-optimal). Is it possible to achieve this using list comprehension and is that the most efficient solution?

Answer 1

The correct method to do this would be a simple for loop using zip() function , Example -

for df_,file in zip(list_df,file_names):
    df_.loc[:,'filename'] = file

---

But if you really must use list comprehension, you cannot use assignment statements inside a list comprehension. Instead of that, you can try creating a function that does the assignment, and call that function using list comprehension.

You would also need to zip() the list of dataframes and file_names list together, to get the elements at their corresponding indexes together.

Example -

def func(df,file):
    df.loc[:,'filename'] = file

[func(df_,file) for df_,file in zip(list_df,file_names)]

Demo -

In [54]: df = pd.DataFrame({ 'A' : pd.Series(1,index=list(range(4)),dtype='float32'),
   ....:                      'B' : 'bar',
   ....:                      'C' : 'foo' })

In [55]: list_df = [df,df,df]

In [56]: file_names = ['source1','source2','source3']

In [57]: def func(df,file):
   ....:     df.loc[:,'filename'] = file
   ....:

In [58]: [func(df,file) for df,file in zip(list_df,file_names)]
Out[58]: [None, None, None]

In [59]: df
Out[59]:
   A    B    C filename
0  1  bar  foo  source3
1  1  bar  foo  source3
2  1  bar  foo  source3
3  1  bar  foo  source3