Pandas df groupby and apply

Question

Suppose the following df:

df = pd.DataFrame(
    {
    'col 1': ['A', 'A', 'B', 'B', 'C'],
    'col 2': ['c2', 'd2', 'e2', 'f2', 'g2'],
    'col 3': ['c3', 'd3', 'e3', 'f3', 'g3'],
    'col 4': ['c4', 'd4', 'e4', 'f4', 'g4'],
    }
    )

Desired output is:

col 1  col 2  col 3
A      c2;d2  c3;d3
B      e2;f2  e3;f3
C      g2     g3

I've succeeded in doing this in the following way:

df = df.groupby('col 1').transform( lambda x: ';'.join(x)).drop_duplicates()

The problem is that this method does not keep column 1 which i really need. and i couldn't get apply working. I've tried like this but it doesn't seem to work after .groupby.

apply(lambda x: '*'.join(x.dropna().values.tolist()), axis=1)

Answer 1

I think you can specify columns in list after groupby with function DataFrameGroupBy.agg, lambda function is not necessary:

df1 = df.groupby('col 1')['col 2','col 3'].agg(';'.join).reset_index()
#alternative
#df1 = df.groupby('col 1', as_index=False)['col 2','col 3'].agg(';'.join)
print (df1)
  col 1  col 2  col 3
0     A  c2;d2  c3;d3
1     B  e2;f2  e3;f3
2     C     g2     g3

If want also remove missing values:

df = pd.DataFrame(
    {
    'col 1': ['A', 'A', 'B', 'B', 'C'],
    'col 2': [np.nan, 'd2', 'e2', 'f2', 'g2'],
    'col 3': ['c3', 'd3', 'e3', 'f3', 'g3'],
    'col 4': ['c4', 'd4', 'e4', 'f4', 'g4'],
    }
    )
print (df)
  col 1 col 2 col 3 col 4
0     A   NaN    c3    c4
1     A    d2    d3    d4
2     B    e2    e3    e4
3     B    f2    f3    f4
4     C    g2    g3    g4

df1 = (df.groupby('col 1', as_index=False)['col 2','col 3']
         .agg(lambda x: ';'.join(x.dropna())))
print (df1)
  col 1  col 2  col 3
0     A     d2  c3;d3
1     B  e2;f2  e3;f3
2     C     g2     g3

Answer 2

Group on col 1 (specifying index as false so that it remains a column). Apply a lambda to each group where you join the values of each group with a semi-colon. Then sort the result in the desired column order (e.g. col 1-3).

df = pd.DataFrame(
    {'col 1': ['A', 'A', 'B', 'B', 'C'],
     'col 2': ['c2', 'd2', 'e2', 'f2', 'g2'],
     'col 3': ['c3', 'd3', 'e3', 'f3', 'g3'],
     'col 4': ['c4', 'd4', 'e4', 'f4', 'g4']})

>>> df.groupby('col 1', as_index=False).agg(
        {'col 2': lambda x: ';'.join(x),
         'col 3': lambda x: ';'.join(x)})[['col 1', 'col 2', 'col 3']]
  col 1  col 2  col 3
0     A  c2;d2  c3;d3
1     B  e2;f2  e3;f3
2     C     g2     g3