Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

Expand pandas dataframe and consolidate columns

Tags:

python

pandas

I have a dataframe that looks like this:

    desc    item    type1    date1    type2    date2    type3    date3
0   this    foo1      A        9/1      B        9/2      C        9/3
1   this    foo2      D        9/4      E        9/5      F        9/6

How do I get it to look like:

     desc    item    type    date
0    this    foo1      A      9/1
1    this    foo1      B      9/2
2    this    foo1      C      9/3
3    this    foo2      D      9/4
4    this    foo2      E      9/5
5    this    foo2      F      9/6

?

like image 882
user12405981 Avatar asked Sep 14 '20 02:09

user12405981


People also ask

How do you expand a Pandas DataFrame?

set_option() to expand the number of displayed columns in a DataFrame. Call pandas. set_option("display. max_columns", width) with width as an integer to set the max_columns displayed to the desired width .

How do I combine columns in a data frame?

By use + operator simply you can combine/merge two or multiple text/string columns in pandas DataFrame. Note that when you apply + operator on numeric columns it actually does addition instead of concatenation.

How do you concatenate two or more Pandas DataFrame?

Use pandas. concat() to concatenate/merge two or multiple pandas DataFrames across rows or columns. When you concat() two pandas DataFrames on rows, it creates a new Dataframe containing all rows of two DataFrames basically it does append one DataFrame with another.

What is pandas Dataframe expanding ()?

In this tutorial, we will learn the Python pandas DataFrame.expanding () method. This is one of the window methods of pandas and it provides expanding transformations. It returns a window sub-classed for the particular operation.

How to combine two text columns into one in a pandas Dataframe?

You can use the following syntax to combine two text columns into one in a pandas DataFrame: df ['new_column'] = df ['column1'] + df ['column2'] If one of the columns isn’t already a string, you can convert it using the astype (str) command: df ['new_column'] = df ['column1'].astype(str) + df ['column2'] And you can use the following syntax to ...

How do I import a pandas Dataframe in Python?

import pandas as pd #import x2 = x.days.apply (lambda x: pd.Series (x)).unstack () #make an unstackeded series, x2 x.drop ('days', axis = 1).join (pd.DataFrame (x2.reset_index (level=0, drop=True))) #drop the days column, join to the x2 series


2 Answers

Check with wide_to_long

out = pd.wide_to_long(df.reset_index(), ['type','date'], i ='index', j = 'drop').reset_index(drop=True)
out
Out[127]: 
  type date
0    A  9/1
1    B  9/2
2    C  9/3

For your updated question, the same concept still applies, you just do not need to reset the index, since item is unique:

pd.wide_to_long(df, stubnames=['type','date'], i='item',j='drop').droplevel(-1).reset_index()



    item  type  date
0   foo1    A   9/1
1   foo2    D   9/4
2   foo1    B   9/2
3   foo2    E   9/5
4   foo1    C   9/3
5   foo2    F   9/6
like image 52
BENY Avatar answered Oct 17 '22 16:10

BENY


You can also use .melt on two dataframes by passing a list to value_vars using list comprehension if a column contains type or date. Then, you can merge these two dataframes on the index:

df = pd.merge(df.melt(id_vars='item', value_vars=[col for col in df.columns if 'type' in col], value_name='type')[['item','type']],
              df.melt(id_vars='item', value_vars=[col for col in df.columns if 'date' in col], value_name='date')['date'],
              how='left', left_index=True, right_index=True).sort_values('type')
df
Out[1]: 
   item type date
0  foo1    A  9/1
2  foo1    B  9/2
4  foo1    C  9/3
1  foo2    D  9/4
3  foo2    E  9/5
5  foo2    F  9/6
like image 42
David Erickson Avatar answered Oct 17 '22 17:10

David Erickson