Simultaneously melt multiple columns in Python Pandas

Question

wondering if pd.melt supports melting multiple columns. I have the below examples trying to have the value_vars as list of lists but i am getting an error:

ValueError: Location based indexing can only have [labels (MUST BE IN THE INDEX), slices of labels (BOTH endpoints included! Can be slices of integers if the index is integers), listlike of labels, boolean] types

Using pandas 0.23.1.

df = pd.DataFrame({'City': ['Houston', 'Austin', 'Hoover'],
                   'State': ['Texas', 'Texas', 'Alabama'],
                   'Name':['Aria', 'Penelope', 'Niko'],
                   'Mango':[4, 10, 90],
                   'Orange': [10, 8, 14], 
                   'Watermelon':[40, 99, 43],
                   'Gin':[16, 200, 34],
                   'Vodka':[20, 33, 18]},
                 columns=['City', 'State', 'Name', 'Mango', 'Orange', 'Watermelon', 'Gin', 'Vodka'])

Desired output:

      City    State       Fruit  Pounds  Drink  Ounces
0  Houston    Texas       Mango       4    Gin    16.0
1   Austin    Texas       Mango      10    Gin   200.0
2   Hoover  Alabama       Mango      90    Gin    34.0
3  Houston    Texas      Orange      10  Vodka    20.0
4   Austin    Texas      Orange       8  Vodka    33.0
5   Hoover  Alabama      Orange      14  Vodka    18.0
6  Houston    Texas  Watermelon      40    nan     NaN
7   Austin    Texas  Watermelon      99    nan     NaN
8   Hoover  Alabama  Watermelon      43    nan     NaN

I tried that and i get the aforementioned error:

df.melt(id_vars=['City', 'State'], 
        value_vars=[['Mango', 'Orange', 'Watermelon'], ['Gin', 'Vodka']],var_name=['Fruit', 'Drink'], 
        value_name=['Pounds', 'Ounces'])

Answer 1

Use double melt for each catogories and then concat, but because duplicted values add cumcount for unique triples in MultiIndex:

df1 = df.melt(id_vars=['City', 'State'], 
              value_vars=['Mango', 'Orange', 'Watermelon'],
              var_name='Fruit', value_name='Pounds')
df2 = df.melt(id_vars=['City', 'State'], 
              value_vars=['Gin', 'Vodka'], 
              var_name='Drink', value_name='Ounces')

df1 = df1.set_index(['City', 'State', df1.groupby(['City', 'State']).cumcount()])
df2 = df2.set_index(['City', 'State', df2.groupby(['City', 'State']).cumcount()])


df3 = (pd.concat([df1, df2],axis=1)
         .sort_index(level=2)
         .reset_index(level=2, drop=True)
         .reset_index())
print (df3)
      City    State       Fruit  Pounds  Drink  Ounces
0   Austin    Texas       Mango      10    Gin   200.0
1   Hoover  Alabama       Mango      90    Gin    34.0
2  Houston    Texas       Mango       4    Gin    16.0
3   Austin    Texas      Orange       8  Vodka    33.0
4   Hoover  Alabama      Orange      14  Vodka    18.0
5  Houston    Texas      Orange      10  Vodka    20.0
6   Austin    Texas  Watermelon      99    NaN     NaN
7   Hoover  Alabama  Watermelon      43    NaN     NaN
8  Houston    Texas  Watermelon      40    NaN     NaN

Answer 2

I ran into the same problem again today, found this question, saw that I already upvoted it, and thought it could be nice to make it a thing since it is a recurring problem for me.

As a result, I wrote a multi_melt function that uses the method that jezarel proposed, but works on iterable inputs (the syntax Martin Petrov used). Note that this version "broadcasts" scalar inputs:

from itertools import cycle
import pandas as pd


def is_scalar(obj):
    if isinstance(obj, str):
        return True
    elif hasattr(obj, "__iter__"):
        return False
    else:
        return True


def multi_melt(
    df: pd.DataFrame,
    id_vars=None,
    value_vars=None,
    var_name=None,
    value_name="value",
    col_level=None,
    ignore_index=True,
) -> pd.DataFrame:

    # Note: we don't broadcast value_vars ... that would seem unintuitive
    value_vars = value_vars if not is_scalar(value_vars[0]) else [value_vars]
    var_name = var_name if not is_scalar(var_name) else cycle([var_name])
    value_name = value_name if not is_scalar(value_name) else cycle([value_name])

    melted_dfs = [
        (
            df.melt(
                id_vars,
                *melt_args,
                col_level,
                ignore_index,
            ).pipe(lambda df: df.set_index([*id_vars, df.groupby(id_vars).cumcount()]))
        )
        for melt_args in zip(value_vars, var_name, value_name)
    ]

    return (
        pd.concat(melted_dfs, axis=1)
        .sort_index(level=2)
        .reset_index(level=2, drop=True)
        .reset_index()
    )

Since its not part of the pandas API, you'll have to pipe it, but otherwise it should work like a normal melt that accepts iterables:

Example:

df = pd.DataFrame(
    {
        "City": ["Houston", "Austin", "Hoover"],
        "State": ["Texas", "Texas", "Alabama"],
        "Name": ["Aria", "Penelope", "Niko"],
        "Mango": [4, 10, 90],
        "Orange": [10, 8, 14],
        "Watermelon": [40, 99, 43],
        "Gin": [16, 200, 34],
        "Vodka": [20, 33, 18],
    },
    columns=["City", "State", "Name", "Mango", "Orange", "Watermelon", "Gin", "Vodka"],
)


df.pipe(
    multi_melt,
    id_vars=["City", "State"],
    value_vars=[["Mango", "Orange", "Watermelon"], ["Gin", "Vodka"]],
    var_name=["Fruit", "Drink"],
    value_name=["Pounds", "Ounces"],
)

Result:

      City    State       Fruit  Pounds  Drink  Ounces
0   Austin    Texas       Mango      10    Gin   200.0
1   Hoover  Alabama       Mango      90    Gin    34.0
2  Houston    Texas       Mango       4    Gin    16.0
3   Austin    Texas      Orange       8  Vodka    33.0
4   Hoover  Alabama      Orange      14  Vodka    18.0
5  Houston    Texas      Orange      10  Vodka    20.0
6   Austin    Texas  Watermelon      99    NaN     NaN
7   Hoover  Alabama  Watermelon      43    NaN     NaN
8  Houston    Texas  Watermelon      40    NaN     NaN

Single Melt:

df.pipe(
    multi_melt,
    id_vars=["City", "State"],
    value_vars=["Mango", "Orange", "Watermelon"],
    var_name="Fruit",
    value_name="Pounds",
)

      City    State       Fruit  Pounds
0   Austin    Texas       Mango      10
1   Hoover  Alabama       Mango      90
2  Houston    Texas       Mango       4
3   Austin    Texas      Orange       8
4   Hoover  Alabama      Orange      14
5  Houston    Texas      Orange      10
6   Austin    Texas  Watermelon      99
7   Hoover  Alabama  Watermelon      43
8  Houston    Texas  Watermelon      40