Pandas dataframe having an additional "layer"

Question

Suppose you have the following dataframe:

import pandas as pd
import numpy as np

df = pd.DataFrame(np.nan,columns=['A','B','C'],index=[0,1,2])

Suppose I want an additional "layer" on top of this pandas dataframe, such that column A, row 0 would have its value, column B, row 0 would have a different value, column C row 0 would have something, column A row 1 and so on. So like a dataframe on top of this existing one.

Is it possible to add other layers? How does one access these layers? Is this efficient, i.e. should I just use a separate data frame all together? And one would save these multiple layers as a csv, by accessing the individual layers?, or is there a function that would break them down into different worksheets in the same workbook?

Answer 1

pandas.DataFrame cannot have 3 dimensions:

DataFrame is a 2-dimensional labeled data structure with columns of potentially different types.

However, there is a way to fake 3-dimensions with MultiIndex / Advanced Indexing:

Hierarchical indexing (MultiIndex)

Hierarchical / Multi-level indexing is very exciting as it opens the door to some quite sophisticated data analysis and manipulation, especially for working with higher dimensional data. In essence, it enables you to store and manipulate data with an arbitrary number of dimensions in lower dimensional data structures like Series (1d) and DataFrame (2d).

If you really need that extra dimension go with pandas.Panel:

Panel is a somewhat less-used, but still important container for 3-dimensional data.

but don't miss this important disclaimer from the docs:

Note: Unfortunately Panel, being less commonly used than Series and DataFrame, has been slightly neglected feature-wise. A number of methods and options available in DataFrame are not available in Panel.

There is also pandas.Panel4D (experimental) in the unlikely chance that you need it.