Python: multidimensional pandas DataFrame

Question

This is my first question.

I have many sets of data. Each of them should be presented in a DataFrame. I have tried to implement this by having a DataFrame as an item of a multidimensional tuple, e.g.:

data[0][1].Glucose.val
data[0][1].Glucose.time

I have predefined the tuple like this:

data = tuple([data_type for _ in range(3)] for _ in range(8))

Addressing this works fine, but if I try to fill the df with new values, all elements in the tuple are overwritten:

for condition in range(8):
    for index in range(3):
        loop_it = condition + row_mult * index
        exp_setting = expIDs[loop_it]

        tempval = pd.read_csv(f"raw_data/{exp_setting}_Glucose.csv", delimiter="\t")
        rundata[condition][index].DOT.val = tempval.val.values
        rundata[condition][index].DOT.time = tempval.t

What the hell am I doing wrong?

THANKS

Answer 1

Tuples are immutable, so you can't replace individual items without overwriting the whole tuple. You could use lists of DataFrames instead.

If your DataFrames all have the same shape, and all the values are numerical, you could also use just one multi-dimensional NumPy array for all the data, e.g.:

import numpy as np

data = np.array([[[1, 2], [3, 4]], 
                 [[5, 6], [7, 8]]]) 

# replace the first item in the second row of the first frame with 9 
data[0, 1, 0] = 9  

print(data)

[[[1 2]
  [9 4]]

 [[5 6]
  [7 8]]]

By the way, pandas did have special data structures for 3- and 4-dimensional DataFrames in earlier versions, but I guess they were found unnecessary. Maybe you can stack the data into one DataFrame with two dimensions. For that, you may want to look into pandas' MultiIndex functionality.