pandas read multi-index dataframe ( reverse to_string() )

Question

I have a text-file which looks like this:

test2.dat:

               col1      col2
idx1 idx2                    
a    0     0.256788  0.862771
     1     0.409944  0.785159
     2     0.822773  0.955309
b    0     0.159213  0.628662
     1     0.463844  0.667742
     2     0.292325  0.768051

Which was created by saving a multi-index pandas DataFrame via file.write(df.to_sring). Now, I want to reverse this operation. But when I try

pandas.read_csv(data, sep=r'\s+', index_col=[0, 1])

it throws an error ParserError: Error tokenizing data. C error: Expected 2 fields in line 3, saw 4

Here is a small MWE:

import pandas
import numpy as np
from itertools import product
df1 = pandas.DataFrame(product(['a', 'b'], range(3)), columns=['idx1', 'idx2'])
df2 = pandas.DataFrame(np.random.rand(6, 2), columns=['col1', 'col2'])
df  = pandas.concat([df1, df2], axis=1)
df.set_index(['idx1','idx2'], inplace=True)
df.to_csv('test.dat', sep=' ')
with open('test2.dat', 'w') as file:
    file.write(df.to_string())

Note that test.dat which was saved via pandas.to_csv() can barely be counted as 'human readable' compared to test2.dat

test.dat:

idx1 idx2 col1 col2
a 0 0.2567883353169065 0.862770538437793
a 1 0.40994403619942743 0.7851591115509821
a 2 0.8227727216889246 0.9553088749178045
b 0 0.1592133339255788 0.6286622783546136
b 1 0.4638439474864856 0.6677423709343185
b 2 0.2923252978245071 0.7680513714069206

Answer 1

Use read_fwf and set columns names by list comprehension:

df = pd.read_fwf('file.csv', header=[0,1])
df.columns = [y for x in df.columns for y in x if not 'Unnamed' in y]

#replace missing values by first column
df.iloc[:, 0] = df.iloc[:, 0].ffill().astype(int)
#set first 2 columns to MultiIndex
df = df.set_index(df.columns[:2].tolist())
print (df)
             col1    col2
idx1 idx2                
1    1     0.1234  0.2345
     2     0.4567  0.2345
     3     0.1244  0.5332
2    1     0.4213  0.5233
     2     0.5423  0.5423
     3     0.5235  0.6233