Export pandas dataframe to json and back to a dataframe with columns in the same order

Question

I wrote two small programs:

• The first one imports data from a .txt file into a pandas data frame, manipulates the data and exports the final data to a json file.
• The second code imports the data from that json file back into a data frame.

Unfortunately, the order of the columns changes when the data is imported from the json file. I have seen several examples online, where OrderedDict was used to create a fixed structure in a new table, but how can I apply OrderedDict to an existing table?

I have tried several versions including the ones below, but none of them worked:

df = OrderedDict(pd.DataFrame.from_dict(json_data, orient='columns'))

and

data = OrderedDict(pd.read_csv('wtx2015.txt', sep=",", header=None))

Code: .txt > pandas data frame > json

import pandas as pd
import json
from pandas import DataFrame
from collections import OrderedDict

pd.set_option("max_columns", 50)

"""Defining functions"""

data = pd.read_csv('wtx2015.txt', sep=",", header=None)
data.columns = ["category1", "category2", "category3", "category4"]

"""Manipulating data"""

print(data.head(n=3))

df = DataFrame(data, columns= ["category1", "category2", "category3", "category4", "category5"])

final = df.to_json(orient='records')
with open('pandas_test.json', 'w') as f_obj:
    f_obj.write(final)

Code: json > pandas data frame

import pandas as pd
import json

file = 'pandas_test.json'
with open(file) as f_obj:
    json_data = json.load(f_obj)

df = pd.DataFrame.from_dict(json_data, orient='columns')

print(df)

Answer 1

You can use parameter orient='split' in to_json/read_json which also save in json column names in list in original ordering:

df = pd.DataFrame({
        'C':list('abcdef'),
         'B':[4,5,4,5,5,4],
         'A':[7,8,9,4,2,3],
})

print (df.to_json(orient='split'))

{"columns":["C","B","A"],
 "index":[0,1,2,3,4,5],
 "data":[["a",4,7],["b",5,8],
         ["c",4,9],["d",5,4],["e",5,2],["f",4,3]]}

df.to_json('file.json', orient='split')
df = pd.read_json('file.json', orient='split')
print (df)
   C  B  A
0  a  4  7
1  b  5  8
2  c  4  9
3  d  5  4
4  e  5  2
5  f  4  3

Another alternative:

df.to_pickle('file')
df = pd.read_pickle('file')

And next alternative is add to json columns names in list:

import json

j = {'columns': df.columns.tolist(), 'data' : df.to_dict(orient='records')}
print (j)
{'columns': ['C', 'B', 'A'], 
 'data': [{'C': 'a', 'B': 4, 'A': 7}, 
          {'C': 'b', 'B': 5, 'A': 8}, 
          {'C': 'c', 'B': 4, 'A': 9}, 
          {'C': 'd', 'B': 5, 'A': 4}, 
          {'C': 'e', 'B': 5, 'A': 2}, 
          {'C': 'f', 'B': 4, 'A': 3}]}

file = 'file.json'
with open(file, 'w') as f_obj:
    json.dump(j, f_obj)

with open(file) as f_obj:
    json_data = json.load(f_obj)

df = pd.DataFrame(json_data['data'], columns=json_data['columns'])
print(df)
   C  B  A
0  a  4  7
1  b  5  8
2  c  4  9
3  d  5  4
4  e  5  2
5  f  4  3