How to save pandas DataFrame's rows as JSON strings?

Question

I have a pandas DataFrame df and I convert each row to JSON string as follows:

df = pd.DataFrame(np.random.randn(50, 4), columns=list('ABCD'))
df_as_json = df.to_json(orient='records')

Then I want to iterate over the JSON strings (rows) of df_as_json and make further processing as follows:

for json_document in df_as_json.split('\n'):
    jdict = json.loads(json_document)
    //...

The problem is that df_as_json.split('\n') does not really split df_as_json into separate JSON strings.

How can I do what I need?

Answer 1

To get each row of the dataframe as a dict, you can use pandas.DataFrame.to_dict():

Code:

df = pd.DataFrame(np.random.randn(10, 4), columns=list('ABCD'))
for jdict in df.to_dict(orient='records'):
    print(jdict)

Results:

{'A': -0.81155648424969018, 'B': 0.54051722275060621, 'C': 2.1858014972680886, 'D': -0.92089743800379931}
{'A': -0.051650790117511704, 'B': -0.79176498452586563, 'C': -0.9181773278020231, 'D': 1.1698955805545324}
{'A': -0.59790963665018559, 'B': -0.63673166723131003, 'C': 1.0493603533698836, 'D': 1.0027811601157812}
{'A': -0.20909149867564752, 'B': -1.8022674158328837, 'C': 1.0849019267782165, 'D': 1.2203116471260997}
{'A': 0.33798033123267207, 'B': 0.13927004774974402, 'C': 1.6671536830551967, 'D': 0.29193412587056755}
{'A': -0.079327003827824386, 'B': 0.58625181818942929, 'C': -0.42365912798153349, 'D': -0.69644626255641828}
{'A': 0.33849577559616656, 'B': -0.42955248285258169, 'C': 0.070860788937864225, 'D': 1.4971679265264808}
{'A': 1.3411846077264038, 'B': -0.20189961315847924, 'C': 1.6294881274421233, 'D': 1.1168181183218009}
{'A': 0.61028134135655399, 'B': 0.48445766812257018, 'C': -0.31117315672299928, 'D': -1.7986688463810827}
{'A': 0.9181074339928279, 'B': 0.84151139156427757, 'C': -1.111794854210024, 'D': -0.7131446510569609}

Answer 2

Starting from v0.19, you can use to_json with lines=True parameter to save your data as a JSON lines file.

df.to_json('file.json', orient='records', lines=True)

This eliminates the need for a loop to save each record, as a solution with to_dict would involve.

The first 5 lines of file.json look like this -

{"A":0.0162261253,"B":0.8770884013,"C":0.1577913843,"D":-0.3097990255}
{"A":-1.2870077735,"B":-0.1610902061,"C":-0.2426829569,"D":-0.3247587907}
{"A":-0.7743891125,"B":-0.9487264737,"C":1.6366125588,"D":0.2943377348}
{"A":1.5128287075,"B":-0.389437321,"C":0.4841038875,"D":0.5315466818}
{"A":-0.1455759399,"B":1.0205229385,"C":0.6776108196,"D":0.832060379}