Flatten double nested JSON

Question

I am trying to flatten a JSON file that looks like this:

{
"teams": [
  {
    "teamname": "1",
    "members": [
      {
        "firstname": "John", 
        "lastname": "Doe",
        "orgname": "Anon",
        "phone": "916-555-1234",
        "mobile": "",
        "email": "[email protected]"
      },
      {
        "firstname": "Jane",
        "lastname": "Doe",
        "orgname": "Anon",
        "phone": "916-555-4321",
        "mobile": "916-555-7890",
        "email": "[email protected]"
      }
    ]
  },
  {
    "teamname": "2",
    "members": [
      {
        "firstname": "Mickey",
        "lastname": "Moose",
        "orgname": "Moosers",
        "phone": "916-555-0000",
        "mobile": "916-555-1111",
        "email": "[email protected]"
      },
      {
        "firstname": "Minny",
        "lastname": "Moose",
        "orgname": "Moosers",
        "phone": "916-555-2222",
        "mobile": "",
        "email": "[email protected]"
      }
    ]
  }       
]

}

I wish to export this to an excel table. My current code is this:

from pandas.io.json import json_normalize
import json
import pandas as pd

inputFile = 'E:\\teams.json'
outputFile = 'E:\\teams.xlsx'

f = open(inputFile)
data = json.load(f)
f.close()

df = pd.DataFrame(data)

result1 = json_normalize(data, 'teams' )
print result1

results in this output:

members                                              teamname
0  [{u'firstname': u'John', u'phone': u'916-555-...        1
1  [{u'firstname': u'Mickey', u'phone': u'916-555-...      2

There are 2 members's data nested within each row. I would like to have an output table that displays all 4 members' data plus their associated teamname.

Answer 1

Use pandas.io.json.json_normalize

json_normalize(data,record_path=['teams','members'],meta=[['teams','teamname']])

output:
         email                firstname lastname mobile      orgname    phone       teams.teamname
0   [email protected]       John    Doe                   Anon      916-555-1234    1
1   [email protected]       Jane    Doe     916-555-7890  Anon      916-555-4321    1
2   [email protected]   Mickey  Moose   916-555-1111  Moosers   916-555-0000    2
3   [email protected]    Minny   Moose                 Moosers   916-555-2222    2

Explanation

from pandas.io.json import json_normalize
import pandas as pd

I've only learned how to use the json_normalize function recently so my explanation might not be right.

Start with what I'm calling 'Layer 0'

json_normalize(data)

output:
     teams
0   [{'teamname': '1', 'members': [{'firstname': '...

There is 1 Column and 1 Row. Everything is inside the 'team' column.

Look into what I'm calling 'Layer 1' by using record_path=

json_normalize(data,record_path='teams')

output:
     members                                          teamname
0   [{'firstname': 'John', 'lastname': 'Doe', 'org...    1
1   [{'firstname': 'Mickey', 'lastname': 'Moose', ...    2

In Layer 1 we have have flattened 'teamname' but there is more inside 'members'.

Look into Layer 2 with record_path=. The notation is unintuitive at first. I now remember it by ['layer','deeperlayer'] where the result is layer.deeperlayer.

json_normalize(data,record_path=['teams','members'])

output:
           email              firstname lastname   mobile     orgname   phone
0   [email protected]      John        Doe                  Anon    916-555-1234
1   [email protected]       Jane        Doe   916-555-7890  Anon    916-555-4321
2   [email protected]   Mickey     Moose   916-555-1111 Moosers 916-555-0000
3   [email protected]    Minny       Moose               Moosers 916-555-2222

Excuse my output, I don't know how to make tables in a response.

Finally we add in Layer 1 columns using meta=

json_normalize(data,record_path=['teams','members'],meta=[['teams','teamname']])

output:
         email                firstname lastname mobile      orgname    phone       teams.teamname
0   [email protected]       John    Doe                   Anon      916-555-1234    1
1   [email protected]       Jane    Doe     916-555-7890  Anon      916-555-4321    1
2   [email protected]   Mickey  Moose   916-555-1111  Moosers   916-555-0000    2
3   [email protected]    Minny   Moose                 Moosers   916-555-2222    2

Notice how we needed a list of lists for meta=[[]] to reference Layer 1. If there was a column we want from Layer 0 and Layer 1 we could do this:

json_normalize(data,record_path=['layer1','layer2'],meta=['layer0',['layer0','layer1']])

The result of the json_normalize is a pandas dataframe.