Extracting dictionary values from a pandas dataframe

Question

I need to extra a features from a dataset I imported from a .json file.

This is what it looks like:

f1 = pd.read_json('https://raw.githubusercontent.com/ansymo/msr2013-bug_dataset/master/data/v02/eclipse/short_desc.json')

print(f1.head())


                                               short_desc
1       [{'when': 1002742486, 'what': 'Usability issue...
10      [{'when': 1002742495, 'what': 'API - VCM event...
100     [{'when': 1002742586, 'what': 'Would like a wa...
10000   [{'when': 1014113227, 'what': 'getter/setter c...
100001  [{'when': 1118743999, 'what': 'Create Help Ind...

In essence, I need to take 'short_desc' as the column name, and populate it with the string values directly below it: 'Usability issue...

So far, I've tried the following:

f1['desc'] = pd.DataFrame([x for x in f1['short_desc']])

Wrong number of items passed 19, placement implies 1

Is there an easy way to accomplish this without the use of loops? Could someone point this newbie in the right direction?

Answer 1

Don't initialise a dataframe and try to assign it to a column - columns are meant to be pd.Series.

You should just assign the list comprehension directly, like this:

f1['desc'] = [x[0]['what'] for x in f1['short_desc']]

---

As an alternative, I would propose a solution not involving any lambda functions, using operator and pd.Series.apply:

import operator

f1['desc'] = f1.short_desc.apply(operator.itemgetter(0))\
                             .apply(operator.itemgetter('what'))
print(f1.desc.head())

1           Usability issue with external editors (1GE6IRL)
10                   API - VCM event notification (1G8G6RR)
100       Would like a way to take a write lock on a tea...
10000     getter/setter code generation drops "F" in ".....
100001    Create Help Index Fails with seemingly incorre...
Name: desc, dtype: object