Convert file to dictionary to count occurrences

Question

I have a file where the content looks as follows:

eng word1
eng word2
eng word3
ita word1
ita word2
fra word1
...

I want to count the number of occurrences of each word in every language. For this purpose i want to read the file in a dict. This is my attempt:

data = open('file', 'r', encoding='utf8')
for line in data:
    lang = line[:3]
    ipa_string = line[3:]
    lang_and_string_dict[lang] = []
    lang_and_string_dict[lang].append(ipa_string)
print(lang_and_string_dict)

This gives me a dict with the right keys but only the last of the words for example for english:

{'eng':[word1]}

Answer 1

Well each time you assign an empty list as value:

data = open('file', 'r', encoding='utf8')
for line in data:
    lang = line[:3]
    ipa_string = line[3:]
    lang_and_string_dict[lang] = []
    lang_and_string_dict[lang].append(ipa_string)
print(lang_and_string_dict)

As a result, the old list containing the previous occurrence is lost. You should only create a list if no such element exists already, like:

data = open('file', 'r', encoding='utf8')
for line in data:
    lang = line[:3]
    ipa_string = line[3:]
    if lang not in lang_and_string_dict:
        lang_and_string_dict[lang] = []
    lang_and_string_dict[lang].append(ipa_string)
print(lang_and_string_dict)

Since this pattern is rather common, you can use a defaultdict as well:

from collection import defaultdict

lang_and_string_dict = defaultdict(list)
with open('file', 'r', encoding='utf8') as data:
    for line in data:
        lang = line[:3]
        ipa_string = line[3:]
    lang_and_string_dict[lang].append(ipa_string)
print(lang_and_string_dict)

A defaultdict is a subclass of dict that uses a factory (here list) in case a key is missing. So each time a key is queried that is not in the dictionary, we construct a list.

You can later convert such defaultdict to a dict with dict(lang_and_string_dict).

Furthermore if you open(..) files, you better do this with a with block. Since if an exception for example arises, then the file is still properly closed.

Answer 2

a simple approach using dict where keys are lang and values are counters of word occurrences

from collections import Counter, defaultdict

lang_and_string_dict = defaultdict(Counter)
with open('file', 'r', encoding='utf8') as f:
    for line in f:
        lang, word = line.split()
        lang_and_string_dict[lang].update([word])


print(lang_and_string_dict)

output

defaultdict(<class 'collections.Counter'>, {'eng': Counter({'word1': 1, 'word2': 1, 'word3': 1}), 'ita': Counter({'word1': 1, 'word2': 1}), 'fra': Counter({'word1': 1})})

Keep in mind the line lang, word = line.split() can cause an error or unexpected behaviour if the lines in the file aren't in exact lang word format, a exception and check is suggested