python regex unicode - extracting data from an utf-8 file

Question

I am facing difficulties for extracting data from an UTF-8 file that contains chinese characters.

The file is actually the CEDICT (chinese-english dictionary) and looks like this :

賓 宾 [bin1] /visitor/guest/object (in grammar)/
賓主 宾主 [bin1 zhu3] /host and guest/
賓利 宾利 [Bin1 li4] /Bentley/
賓士 宾士 [Bin1 shi4] /Taiwan equivalent of 奔馳|奔驰[Ben1 chi2]/
賓夕法尼亞 宾夕法尼亚 [Bin1 xi1 fa3 ni2 ya4] /Pennsylvania/
賓夕法尼亞大學 宾夕法尼亚大学 [Bin1 xi1 fa3 ni2 ya4 Da4 xue2] /University of Pennsylvania/
賓夕法尼亞州 宾夕法尼亚州 [Bin1 xi1 fa3 ni2 ya4 zhou1] /Pennsylvania/

Until now, I manage to get the first two fields using split() but I can't find out how I should proceed to extract the two other fields (let's say for the second line "bin1 zhu3" and "host and guest". I have been trying to use regex but it doesn't work for a reason I ignore.

#!/bin/python
#coding=utf-8
import re

class REMatcher(object):
    def __init__(self, matchstring):
        self.matchstring = matchstring

    def match(self,regexp):
        self.rematch = re.match(regexp, self.matchstring)
        return bool(self.rematch)

    def group(self,i):
        return self.rematch.group(i)

def look(character):
    myFile = open("/home/quentin/cedict_ts.u8","r")
    for line in myFile:
        line = line.rstrip()
        elements = line.split(" ")
        try:
            if line != "" and elements[1] == character:
                myFile.close()
                return line
        except:
            myFile.close()
            break
    myFile.close()
    return "Aucun résultat :("


translation = look("賓主") # translation contains one line of the file
elements = translation.split()
traditionnal = elements[0]
simplified = elements[1]
print "Traditionnal:" + traditionnal
print "Simplified:" + simplified

m = REMatcher(translation)
tr = ""
if m.match(r"\[(\w+)\]"):
    tr = m.group(1) 
print "Pronouciation:" + tr

Any help appreciated.

Answer 1

This builds a dictionary to look up translations by either simplified or traditional characters and works in both Python 2.7 and 3.3:

# coding: utf8
import re
import codecs

# Process the whole file decoding from UTF-8 to Unicode
with codecs.open('cedict_ts.u8',encoding='utf8') as datafile:
    D = {}
    for line in datafile:
        # Skip comment lines
        if line.startswith('#'):
            continue
        trad,simp,pinyin,trans = re.match(r'(.*?) (.*?) \[(.*?)\] /(.*)/',line).groups()
        D[trad] = (simp,pinyin,trans)
        D[simp] = (trad,pinyin,trans)

Output (Python 3.3):

>>> D['马克']
('馬克', 'Ma3 ke4', 'Mark (name)')
>>> D['一路顺风']
('一路順風', 'yi1 lu4 shun4 feng1', 'to have a pleasant journey (idiom)')
>>> D['馬克']
('马克', 'Ma3 ke4', 'Mark (name)')

Output (Python 2.7, you have to print strings to see non-ASCII characters):

>>> D[u'马克']
(u'\u99ac\u514b', u'Ma3 ke4', u'Mark (name)')
>>> print D[u'马克'][0]
馬克

Answer 2

I would continue to use splits instead of regular expressions, with the maximum split number given. It depends on how consistent the format of the input file is.

elements = translation.split(' ',2)
traditionnal = elements[0]
simplified = elements[1]
rest = elements[2]
print "Traditionnal:" + traditionnal
print "Simplified:" + simplified
elems = rest.split(']')
tr = elems[0].strip('[')
print "Pronouciation:" + tr

Output:

Traditionnal:賓主
Simplified:宾主
Pronouciation:bin1 zhu3

EDIT: To split the last field into a list, split on the /:

translations = elems[1].strip().strip('/').split('/')
#strip the spaces, then the first and last slash, 
#then split on the slashes

Output (for the first line of input):

['visitor', 'guest', 'object (in grammar)']