python check if utf-8 string is uppercase

Question

I am having trouble with .isupper() when I have a utf-8 encoded string. I have a lot of text files I am converting to xml. While the text is very variable the format is static. words in all caps should be wrapped in <title> tags and everything else <p>. It is considerably more complex then this, but this should be sufficent for my question.

My problem is that this is an utf-8 file. This is a must, as there will be some many non-English characters in the final output. This may be time to provide a brief example:

inputText.txt

RÉSUMÉ

Bacon ipsum dolor sit amet strip steak t-bone chicken, irure ground round nostrud aute pancetta ham hock incididunt aliqua. Dolore short loin ex chicken, chuck drumstick ut hamburger ut andouille. In laborum eiusmod short loin, spare ribs enim ball tip sausage. Tenderloin ut consequat flank. Tempor officia sirloin duis. In pancetta do, ut dolore t-bone sint pork pariatur dolore chicken exercitation. Nostrud ribeye tail, ut ullamco venison mollit pork chop proident consectetur fugiat reprehenderit officia ut tri-tip.

DesiredOutput

    <title>RÉSUMÉ</title>
    <p>Bacon ipsum dolor sit amet strip steak t-bone chicken, irure ground round nostrud
       aute pancetta ham hock incididunt aliqua. Dolore short loin ex chicken, chuck drumstick
       ut hamburger ut andouille. In laborum eiusmod short loin, spare ribs enim ball tip sausage.
       Tenderloin ut consequat flank. Tempor officia sirloin duis. In pancetta do, ut dolore t-bone
       sint pork pariatur dolore chicken exercitation. Nostrud ribeye tail, ut ullamco venison
       mollit pork chop proident consectetur fugiat reprehenderit officia ut tri-tip.
   </p>

Sample Code

    #!/usr/local/bin/python2.7
    # yes this is an alt-install of python

    import codecs
    import sys
    import re
    from xml.dom.minidom import Document

    def main():
        fn = sys.argv[1]
        input = codecs.open(fn, 'r', 'utf-8')
        output = codecs.open('desiredOut.xml', 'w', 'utf-8')
        doc = Documents()
        doc = parseInput(input,doc)
        print>>output, doc.toprettyxml(indent='  ',encoding='UTF-8')

    def parseInput(input, doc):
        tokens = [re.split(r'\b', line.strip()) for line in input if line != '\n'] #remove blank lines

        for i in range(len(tokens)):
            # THIS IS MY PROBLEM. .isupper() is never true.
            if str(tokens[i]).isupper(): 
                 title = doc.createElement('title')
                 tText = str(tokens[i]).strip('[\']')
                 titleText = doc.createTextNode(tText.title())
                 doc.appendChild(title)
                 title.appendChild(titleText)
            else: 
                p = doc.createElement('p')
                pText = str(tokens[i]).strip('[\']')
                paraText = doc.createTextNode(pText)
                doc.appendChild(p)
                p.appenedChild(paraText)

       return doc

if __name__ == '__main__':
    main()

ultimately it is pretty straight forward, I would accept critiques or suggestions on my code. Who wouldn't? In particular I am unhappy with str(tokens[i]) perhaps there is a better way to loop through a list of strings?

But the purpose of this question is to figure out the most efficient way to check if an utf-8 string is capitalized. Perhaps I should look into crafting a regex for this.

Do note, I did not run this code and it may not run just right. I hand picked the parts from working code and may have mistyped something. Alert me and I will correct it. lastly, note I am not using lxml

Answer 1

The primary reason that your published code fails (even with only ascii characters!) is that re.split() will not split on a zero-width match. r'\b' matches zero characters:

>>> re.split(r'\b', 'foo-BAR_baz')
['foo-BAR_baz']
>>> re.split(r'\W+', 'foo-BAR_baz')
['foo', 'BAR_baz']
>>> re.split(r'[\W_]+', 'foo-BAR_baz')
['foo', 'BAR', 'baz']

Also, you need flags=re.UNICODE to ensure that Unicode definitions of \b and \W etc are used. And using str() where you did is at best unnecessary.

So it wasn't really a Unicode problem per se at all. However some answerers tried to address it as a Unicode problem, with varying degrees of success ... here's my take on the Unicode problem:

The general solution to this kind of problem is to follow the standard bog-simple advice that applies to all text problems: Decode your input from bytestrings to unicode strings as early as possible. Do all processing in unicode. Encode your output unicode into byte strings as late as possible.

So: byte_string.decode('utf8').isupper() is the way to go. Hacks like byte_string.decode('ascii', 'ignore').isupper() are to be avoided; they can be all of (complicated, unneeded, failure-prone) -- see below.

Some code:

# coding: ascii
import unicodedata

tests = (
    (u'\u041c\u041e\u0421\u041a\u0412\u0410', True), # capital of Russia, all uppercase
    (u'R\xc9SUM\xc9', True), # RESUME with accents
    (u'R\xe9sum\xe9', False), # Resume with accents
    (u'R\xe9SUM\xe9', False), # ReSUMe with accents
    )

for ucode, expected in tests:
    print
    print 'unicode', repr(ucode)
    for uc in ucode:
        print 'U+%04X %s' % (ord(uc), unicodedata.name(uc))
    u8 = ucode.encode('utf8')
    print 'utf8', repr(u8)
    actual1 = u8.decode('utf8').isupper() # the natural way of doing it
    actual2 = u8.decode('ascii', 'ignore').isupper() # @jathanism
    print expected, actual1, actual2

Output from Python 2.7.1:

unicode u'\u041c\u041e\u0421\u041a\u0412\u0410'
U+041C CYRILLIC CAPITAL LETTER EM
U+041E CYRILLIC CAPITAL LETTER O
U+0421 CYRILLIC CAPITAL LETTER ES
U+041A CYRILLIC CAPITAL LETTER KA
U+0412 CYRILLIC CAPITAL LETTER VE
U+0410 CYRILLIC CAPITAL LETTER A
utf8 '\xd0\x9c\xd0\x9e\xd0\xa1\xd0\x9a\xd0\x92\xd0\x90'
True True False

unicode u'R\xc9SUM\xc9'
U+0052 LATIN CAPITAL LETTER R
U+00C9 LATIN CAPITAL LETTER E WITH ACUTE
U+0053 LATIN CAPITAL LETTER S
U+0055 LATIN CAPITAL LETTER U
U+004D LATIN CAPITAL LETTER M
U+00C9 LATIN CAPITAL LETTER E WITH ACUTE
utf8 'R\xc3\x89SUM\xc3\x89'
True True True

unicode u'R\xe9sum\xe9'
U+0052 LATIN CAPITAL LETTER R
U+00E9 LATIN SMALL LETTER E WITH ACUTE
U+0073 LATIN SMALL LETTER S
U+0075 LATIN SMALL LETTER U
U+006D LATIN SMALL LETTER M
U+00E9 LATIN SMALL LETTER E WITH ACUTE
utf8 'R\xc3\xa9sum\xc3\xa9'
False False False

unicode u'R\xe9SUM\xe9'
U+0052 LATIN CAPITAL LETTER R
U+00E9 LATIN SMALL LETTER E WITH ACUTE
U+0053 LATIN CAPITAL LETTER S
U+0055 LATIN CAPITAL LETTER U
U+004D LATIN CAPITAL LETTER M
U+00E9 LATIN SMALL LETTER E WITH ACUTE
utf8 'R\xc3\xa9SUM\xc3\xa9'
False False True

The only differences with Python 3.x are syntactical -- the principle (do all processing in unicode) remains the same.