Using unicode (Hebrew characters) with regular expression

Question

I wrote script that find expressions in web page:

import sre, urllib2, sys, BaseHTTPServer
# -*- coding: utf-8 -*-    
address = sys.argv[1]
web_handle = urllib2.urlopen(address)
website_text = website_handle.read()    
matches = sre.findall(u"עברית", website_text)
for item in matches:
    print iten

This script works if I use a "regular" regular expression (without Hebrew characters) and doesn't match anything if I use them. What am I doing wrong?

edit example: url = https://en.wikipedia.org/wiki/Category:Countries

Answer 1

You need to ensure that the input string is also in UTF8 format.

Use unicode function with utf-8 as second argument:

website_text = unicode(website_text, "utf-8")

Everything should be in consistent encoding for unicode to work in Python 2.