Unicode input with raw_input() in python

Question

I´am just starting to learn python using LPTHW, and it´s really good. I´am just a couple of days in to my studies and come up to excercise 16 it looks like this:

-*- coding: utf-8 -*-

from sys import argv

script, filename = argv

print "We're going to erase %r." % filename
print "If you don't want that, hit CTRL-C (^C)."
print "If you do want that, hit RETURN."

raw_input("?")

print "Opening the file..."
target = open(filename, 'w')

print "Truncating the file.  Goodbye!"
target.truncate()

print "Now I'm going to ask you for three lines."

line1 = raw_input("line 1: ")
line2 = raw_input("line 2: ")
line3 = raw_input("line 3: ")

print "I'm going to write these to the file."

target.write("%r\n%r\n%r\n" % (line1, line2, line3))

print "And finally, we close it."
target.close()

The problem is that i'm from a country with the letters "Å", "Ä" and "Ö" in the alphabet, but when i am using these letters the output in the file (test.txt) looks something like this: u'hej' u'\xc5je' u'l\xe4get'

When i decode a string a can do something like this: "hallå".decode("utf-8")

And it will print just fine

But i also want the input from a user to be correct, even when using odd characters. I have tried different things that either does not work or gives me errors when running, like for example

line1 = raw_input("line 1: ").decode("utf-8")

I tried to google my problems but i did´t feel like the answers given was not very straight forward or written for much more experienced users.

If someone would take some time to explain the encoding/decoding of unicode characters in a beginner firendly way and give me an example of how i can get it to work i would really appriciate it

If it helps, iam on Windows 10, running python 2.7.10 and my system locale is set to swedish

Answer 1

Here's a way to decode stdin. It generally works from the Console but IDEs sometimes replace the stdin object and don't always support the encoding parameter. I also modernized the code a bit, using with and io.open to handle encodings. Note that the file will be written in UTF-8, so open it with Notepad to see it correctly. Using type <filename> from the console will try to display the file with the console's stdout encoding.

#!python2
import sys
import io

script, filename = sys.argv

print "We're going to erase %s." % filename
print "If you don't want that, hit CTRL-C (^C)."
print "If you do want that, hit RETURN."

raw_input("?")

print "Now I'm going to ask you for three lines."

line1 = raw_input("line 1: ").decode(sys.stdin.encoding)
line2 = raw_input("line 2: ").decode(sys.stdin.encoding)
line3 = raw_input("line 3: ").decode(sys.stdin.encoding)

print "I'm going to write these to the file."

with io.open(filename, 'wt', encoding='utf8') as target:
    target.write(u"%s\n%s\n%s\n" % (line1, line2, line3))

Answer 2

Your output indicates that raw_input() already accepts Å, ä just fine in your environment.

Either your code does not correspond to the output or your IDE is too helpful. raw_input() should return str type (bytes) but the output shows that you're saving text representations of unicode objects: u'hej' u'\xc5je' u'l\xe4get'.

The smallest code change that would produce your desirable result is using %s (save string as is) instead of %r (save its ascii printable representation as returned by repr() function) in the format string as suggested in @chepner's answer.

---

If someone would take some time to explain the encoding/decoding of unicode characters in a beginner firendly way and give me an example of how i can get it to work i would really appriciate it

Unicode handling on Python 2 requires understanding of what API returns text and what API returns binary data. Some API use a mixture such as ascii-based network protocols.

Python 2 allows str type to represent both human-readable text and binary data and it may create confusion. I recommend to start with Python 3 that is more strict for Unicode-related issues.

---

In general, while working with Unicode you should convert encoded text into Unicode on input as soon as possible (e.g., using .decode()) and convert Unicode text to bytes on output as late as possible. @Mark Tolonen's answer demonstrate this approach:

• it uses .decode(sys.stdin.encoding) to decode bytes returned from raw_input() into Unicode text. If raw_input() already returns Unicode in your environment (to check print type(raw_input('input something'))) then you could omit .decode() call
• io.open(..., encoding='utf-8').write(u'some text') convert Unicode text to bytes (encodes it using utf-8 encoding).

This general approach is known as Unicode sandwich.

.decode(sys.stdin.encoding) may fail. To support arbitrary Unicode input in Windows console, install win-unicode-console Python package.