Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

UnicodeEncodeError: 'ascii' codec can't encode character u'\u201c' in position 34: ordinal not in range(128)

I have been working on a program to retrieve questions from Stack Overflow. Till yesterday the program was working fine, but since today I'm getting the error

"Message    File Name   Line    Position    
Traceback               
<module>    C:\Users\DPT\Desktop\questions.py   13      
UnicodeEncodeError: 'ascii' codec can't encode character u'\u201c' in position 34: ordinal not in range(128)"

Currently, the questions are being displayed, but I seem to be unable to copy the output to a new text file.

import sys
sys.path.append('.')
import stackexchange
so = stackexchange.Site(stackexchange.StackOverflow)
term= raw_input("Enter the keyword for Stack Exchange")
print 'Searching for %s...' % term,
sys.stdout.flush()
qs = so.search(intitle=term)
print '\r--- questions with "%s" in title ---' % (term)
for q in qs:
  print '%8d %s' % (q.id, q.title)
  with open('E:\questi.txt', 'a+') as question:
     question.write(q.title)

 time.sleep(10)
 with open('E:\questi.txt') as intxt:
   data = intxt.read()

regular = re.findall('[aA-zZ]+', data)
print(regular)

tokens = set(regular)

with open('D:\Dictionary.txt', 'r') as keywords:
  keyset = set(keywords.read().split())


with open('D:\Questionmatches.txt', 'w') as matches:
  for word in keyset:
    if word in tokens:
        matches.write(word + '\n')
like image 992
Aaron Misquith Avatar asked Jun 17 '14 13:06

Aaron Misquith


People also ask

How do I fix UnicodeEncodeError in Python?

Only a limited number of Unicode characters are mapped to strings. Thus, any character that is not-represented / mapped will cause the encoding to fail and raise UnicodeEncodeError. To avoid this error use the encode( utf-8 ) and decode( utf-8 ) functions accordingly in your code.

What is UnicodeEncodeError?

The UnicodeEncodeError normally happens when encoding a unicode string into a certain coding. Since codings map only a limited number of unicode characters to str strings, a non-presented character will cause the coding-specific encode() to fail. Encoding from unicode to str. >>>


2 Answers

q.title is a Unicode string. When writing that to a file, you need to encode it first, preferably a fully Unicode-capable encoding such as UTF-8 (if you don't, Python will default to using the ASCII codec which doesn't support any character codepoint above 127).

question.write(q.title.encode("utf-8"))

should fix the problem.

By the way, the program tripped up on character (U+201C).

like image 121
Tim Pietzcker Avatar answered Oct 17 '22 11:10

Tim Pietzcker


I ran into this as well using Transifex API

response['source_string']

UnicodeEncodeError: 'ascii' codec can't encode character u'\u2019' in position 3: ordinal not in range(128)

Fixed with response['source_string'].encode("utf-8")

import requests

username = "api"
password = "PASSWORD"

AUTH = (username, password)

url = 'https://www.transifex.com/api/2/project/project-site/resource/name-of-resource/translation/en/strings/?details'

response = requests.get(url, auth=AUTH).json()

print response['key'], response['context']
print response['source_string'].encode("utf-8")
like image 26
Vinnie James Avatar answered Oct 17 '22 13:10

Vinnie James