Python3 Error: TypeError: Can't convert 'bytes' object to str implicitly

Question

I am working on exercise 41 in learnpythonthehardway and keep getting the error:

  Traceback (most recent call last):   File ".\url.py", line 72, in <module>     question, answer = convert(snippet, phrase)   File ".\url.py", line 50, in convert     result = result.replace("###", word, 1) TypeError: Can't convert 'bytes' object to str implicitly 

I am using python3 while the books uses python2, so I have made some changes. Here is the script:

#!/usr/bin/python # Filename: urllib.py  import random from random import shuffle from urllib.request import urlopen import sys  WORD_URL = "http://learncodethehardway.org/words.txt" WORDS = []  PHRASES = {             "class ###(###):":                 "Make a class named ### that is-a ###.",             "class ###(object):\n\tdef __init__(self, ***)" :                 "class ### has-a __init__ that takes self and *** parameters.",             "class ###(object):\n\tdef ***(self, @@@)":                 "class ### has-a funciton named *** that takes self and @@@ parameters.",             "*** = ###()":                 "Set *** to an instance of class ###.",             "***.*** = '***'":                 "From *** get the *** attribute and set it to '***'." }  # do they want to drill phrases first PHRASE_FIRST = False if len(sys.argv) == 2 and sys.argv[1] == "english":     PHRASE_FIRST = True  # load up the words from the website for word in urlopen(WORD_URL).readlines():     WORDS.append(word.strip())  def convert(snippet, phrase):     class_names = [w.capitalize() for w in                     random.sample(WORDS, snippet.count("###"))]     other_names = random.sample(WORDS, snippet.count("***"))     results = []     param_names = []      for i in range(0, snippet.count("@@@")):         param_count = random.randint(1,3)         param_names.append(', '.join(random.sample(WORDS, param_count)))      for sentence in snippet, phrase:         result = sentence[:]          # fake class names         for word in class_names:             result = result.replace("###", word, 1)          # fake other names         for word in other_names:             result = result.replace("***", word, 1)          # fake parameter lists         for word in param_names:             result = result.replace("@@@", word, 1)          results.append(result)      return results  # keep going until they hit CTRL-D try:     while True:         snippets = list(PHRASES.keys())         random.shuffle(snippets)          for snippet in snippets:             phrase = PHRASES[snippet]             question, answer = convert(snippet, phrase)             if PHRASE_FIRST:                 question, answer = answer, question              print(question)              input("> ")             print("ANSWER: {}\n\n".format(answer)) except EOFError:     print("\nBye") 

What exactly am I doing wrong here? Thanks!

Answer 1

urlopen() returns a bytes object, to perform string operations over it you should convert it to str first.

for word in urlopen(WORD_URL).readlines():     WORDS.append(word.strip().decode('utf-8')) # utf-8 works in your case 

To get the correct charset : How to download any(!) webpage with correct charset in python?

Answer 2

In Python 3, the urlopen function returns an HTTPResponse object, which acts like a binary file. So, when you do this:

for word in urlopen(WORD_URL).readlines():     WORDS.append(word.strip()) 

… you end up with a bunch of bytes objects instead of str objects. So when you do this:

result = result.replace("###", word, 1) 

… you end up trying to replace the string "###" within the string result with a bytes object, instead of a str. Hence the error:

TypeError: Can't convert 'bytes' object to str implicitly 

The answer is to explicitly decode the words as soon as you get them. To do that, you have to figure out the right encoding from the HTTP headers. How do you do that?

In this case, I read the headers, I can tell that it's ASCII, and it's obviously a static page, so:

for word in urlopen(WORD_URL).readlines():     WORDS.append(word.strip().decode('ascii')) 

But in real life, you usually need to write code that reads the headers and dynamically figures it out. Or, better, install a higher-level library like requests, which does that for you automatically.