How to convert unicode text to normal text

Question

I am learning Beautiful Soup in Python.

I am trying to parse a simple webpage with list of books.

E.g

<a href="https://www.nostarch.com/carhacking">The Car Hacker’s Handbook</a>

I use the below code.

import requests, bs4
res = requests.get('http://nostarch.com')
res.raise_for_status()
nSoup = bs4.BeautifulSoup(res.text,"html.parser")
elems = nSoup.select('.product-body a')

#elems[0] gives
<a href="https://www.nostarch.com/carhacking">The Car Hacker\u2019s Handbook</a>

And

#elems[0].getText() gives
u'The Car Hacker\u2019s Handbook'

But I want the proper text which is given by,

s = elems[0].getText()
print s
>>>The Car Hacker’s Handbook

How to modify my code in order to give "The Car Hacker’s Handbook" output instead of "u'The Car Hacker\u2019s Handbook'" ?

Kindly help.

Answer 1

Have you tried using the encode method?

elems[0].getText().encode('utf-8')

More info about unicode and python can be found in https://docs.python.org/2/howto/unicode.html

Moreover, to discover if your string is really utf-8 encoded you can use chardet and run the following command:

>>> import chardet
>>> chardet.detect(elems[0].getText()) 
{'confidence': 0.5, 'encoding': 'utf-8'}