UnicodeEncodeError: 'latin-1' codec can't encode character '\u2013' in position 637: ordinal not in range(256)

Question

This is driving me crazy and i have tried different suggestions from the community but it doesnt seem to work. I tried even recreating the db using just utf-8 and still it gives me this error.

Basically i am using pymysql module and writing to the db.

openconnect = pymysql.connect(host='xxxx',port=3306,user='xxx',passwd='xxx',db='xxxx')
opencursor = openconnect.cursor()

one of my column is having the problem, basically i tried these options...

the columns that cause issue is subject and i tried the below

subject = (df.Subject[i])

subject.encode('latin-1', 'ignore')

and then try to write to the db it fails. if i try subject.encode('latin-1') also it fails.

I have two options, either fix the encoding or how i can set the coalition on pymysql to use utf-8 ? I verified the db, coalition on mysql is set to utf-8. Really appreciate your input on this..

still struggling with this. cheers Kabeer

Answer 1

I was able to resolve the issue by defining a charset in pymysql connect:

openconnect = pymysql.connect(
  host='xxxx',port=3306,user='xxx',passwd='xxx',db='xxxx',charset='utf8'
)

Please note it is utf8 and not utf-8.