Scraping ERROR in Python: 'charmap' codec can't encode character / can't concat str to bytes

Question

I get above ERRORs when I try to scrape some text with Finish-Names from an 'url'. The solutions I tried and corresponding ERRORs, are commented below in the code. I neither know how to fix these, nor what the exact issue is. I'm a beginner in Python. Any help appreciated.

My Code:

from lxml import html
import requests

page = requests.get('url')

site = page.text  # ERROR -> 'charmap' codec can't encode character '\x84' in  
      #  position {x}: character maps to <undefined>
# site = site.encode('utf-8', errors='replace')  # ERROR -> can't concat str to bytes
# site = site.encode('ascii', errors='replace')  # ERROR -> can't concat str to bytes

with open('url.txt', 'a') as file:
    try:
        file.write(site + '\n')
    except Exception as err:
        file.write('an ERROR occured: ' + str(err) + '\n')

and the original Exception:

Traceback (most recent call last):
  File "...\parse.py", line 12, in <module> 
  file.write(site + '\n') File 
"...\python36\lib\encodings\cp1252.py", line 19, in encode return 
codecs.charmap_encode(input,self.errors,encoding_table)[0] 
UnicodeEncodeError: 'charmap' codec can't encode character '\x84' in position 
12591: character maps to <undefined>

regards, Dominik

Answer 1

Try this instead

with open('url.txt', 'a',encoding='utf-8') as file: