I am using python 3.5.2
and scrapy 1.1.1
.
There is an error when running the code below:
#-*- coding:utf-8-*-
import random
import base64
class ProxyMiddleware(object):
def process_request(self, request, spider):
proxy = random.choice(PROXIES)
if proxy['user_pass'] is not None:
request.meta['proxy'] = "http://%s" % proxy['ip_port']
encoded_user_pass = base64.encodebytes(proxy['user_pass'])
request.headers['Proxy-Authorization'] = 'Basic ' + encoded_user_pass
print("ok!" + proxy['ip_port'])
else:
print("fail!" + proxy['ip_port'])
request.meta['proxy'] = "http://%s" % proxy['ip_port']
error:
File "C:\Users\dell\AppData\Local\Programs\Python\Python35\lib\base64.py", line 518, in _input_type_check
m = memoryview(s)
TypeError: memoryview: a bytes-like object is required, not 'str'
I think the error is related to this sentence:
encoded_user_pass = base64.encodebytes(proxy['user_pass'])
But I don't know how to solve it.
Some help please,
thanks in advance!
Edit:
encoded_user_pass = base64.encodebytes(proxy['user_pass'])
was changed to
encoded_user_pass = base64.encodebytes(proxy['user_pass'].encode())
there is another error:
request.headers['Proxy-Authorization'] = 'Basic ' + encoded_user_pass
TypeError: Can't convert 'bytes' object to str implicitly
what should I do?
Function base64.encodebytes()
is expecting bytes value and it seems like you are providing it a string.
To fix that you can simply encode your string value (encode() function turns your string object into bytes object):
base64.encodebytes('foo'.encode())
or in your case:
encoded_user_pass = base64.encodebytes(proxy['user_pass'].encode())
As the name suggests, base64.encodebytes
encodes bytes, not strings. As outlined in the documentation, it requires a bytes-like object (just like the error message also tells you).
Your proxy['user_pass']
value is a string however. In order to convert that into bytes, call str.encode
:
encoded_user_pass = base64.encodebytes(proxy['user_pass'].encode())
If you love us? You can donate to us via Paypal or buy me a coffee so we can maintain and grow! Thank you!
Donate Us With