Enabling cookies with urllib

Question

I would like to parse a website with urllib python library. I wrote this:

import urllib as web
source_rep.urlopen(url_rep).read()
print source_rep

The website returns to me a message saying that I should enable cookie. How can I do that with python?

Answer 1

This answer is tested with Python 3.7. I would typically use a new opener for each random URL that I want cookies for.

from urllib.request import build_opener, HTTPCookieProcessor, Request
url = 'https://www.cell.com/cell-metabolism/fulltext/S1550-4131(18)30630-2'
opener = build_opener(HTTPCookieProcessor())

Without a Request object:

response = opener.open(url, timeout=30)
content = response.read()

With a Request object:

request = Request(url)
response = opener.open(request, timeout=30)
content = response.read()