Login to a website via Python - how to deal with CSRF?

Question

I'm using Python 3 for a script that will monitor updates in a user's profile on a webpage. The login to this site is protected by CSRF countermeasures, which is a good thing. However, I can't get my script to login to this site.

• My approach using mechanicalsoup:

  import mechanicalsoup
  
  browser = mechanicalsoup.Browser()
  login_page = browser.get(base_url)
  login_form = login_page.soup.select(".form-signin")[0]
  
  login_form.find(attrs={"name": "username"})['value'] = 'username'
  login_form.find(attrs={"name": "password"})['value'] = 'password'
  
  page2 = browser.submit(login_form, login_url)
  print(str(page2.text))
  

• My approach using robobrowser:

  import re
  from robobrowser import RoboBrowser
  
  browser = RoboBrowser(history=True)
  browser.open(base_url)
  form = browser.get_form(action='/login/')
  
  form["username"] = 'username'
  form["password"] = 'password'
  
  browser.submit_form(form)
  print(str(browser.select))
  

In both cases I end up with a HTTP status of 403 and a message saying CSRF verification failed. Request aborted.

• Any ideas how to fix this?
• The form in question has a hidden input containing a CSRF token. I guess mechanicalsoup and robobrowser will submit this input as well. Am I right? Or do I have to treat it specially?
• I thought the session used by this two packages would handle everything like cookies and so on. Is there something I've missed?

Answer 1

I got the robobrowser variant to work by setting the Referer header.

browser.session.headers['Referer'] = base_url

So the complete code that worked for me is the following:

import re
from robobrowser import RoboBrowser

browser = RoboBrowser(history=True)
browser.open(base_url)
form = browser.get_form(action='/login/')

form["username"] = 'username'
form["password"] = 'password'
browser.session.headers['Referer'] = base_url

browser.submit_form(form)
print(str(browser.select))

Answer 2

You are only adding username and password to the form you are submitting, you need to add the csrf token field as well. See below, I'm assuming you can figure out the field name and token value.

form["username"] = 'username'
form["password"] = 'password'
form["csrffieldname"] = 'csrfvalue' # This is what you are missing

The token value will be different for each form submission, so you'll have to get the form and parse out the csrf token value and submit it before the timeout expires for the token.