Fill forms using selenium or requests

Question

I'm trying to enter this site to retrieve my bank account, first I tried with selenium, but only filled username (maybe because it has 2 forms):

from selenium import webdriver

driver = webdriver.Firefox()
driver.get("https://www.santandertotta.pt/pt_PT/Particulares.html")
user = driver.find_element_by_name("usr")
user.send_keys("user")
pas = driver.find_element_by_name("claveConsultiva")
pas.send_keys("password")
login = driver.find_element_by_id("login_button").click()

Then, I gone rambo mode :) trying figured out why I can't fill password space, and what are the hidden values of the form using requests, this is the code:

url = "https://www.particulares.santandertotta.pt/pagina/indice/0,,276_1_2,00.html"     
user_agent = {"user-agent" : "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/..."}
session = requests.session()
r = session.get(url)
soup = BeautifulSoup(r.text, "html.parser")    
data = {t['name']:t.get('value') for t in soup.find_all('input', attrs={'type': 'hidden'})}
print(data)    

But just received an empty dict. What is the best approach for enter a site with login and scrape?

Answer 1

You cannot get access to Password field because it's not present on main page. To handle Password field you have to click Login button to get to Login page. Also you need to switch to iframe which contains authentication form

from selenium.webdriver.support.ui import WebDriverWait as wait
from selenium.webdriver.support import expected_conditions as EC

driver.get("https://www.santandertotta.pt/pt_PT/Particulares.html")
driver.find_element_by_xpath("//input[@title='Login de Particulares']").click()
wait(driver, 10).until(EC.frame_to_be_available_and_switch_to_it("ws"))
user = driver.find_element_by_name("identificacionUsuario")
user.send_keys("user")
pas = driver.find_element_by_name("claveConsultiva")
pas.send_keys("password")
pas.submit()