Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

Using Python's Requests library to navigate webpages / Click buttons

I'm new to web programming, and have recently began looking into using Python to automate some manual processes. What I'm trying to do is log into a site, click some drop-down menus to select settings, and run a report.

I've found the acclaimed requests library: http://docs.python-requests.org/en/latest/user/advanced/#request-and-response-objects and have been trying to figure out how to use it.

I've successfully logged in using bpbp's answer on this page: How to use Python to login to a webpage and retrieve cookies for later usage?

My understanding of "clicking" a button is to write a post() command that mimics a click: Python - clicking a javascript button

My question (since I'm new to web programming and this library) is how I would go about pulling the data I need to figure out how I would construct these commands. I've been looking into [RequestObject].headers, .text, etc. Any examples would be great.

As always, thanks for your help!

EDIT::: To make this question more concrete, I'm having trouble interacting with different aspects of a web-page. The following image shows what I'm actually trying to do:

Example

I'm on a web-page that looks like this. There is a drop-down menu with click-able dates that can be changed. My goal is to automate changing the date to the most recent date, "click"'Save and Run', and download the report when it's finished running.

like image 612
so13eit Avatar asked Jul 31 '13 15:07

so13eit


People also ask

What does requests library do in Python?

The requests library is the de facto standard for making HTTP requests in Python. It abstracts the complexities of making requests behind a beautiful, simple API so that you can focus on interacting with services and consuming data in your application.

How do you scrape data using requests?

Using requests library, we can fetch the content from the URL given and beautiful soup library helps to parse it and fetch the details the way we want. You can use a beautiful soup library to fetch data using Html tag, class, id, css selector and many more ways.


1 Answers

The only solution to this I have found is Selenium. If it werent a javascript heavy website you could try mechanize but for this you need to render the javascript and then inject javascript...like Selenium does.

Upside: You can record actions in Firefox (using selenium) then export those actions to python. The downside is that this code has to open a browser window to run.

like image 158
frodopwns Avatar answered Oct 23 '22 12:10

frodopwns