Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

Record http form posts via a browser

I'm trying to automate the login to a website and submission of a form.

Is there a browser plugin (for firefox or Chrome) that allows you to record HTTP GET and POST requests in a form that allows them to be played back at a later point? I'm looking for something that will be possible to automate from a script e.g. via curl or wget.

I've tried using the Chrome developer tools to capture POST form data but I get errors when trying to replicate the request with wget which suggests I'm missing some cookies or other parameters. Ideally there would a nice automated way of doing this rather than doing lots of trial and error.

like image 444
Alex Spurling Avatar asked Feb 25 '11 16:02

Alex Spurling


People also ask

How do I track HTTP requests in my browser?

To view the request or response HTTP headers in Google Chrome, take the following steps : In Chrome, visit a URL, right click , select Inspect to open the developer tools. Select Network tab. Reload the page, select any HTTP request on the left panel, and the HTTP headers will be displayed on the right panel.


2 Answers

For a simple interaction, you don't really need a tool like Selenium that will record and playback requests.

You only need the tools you've already mentioned:

  1. Chrome already comes with the Developer Tools that you need: use the Network tab. No plugin to download. I don't know if Safari will work -- I don't see a "Network" tab in its Developer Tools.
  2. Both curl and wget support cookies and POST data, but I've only tried curl for automation.

There are several key steps that need to be done properly (this takes some experience):

  1. The sequence of pages that are requested needs to model real user interaction. This is important because you have no idea exactly how the backend handles forms or authentication. This is where the Network tab of Chrome's Developer Tools comes in. (Note that there is "record" button that will prevent the clearing of the log.) When you prepare to log a real user interaction for your analysis, don't forget to clear your cookies at the beginning of each session.
  2. You need to use all the proper options of curl and wget that will ensure that cookies and redirects are properly processed.
  3. All POST form fields will likely need to be sent (you'll often see fields with nonce values to prevent CSRF

Here's a sample of 3 curl calls that I wrote for an automation script that I wrote to download broadband usage from my ISP:

curl \     --silent \     --location \     --user-agent "$USER_AGENT" \     --cookie-jar "$COOKIES_PATH.txt" \     'https://idp.optusnet.com.au/idp/optus/Authn/Service?spEntityID=https%3A%2F%2Fwww.optuszoo.com.au%2Fshibboleth&j_principal_type=ISP' >$USAGE_PATH-1.html 2>&1 && sleep 3 &&  # --location because the previous request returns with a series of redirects "302 Moved Temporarily" or "302 Found" curl \     --silent \     --location \     --user-agent "$USER_AGENT" \     --cookie "$COOKIES_PATH.txt" \     --cookie-jar "$COOKIES_PATH.txt" \     --referer 'https://idp.optusnet.com.au/idp/optus/Authn/Service?spEntityID=https%3A%2F%2Fwww.optuszoo.com.au%2Fshibboleth&j_principal_type=ISP' \     --data "spEntityID=https://www.optuszoo.com.au/shibboleth&j_principal_type=ISP&j_username=$OPTUS_USERNAME&j_password=$OPTUS_PASSWORD&j_security_check=true" \     'https://idp.optusnet.com.au/idp/optus/Authn/Service' >$USAGE_PATH-2.html 2>&1 && sleep 1 &&  curl \     --silent \     --location \     --user-agent "$USER_AGENT" \     --cookie "$COOKIES_PATH.txt" \     --cookie-jar "$COOKIES_PATH.txt" \     --referer 'https://www.optuszoo.com.au/' \     'https://www.optuszoo.com.au//r/ffmu' >$USAGE_PATH-3.html 2>/dev/null 

Note the careful use of --cookie-jar, --cookie, and --location. The sleeps, --user-agent, and --referer may not be necessary (the backend may not check) but they're simple enough that I include them to minimize the chance of errors.

In this example, I was lucky that there were no dynamic POST fields, e.g. anti-CSRF nonce fields, that I would have had to extract and pass on to a subsequent request. That's because this automation is for authentication. For automating other types of web interactions, after the user's already logged in, you're likely to run into more of these dynamically-generated fields.

like image 95
huyz Avatar answered Oct 06 '22 05:10

huyz


Not exactly a browser plugin, but Fiddler can capture all the HTTP data passing back and forth; with FiddlerScript or FiddlerCore, it is then simple to export that into a text file - and pass that into cURL as request headers and request body.

like image 41
Piskvor left the building Avatar answered Oct 06 '22 04:10

Piskvor left the building