Command Line / Shell based rendering of URLs employing Javascript (Jive Server) for subsequent processing

Question

I have the following situation.

We are using a LaTeX-based documentation system which needs to obtain information that is posted to a Jive (Clearspace) webpage. This information is provided on a webpage in Javascript which cannot be properly rendered using conventional tools like cURL, wget, or lynx.

I am looking for some means to be able to do the following (bash/command line preferred)

1) Login to the Jive server with appropriate credentials (will probably need to save a cookie) 2) Access the specific page and dump the relevant information to a text file

Any ideas/suggestions? I typically do most of these automated procedures using bash due to the fact that it makes integration with LaTeX (compiled with command line) much easier. Yet, I am open to other routes with a bit of guidance.

Answer 1

It is possible to post and save cookies with wget (and presumably cURL). Here is an example from the wget man page:

 wget --save-cookies cookies.txt \
                        --post-data 'user=foo&password=bar' \
                        --keep-session-cookies
                        http://server.com/auth.php

You can them make the subsequent request to the page that displays the data with

 --load-cookie cookies.txt

As for figuring out what to put in the post-data option, I recommend installing a debugging proxy like fiddler2. This will make it easy to understand the entire structure of the http messages sent to the server and allow you to reverse engineer the forms.