How to programmatically logon to a URL, keep the session, and browse around to different pages

Question

I am working on small Java project to programmatically connect to a website with username/password, after login, browse to different links on the site to download some data. First, I need to connect to the website with username/password, second, while I keep the session open, go to other links to download data.

How do I do this in Java?

Any help will be highly appreciated!

Answer 1

Check out the Apache HTTPClient, it can do all this for you.

Edit: Apache HTTPClient has authentication and cookie handling features included, which will save you a lot of work doing this yourself.

Answer 2

If you want to extract some data HtmlUnit can help you a lot it can manage the authentication and also help you with data extraction.