How to have Java application interact with a website

Question

I have a program that takes data from an excel file and manipulates it for the user. But in order to get updates to the excel file they need to be downloaded from a website. I originally tried using the robot class to navigate to the website, login with username and password, then navigate to the correct section of the website and find the button that says "download excel spreadsheet" and click it. But i understand that is a horrible way of doing it and it doesn't always work. What is a better way i can do this so that my program can go to the website and navigate to the page i want and then download the data. I read about 'page scrapping' but i don't think that would allow me to do this. I really want to interact with the webpage not so much download the contents of it. Any help would be great. Thanks, Peter

Answer 1

If you actually need to interact with the website then selenium/webdriver is perfect for your needs:

http://code.google.com/p/selenium/wiki/GettingStarted

Sample Google search:

package org.openqa.selenium.example;

import org.openqa.selenium.By;
import org.openqa.selenium.WebDriver;
import org.openqa.selenium.WebElement;
import org.openqa.selenium.htmlunit.HtmlUnitDriver;

public class Example  {
    public static void main(String[] args) {
        // Create a new instance of the html unit driver
        // Notice that the remainder of the code relies on the interface, 
        // not the implementation.
        WebDriver driver = new HtmlUnitDriver();

        // And now use this to visit Google
        driver.get("http://www.google.com");

        // Find the text input element by its name
        WebElement element = driver.findElement(By.name("q"));

        // Enter something to search for
        element.sendKeys("Cheese!");

        // Now submit the form. WebDriver will find the form for us from the element
        element.submit();

        // Check the title of the page
        System.out.println("Page title is: " + driver.getTitle());
    }
}