HttpPost works in Java project, but not on Android

Question

I've written some code for my Android device to login to a web site over HTTPS and parse some data out of the resulting pages. An HttpGet happens first to get some info needed for login, then an HttpPost to do the actual login process.

The code below works great in a Java project within Eclipse which has the following JAR files on the build path: httpcore-4.1-beta2.jar, httpclient-4.1-alpha2.jar, httpmime-4.1-alpha2.jar, and commons-logging-1.1.1.jar.

public static MyBean gatherData(String username, String password) {
    MyBean myBean = new MyBean();
    try {
        HttpResponse response = doHttpGet(URL_PAGE_LOGIN, null, null);
        System.out.println("Got login page");
        String content = EntityUtils.toString(response.getEntity());
        String token = ContentParser.getToken(content);
        String cookie = getCookie(response);
        System.out.println("Performing login");
        System.out.println("token = "+token +" || cookie = "+cookie);
        response = doLoginPost(username,password,cookie, token);
        int respCode = response.getStatusLine().getStatusCode();
        if (respCode != 302) {
            System.out.println("ERROR: not a 302 redirect!: code is \""+ respCode+"\"");
            if (respCode == 200) {
                System.out.println(getHeaders(response));
                System.out.println(EntityUtils.toString(response.getEntity()).substring(0, 500));
            }
        } else {
            System.out.println("Logged in OK, loading account home");
            // redirect handler and rest of parse removed
        }
    }catch (Exception e) {
        System.out.println("ERROR in gatherdata: "+e.toString());
        e.printStackTrace();
    }
    return myBean;
}
private static HttpResponse doHttpGet(String url, String cookie, String referrer) {
    try {
        HttpClient client = new DefaultHttpClient();
        client.getParams().setParameter(CoreProtocolPNames.PROTOCOL_VERSION, HttpVersion.HTTP_1_1);
        client.getParams().setParameter(CoreProtocolPNames.HTTP_CONTENT_CHARSET, "UTF-8");
        HttpGet httpGet = new HttpGet(url);
        httpGet.getParams().setParameter(CoreProtocolPNames.PROTOCOL_VERSION, HttpVersion.HTTP_1_1);
        httpGet.setHeader(HEADER_USER_AGENT,HEADER_USER_AGENT_VALUE);
        if (referrer != null && !referrer.equals("")) httpGet.setHeader(HEADER_REFERER,referrer);
        if (cookie != null && !cookie.equals("")) httpGet.setHeader(HEADER_COOKIE,cookie);
        return client.execute(httpGet);
    } catch (Exception e) {
        e.printStackTrace();
        throw new ConnectException("Failed to read content from response");
    }
}
private static HttpResponse doLoginPost(String username, String password, String cookie, String token) throws ClientProtocolException, IOException {
    try {
        HttpClient client = new DefaultHttpClient();
        client.getParams().setParameter(CoreProtocolPNames.PROTOCOL_VERSION, HttpVersion.HTTP_1_1);
        client.getParams().setParameter(CoreProtocolPNames.HTTP_CONTENT_CHARSET, "UTF-8");
        HttpPost post = new HttpPost(URL_LOGIN_SUBMIT);
        post.getParams().setParameter(CoreProtocolPNames.PROTOCOL_VERSION, HttpVersion.HTTP_1_1);
        post.setHeader(HEADER_USER_AGENT,HEADER_USER_AGENT_VALUE);
        post.setHeader(HEADER_REFERER, URL_PAGE_LOGIN);
        post.setHeader(HEADER_COOKIE, cookie);
        post.setHeader("Content-Type","application/x-www-form-urlencoded");
        List<NameValuePair> formParams = new ArrayList<NameValuePair>();
        formParams.add(new BasicNameValuePair("org.apache.struts.taglib.html.TOKEN", token));
        formParams.add(new BasicNameValuePair("showLogin", "true"));
        formParams.add(new BasicNameValuePair("upgrade", ""));
        formParams.add(new BasicNameValuePair("username", username));
        formParams.add(new BasicNameValuePair("password", password));
        formParams.add(new BasicNameValuePair("submit", "Secure+Log+in"));
        UrlEncodedFormEntity entity = new UrlEncodedFormEntity(formParams,HTTP.UTF_8);
        post.setEntity(entity);
        return client.execute(post);
    } catch (Exception e) {
        e.printStackTrace();
        throw new ConnectException("ERROR in doLoginPost(): "+e.getMessage());
    }
}

The server (which is not under my control) returns a 302 redirect when the login was successful, and 200 if it fails and re-loads the login page. When run with the above JAR files I get the 302 redirect, however if I run the exact same code from an Android project with the 1.6 Android JAR file on the build path I get the 200 response from the server. I get the same 200 response when running the code on my 2.2 device.

My android application has internet permissions, and the HttpGet works fine. I'm assuming that the problem lies in the fact that HttpPost (or some other class) is different in some significant way between the Android JAR version and the newer Apache versions.

I've tried adding the Apache libraries to the build path of the Android project, but due to the duplicate classes I get messages like: INFO/dalvikvm(390): DexOpt: not resolving ambiguous class 'Lorg/apache/http/impl/client/DefaultHttpClient;' in the log. I've also tried using a MultipartEntity instead of the UrlEncodedFormEntity but I get the same 200 result.

So, I have a few questions:

• Can I force the code running under Android to use the newer Apache libraries in preference to the Android versions?
• If not, does anyone have any ideas how can I alter my code so that it works with the Android JAR file?
• Are there any other, totally different approaches to doing an HttpPost in Android?
• Any other ideas?

I've read a lot of posts and code, but I'm not getting anywhere.

Answer 1

I have now given up on getting the HttpClient route to give the expected response from the server when run on Android. Instead I rewrote the doPost method above to use an HttpsURLConnection instead. Here's the new (working) version in the hope that it's useful to someone.

private static LoginBean altPost(String username, String password, String cookie, String token){
    LoginBean loginBean = new LoginBean();
    HttpsURLConnection urlc = null;
    OutputStreamWriter out = null;
    DataOutputStream dataout = null;
    BufferedReader in = null;
    try {
        URL url = new URL(URL_LOGIN_SUBMIT);
        urlc = (HttpsURLConnection) url.openConnection();
        urlc.setRequestMethod("POST");
        urlc.setDoOutput(true);
        urlc.setDoInput(true);
        urlc.setUseCaches(false);
        urlc.setAllowUserInteraction(false);
        urlc.setRequestProperty(HEADER_USER_AGENT, HEADER_USER_AGENT_VALUE_FF);
        urlc.setRequestProperty("Cookie", cookie);
        urlc.setRequestProperty("Content-Type","application/x-www-form-urlencoded");
        String output = "org.apache.struts.taglib.html.TOKEN="+ URLEncoder.encode(token, HTTP.UTF_8)
                +"&showLogin=true&upgrade=&username="+ URLEncoder.encode(username, HTTP.UTF_8)
                +"&password="+ URLEncoder.encode(password, HTTP.UTF_8)+"&submit="
                +URLEncoder.encode("Secure+Log+in", HTTP.UTF_8);
        dataout = new DataOutputStream(urlc.getOutputStream());
        // perform POST operation
        dataout.writeBytes(output);
        // get response info
        loginBean.setResponseCode(urlc.getResponseCode());
        // get required headers
        String headerName = null;
        StringBuffer newCookie = new StringBuffer(100);
        String redirectLocation = "";
        for (int i=1; (headerName = urlc.getHeaderField(i)) != null;i++) {
            if (headerName.indexOf(COOKIE_VALUE_SESSION) > -1) {
                if (newCookie.length() > 0) {newCookie.append("; ");}
                newCookie.append(headerName);
            }
            if (headerName.indexOf(COOKIE_VALUE_AUTH) > -1) {
                if (newCookie.length() > 0) {newCookie.append("; ");}
                newCookie.append(headerName);
            }
            if (headerName.indexOf("https://") > -1) {
                redirectLocation = headerName;
            }
        }
        loginBean.setCookie(newCookie.toString());
        loginBean.setRedirectUrl(redirectLocation);

        in = new BufferedReader(new InputStreamReader(urlc.getInputStream()),8096);
        String response;
        // write html to System.out for debug
        while ((response = in.readLine()) != null) {
            System.out.println(response);
        }
        in.close();
    } catch (ProtocolException e) {
        e.printStackTrace();
    } catch (IOException e) {
        e.printStackTrace();
    } finally {
        if (out != null) {
            try {
                out.close();
            } catch (IOException e) {
                e.printStackTrace();
            }
        }
        if (in != null) {
            try {
                in.close();
            } catch (IOException e) {
                e.printStackTrace();
            }
        }
    }
    return loginBean;
}

I still have no idea why the HttpClient way didn't work properly.

Answer 2

To avoid the collisions, use this JAR file for httpclient:

httplib

And this post would also be very useful:

An answer to Stack Overflow question Apache HTTP client or URLConnection