POSTing to HTTPS in Java, weird behaviour

Question

I decided to POST some stuff with Java but got stuck on such a seemingly trivial task. Then I got frustrated and rewrote the whole thing in php, and it works flawlessly. I tried comparing the outputs of both Java and PHP results, and what I notice that Curl sets Content-Type: multipart/form-data; whereas Java (correctly?) sets it to application/x-www-form-urlencoded; but this triggers 500 internal server error. What the hell is going on?

Java debug:

2017/12/11 19:10:36:236 EET [DEBUG] wire - http-outgoing-0 >> "POST /auth/register HTTP/1.1[\r][\n]"
2017/12/11 19:10:36:243 EET [DEBUG] wire - http-outgoing-0 >> "Cookie: XSRF-TOKEN=eyJpdiI6InJJRjdORWdzUVcwQVZMR2l2WnNHUWc9PSIsInZhbHVlIjoiZVFSc0V5MCtURFE4dTZHZGlwVXdna0VGMGZtNzh0aEF4eXJcL0ZcL3d6QWM4NVdJejJaeEptUDFcL0ZDeXBEOGlDMTltbEQ4cFg0c1wvK3h4Nkp3VEhJTmFRPT0iLCJtYWMiOiIzNmRjZGNmZGNmYTFkYTQzODQ2NjFkZWY3ZWVlZGJmNzBiNDFhNTQwNDU3ODAzMTA4MGNhYWRiY2VhNDU2ZmU2In0%3D;laravel_session=eyJpdiI6ImF3cGg2TUFvWm54b3J4Nml5NnlBYlE9PSIsInZhbHVlIjoiXC85ZzdJS2drRWxlWGExXC93bHFVNXRtTmFtTmcyblJ4cXY4eUhCY2toaWJGaFBcL2NjQllKekVrUWFvblhydWtSeHpySm4yWGlWbHE3Y3dXZjFxd3lXV3c9PSIsIm1hYyI6IjRlNGZiMjU4NDRmMWVjZjc1YzExYWM4ZjJlMTUyNzI0ZTY3NTAwYTUyZTdlNTdiZmQ2ZDg1NTk1OGE4OGQ3ZGMifQ%3D%3D[\r][\n]"
2017/12/11 19:10:36:243 EET [DEBUG] wire - http-outgoing-0 >> "User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:57.0) Gecko/20100101 Firefox/57.0[\r][\n]"
2017/12/11 19:10:36:243 EET [DEBUG] wire - http-outgoing-0 >> "Accept: */*[\r][\n]"
2017/12/11 19:10:36:243 EET [DEBUG] wire - http-outgoing-0 >> "Content-Length: 146[\r][\n]"
2017/12/11 19:10:36:243 EET [DEBUG] wire - http-outgoing-0 >> "Content-Type: application/x-www-form-urlencoded; charset=UTF-8[\r][\n]"
2017/12/11 19:10:36:243 EET [DEBUG] wire - http-outgoing-0 >> "Host: xxx[\r][\n]"
2017/12/11 19:10:36:243 EET [DEBUG] wire - http-outgoing-0 >> "Connection: Keep-Alive[\r][\n]"
2017/12/11 19:10:36:243 EET [DEBUG] wire - http-outgoing-0 >> "Accept-Encoding: gzip,deflate[\r][\n]"
2017/12/11 19:10:36:243 EET [DEBUG] wire - http-outgoing-0 >> "[\r][\n]"
2017/12/11 19:10:36:244 EET [DEBUG] wire - http-outgoing-0 >> "username=jnfjkwejf&_token=ZTE8t5XDf2vmBcAWpSenELkEtqvhIp9FBSy0E1Ez&domain=xxx&password=qweqweqwe&password_confirmation=qweqweqwe&captcha=U8Aq3"
2017/12/11 19:10:36:319 EET [DEBUG] wire - http-outgoing-0 << "HTTP/1.1 500 Internal Server Error[\r][\n]"

---

Java code:

HttpHost target = new HttpHost("xxx", 443, "https");
        SSLContext sslcontext = SSLContexts.createSystemDefault();
        SSLConnectionSocketFactory sslConnectionSocketFactory = new SSLConnectionSocketFactory(
                sslcontext, new String[] { "TLSv1.2", "SSLv3" }, null,
                SSLConnectionSocketFactory.getDefaultHostnameVerifier());

        Registry<ConnectionSocketFactory> socketFactoryRegistry = RegistryBuilder.<ConnectionSocketFactory>create()
                .register("http", PlainConnectionSocketFactory.INSTANCE)
                .register("https", sslConnectionSocketFactory)
                .build();
    PoolingHttpClientConnectionManager cm = new PoolingHttpClientConnectionManager(socketFactoryRegistry);

    CloseableHttpClient httpsclient = HttpClients.custom()
            .setSSLSocketFactory(sslConnectionSocketFactory)
            .setConnectionManager(cm)
            .build();
    ArrayList<NameValuePair> formparams = new ArrayList<>();
    formparams.add(new BasicNameValuePair("username", "jnfjkwejf"));
    formparams.add(new BasicNameValuePair("_token", _token));
    formparams.add(new BasicNameValuePair("domain", "xxx"));
    formparams.add(new BasicNameValuePair("password", "qweqweqwe"));
    formparams.add(new BasicNameValuePair("password_confirmation", "qweqweqwe"));
    formparams.add(new BasicNameValuePair("captcha", captcha));
    UrlEncodedFormEntity entity = new UrlEncodedFormEntity(formparams, Consts.UTF_8);
    HttpPost httppost = new HttpPost("/auth/register");
    String fullCookie =
            captchaConn.getHeaderFields().get("Set-Cookie").get(1).split(";")[0] + ";" +
            captchaConn.getHeaderFields().get("Set-Cookie").get(0).split(";")[0];
    httppost.addHeader("Cookie", fullCookie);
    httppost.addHeader("User-Agent", "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:57.0) Gecko/20100101 Firefox/57.0");
    httppost.addHeader("Accept", "*/*");
    httppost.setEntity(entity);
    CloseableHttpResponse execute = httpsclient.execute(httppost);

PHP:

> POST /auth/register HTTP/1.1
Host: xxx
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:57.0) Gecko/20100101 Firefox/57.0
Accept: */*
Cookie: XSRF-TOKEN=eyJpdiI6Im4zSXNHanBpM1FmXC9oZ0dEeFZFbElRPT0iLCJ2YWx1ZSI6Inh2S2FKSktQMGFxREMwVU5KZHkzSVEzaXFyVnNWdEpiQ1IzVFd4Y3c3RWFxQXV6YXFRaFNuQlBSU2M5bEs5azh2dG9zSjFoQWlNVW00dGgzeW1IeFhRPT0iLCJtYWMiOiI4MTUwZTk3ZDNkZDMyOTkxMjRkNjRhY2I5MjEzMDZmNTk5NzUwYjA5NDY3YmY0OWQ4YzQ1NmMxNTVjZDIwNzNkIn0%3D; laravel_session=eyJpdiI6IldnbGd1STlUSVZXc0NFbWZLTVhGZEE9PSIsInZhbHVlIjoiRDBndzlSZFJLUlZHYWdJWmppZmNCOWRYV2liNnV1NFJkMzNoNEpyVGRBaXlEUm94MzNSbmZ4YVdDeHM0OTNNa21qZmQ1Tjd4UVJrK2pYS3BUQUhsbFE9PSIsIm1hYyI6ImZkNTBmZjYxOTc0ZmUxNzIzZTNlOTBkYWJmMzBkODhkODkxNTk1Mjc5Nzg5MTI5NmJkYzJlYzBjOTEyMGM0OTUifQ%3D%3D
Content-Length: 718
Expect: 100-continue
Content-Type: multipart/form-data; boundary=------------------------e939a893b45ae97c

< HTTP/1.1 100 Continue
< HTTP/1.1 302 Found
< Server: nginx/1.10.3
< Content-Type: text/html; charset=UTF-8
< Transfer-Encoding: chunked
< Connection: keep-alive
< Cache-Control: no-cache
...

---

Curl options:

curl_setopt($ch, CURLOPT_URL, 'https://xxx/auth/register');
curl_setopt($ch, CURLOPT_FOLLOWLOCATION, 1);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
curl_setopt($ch, CURLOPT_HEADER, 1);
curl_setopt($ch, CURLOPT_USERAGENT, 'Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:57.0) Gecko/20100101 Firefox/57.0');
curl_setopt($ch, CURLOPT_COOKIEFILE, __DIR__ . '.\cookiejar');
curl_setopt($ch, CURLOPT_COOKIEJAR, __DIR__ . '.\cookiejar');
curl_setopt($ch, CURLOPT_POST, 1);
curl_setopt($ch, CURLOPT_POSTFIELDS, $postfields);

Answer 1

If your server is expecting Content-Type: multipart/form-data, then you should use MultipartEntityBuilder instead of UrlEncodedFormEntity.

Your entity creation code could be something like

HttpEntity entity = MultipartEntityBuilder
    .create()
    .addTextBody("username", "jnfjkwejf")
    .addTextBody("_token", _token)
    .addTextBody("domain", "xxx")
    .addTextBody("password", "qweqweqwe")
    .addTextBody("password_confirmation", "qweqweqwe")
    .addTextBody("captcha", captcha)
    .build();

Refer this link for more details. Apache HttpClient making multipart form post

Update - One more point.

Your cookie header does not look to be correct in Java case. The cookie pairs should be separated by a semi-colon and a space ('; '). space is missing in your java case. Ref - https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Cookie

Answer 2

curl can use any of the two encodings, just like php or java, it is simply a matter of configuring the request type correctly.

urlencoded is old style multipart is the new kid on the block, java must be capable of generating both, perhaps finding a new java example?

Also, why not try building the actual request yourself and send the actual bytes you need to send, by creating the request yourself byte per byte? consider it a learning exercise.

In order to debug why the server sends 500 errors, I recommend you read your server's source code and also read your server's logs, it may only like the new way of doing stuff.

btw I like the old way, i like the simple. it used to be simple, an POST request, back in my day... now, the new way, feels wasteful, too many bytes, too few data.