HttpURLConnection is not returning all headers

Question

I'm printing HTTP headers using the following code.

    URL url = new Url("htttp://example.com/example");
    HttpURLConnection conn = (HttpURLConnection) url.openConnection();

    Map<String, List<String>> map = conn.getHeaderFields();
    for (Map.Entry<String, List<String>> entry : map.entrySet()) {
        System.out.println("Key : " + entry.getKey() +
                " ,Value : " + entry.getValue());
    }

Here's the output :

Key : null ,Value : [HTTP/1.1 200 OK]
Key : Accept-Ranges ,Value : [bytes]
Key : Cache-Control ,Value : [max-age=604800, public]
Key : Connection ,Value : [Keep-Alive]
Key : Date ,Value : [Mon, 03 Oct 2016 18:01:06 GMT]
Key : ETag ,Value : ["159eb4-53dce1f957880-gzip"]
Key : Expires ,Value : [Mon, 10 Oct 2016 18:01:06 GMT]
Key : Keep-Alive ,Value : [timeout=5, max=100]
Key : Last-Modified ,Value : [Sat, 01 Oct 2016 13:59:46 GMT]
Key : Server ,Value : [Apache/2.4.12 (Unix) OpenSSL/1.0.1e-fips mod_bwlimited/1.4]
Key : Transfer-Encoding ,Value : [chunked]
Key : Vary ,Value : [Accept-Encoding,User-Agent]

Now I'm trying via curl:

$ curl -v -D - http://example.com/example -o /dev/null
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0*   Trying 111.111.111.111...
  0     0    0     0    0     0      0      0 --:--:--  0:00:01 --:--:--     0* Connected to example.com (111.111.111.111) port 80 (#0)
> GET example.com/example HTTP/1.1
> Host: example.com
> User-Agent: curl/7.47.0
> Accept: */*
> 
< HTTP/1.1 200 OK
HTTP/1.1 200 OK
< Date: Mon, 03 Oct 2016 18:25:11 GMT
Date: Mon, 03 Oct 2016 18:25:11 GMT
< Server: Apache/2.4.12 (Unix) OpenSSL/1.0.1e-fips mod_bwlimited/1.4
Server: Apache/2.4.12 (Unix) OpenSSL/1.0.1e-fips mod_bwlimited/1.4
< Last-Modified: Sat, 01 Oct 2016 13:59:46 GMT
Last-Modified: Sat, 01 Oct 2016 13:59:46 GMT
< ETag: "159eb4-53dce1f957880"
ETag: "159eb4-53dce1f957880"
< Accept-Ranges: bytes
Accept-Ranges: bytes
< Content-Length: 1416884
Content-Length: 1416884
< Cache-Control: max-age=604800, public
Cache-Control: max-age=604800, public
< Expires: Mon, 10 Oct 2016 18:25:11 GMT
Expires: Mon, 10 Oct 2016 18:25:11 GMT
< Vary: Accept-Encoding,User-Agent
Vary: Accept-Encoding,User-Agent

< 
{ [1080 bytes data]
 30 1383k   30  427k    0     0   116k      0  0:00:11  0:00:03  0:00:08  116k^C

See how the Content-Length header is missing in the java output? Why does this happen? How can I fix this?

Answer 1

Content-Length is missing because the server sent the response in chunks. It's the expected behavior. Look at the Transfer-Encoding header in the Java response, and here's RFC 2616, 4.4 Message Length (see the 3rd item of the list).