Tomcat gzip while chunked issue

Question

I'm expiriencing some problem with one of my data source services. As it says in HTTP response headers it's running on Apache-Coyote/1.1. Server gives responses with Transfer-Encoding: chunked, here sample response:

HTTP/1.1 200 OK
Server: Apache-Coyote/1.1
Content-Type: text/xml;charset=utf-8
Transfer-Encoding: chunked
Content-Encoding: gzip
Date: Tue, 30 Mar 2010 06:13:52 GMT

And problem is when I'm requesting server to send gzipped request it often sends not full response. I recieve response, see that last chunk recieved, but then after ungzipping I see that response is partial. I never seen such behavior with gzip turned off in request headers.

So my question is: is it common tomcat issue? maybe one of it's mod which is doing compression? Or maybe it maybe some kind of proxy issue? I can't tell about versions of tomcat or what gzip mod they use, but feel free to ask, i'll try ask my service provider.

Thanks.

Answer 1

Since the content length of a gzipped response is unpredictable and it's potentially expensive and slow to compress it fully in memory first, then calculate the length and then stream the gzipped response from memory, the average webserver will send them in chunks using Transfer-Encoding: chunked without a Content-Length header.

Since it's a homegrown HTTP client, it sounds like as if it doesn't handle chunked requests correctly. You have to determine the Transfer-Encoding response header and if it equals to chunked, then you have to parse it as a chunked stream.

You can learn from the aforementioned HTTP spec links and from Wikipedia how to parse a chunked stream. Each chunk consists of a header denoting the chunk length in hexadecimal, then a CRLF, then the actual chunk content, then a CRLF. This is repeated until a chunk with a header denoting the chunk length of 0. You need to ungzip the chunks separately and then glue them together.

To save all the tedious coding work (likely also for the remnant of your homegrown HTTP client), I strongly recommend to have a look at Apache HttpComponents Client.