Why do real-world servers prefer gzip over deflate encoding?

Question

We already know deflate encoding is a winner over gzip with respect to speed of encoding, decoding and compression size.

So why do no large sites (that I can find) send it (when I use a browser that accepts it)?

Yahoo claims deflate is "less effective". Why?

I maintain HTTP server software that prefers deflate, so I'd like to know if there's some really good reason not to continue doing so.

Answer 1

There is some confusion about the naming between the specifications and the HTTP:

• DEFLATE as defined by RFC 1951 is a compressed data format.
• ZLIB as defined by RFC 1950 is a compressed data format that uses the DEFLATE data format.
• GZIP as defined by RFC 1952 is a file format that uses the DEFLATE compressed data format.

But the HTTP uses a different naming:

• gzip An encoding format produced by the file compression program "gzip" (GNU zip) as described in RFC 1952 [25]. This format is a Lempel-Ziv coding (LZ77) with a 32 bit CRC.

• deflate The "zlib" format defined in RFC 1950 [31] in combination with the "deflate" compression mechanism described in RFC 1951 [29].

So to sum up:

• gzip is the GZIP file format.
• deflate is actually the ZLIB data format. (But some clients do also accept the actual DEFLATE data format for deflate.)

See also this answer on the question What's the difference between the "gzip" and "deflate" HTTP 1.1 encodings?:

What's the difference between the "gzip" and "deflate" HTTP 1.1 encodings?

"gzip" is the gzip format, and "deflate" is the zlib format. They should probably have called the second one "zlib" instead to avoid confusion with the raw deflate compressed data format. While the HTTP 1.1 RFC 2616 correctly points to the zlib specification in RFC 1950 for the "deflate" transfer encoding, there have been reports of servers and browsers that incorrectly produce or expect raw deflate data per the deflate specification in RFC 1951, most notably Microsoft. So even though the "deflate" transfer encoding using the zlib format would be the more efficient approach (and in fact exactly what the zlib format was designed for), using the "gzip" transfer encoding is probably more reliable due to an unfortunate choice of name on the part of the HTTP 1.1 authors.

Answer 2

From my minimal testing it appears most HTTPds either:

1. don't support deflate on-the-fly: Apache's mod_deflate (a suprise), GWS
2. or prefer to send gzip: IIS, lighttpd's mod_compress

So to send deflate on the most popular server (Apache), you must maintain pre-encoded files and use mod_negotiate (you might even have to use type-maps to prefer deflate).

I'd guess, due to this hassle, deflate is just rarely used, and therefore bugs are more likely to exist in client deflate support than in gzip support.