Is it possible to use gzip compression with Server-Sent Events (SSE)?

Question

I would like to know if it is possible to enable gzip compression for Server-Sent Events (SSE ; Content-Type: text/event-stream).

It seems it is possible, according to this book: http://chimera.labs.oreilly.com/books/1230000000545/ch16.html

But I can't find any example of SSE with gzip compression. I tried to send gzipped messages with the response header field Content-Encoding set to "gzip" without success.

For experimenting around SSE, I am testing a small web application made in Python with the bottle framework + gevent ; I am just running the bottle WSGI server:

@bottle.get('/data_stream')
def stream_data():
    bottle.response.content_type = "text/event-stream"
    bottle.response.add_header("Connection", "keep-alive")
    bottle.response.add_header("Cache-Control", "no-cache")
    bottle.response.add_header("Content-Encoding", "gzip")
    while True:
        # new_data is a gevent AsyncResult object,
        # .get() just returns a data string when new
        # data is available
        data = new_data.get()
        yield zlib.compress("data: %s\n\n" % data)
        #yield "data: %s\n\n" % data

The code without compression (last line, commented) and without gzip content-encoding header field works like a charm.

EDIT: thanks to the reply and to this other question: Python: Creating a streaming gzip'd file-like?, I managed to solve the problem:

@bottle.route("/stream")
def stream_data():
    compressed_stream = zlib.compressobj()
    bottle.response.content_type = "text/event-stream"
    bottle.response.add_header("Connection", "keep-alive")
    bottle.response.add_header("Cache-Control", "no-cache, must-revalidate")
    bottle.response.add_header("Content-Encoding", "deflate")
    bottle.response.add_header("Transfer-Encoding", "chunked")
    while True:
        data = new_data.get()
        yield compressed_stream.compress("data: %s\n\n" % data)
        yield compressed_stream.flush(zlib.Z_SYNC_FLUSH)

Answer 1

TL;DR: If the requests are not cached, you likely want to use zlib and declare Content-Encoding to be 'deflate'. That change alone should make your code work.

---

If you declare Content-Encoding to be gzip, you need to actually use gzip. They are based on the the same compression algorithm, but gzip has some extra framing. This works, for example:

import gzip
import StringIO
from bottle import response, route
@route('/')
def get_data():
    response.add_header("Content-Encoding", "gzip")
    s = StringIO.StringIO()
    with gzip.GzipFile(fileobj=s, mode='w') as f:
        f.write('Hello World')
    return s.getvalue()

That only really makes sense if you use an actual file as a cache, though.

Answer 2

There's also middleware you can use so you don't need to worry about gzipping responses for each of your methods. Here's one I used recently.

https://code.google.com/p/ibkon-wsgi-gzip-middleware/

This is how I used it (I'm using bottle.py with the gevent server)

from gzip_middleware import Gzipper
import bottle
app = Gzipper(bottle.app())
run(app = app, host='0.0.0.0', port=8080, server='gevent')

For this particular library, you can set w/c types of responses you want to compress by modifying the DEFAULT_COMPRESSABLES variable for example

DEFAULT_COMPRESSABLES = set(['text/plain', 'text/html', 'text/css',
'application/json', 'application/x-javascript', 'text/xml',
'application/xml', 'application/xml+rss', 'text/javascript',     
'image/gif'])

All responses go through the middleware and get gzipped without modifying your existing code. By default, it compresses responses whose content-type belongs to DEFAULT_COMPRESSABLES and whose content-length is greater than 200 characters.