Python secure websocket memory consumption

Question

I am writing a web socket server in python. I have tried the approach below with txws, autobahn, and tornado, all with similar results.

I seem to have massive memory consumption with secure websockets and I cannot figure out where or why this might be happening. Below is an example in tornado, but I can provide examples in autobahn or txws.

import tornado.httpserver
import tornado.websocket
import tornado.ioloop
import tornado.web
import json 

class AuthHandler(tornado.websocket.WebSocketHandler):

    def open(self):
        print 'new connection for auth'

    def on_message(self, message):
        message = json.loads(message)
        client_id = message['client_id']
        if client_id not in app.clients:
            app.clients[client_id] = self
        self.write_message('Agent Recorded')

    def on_close(self):
        print 'auth connection closed'


class MsgHandler(tornado.websocket.WebSocketHandler):

    def open(self):
        print 'new connection for msg'

    def on_message(self, message):
        message = json.loads(message)
        to_client = message['client_id']
        if to_client in app.clients:
            app.clients[to_client].write_message('You got a message')

    def on_close(self):
        print 'msg connection closed'


app = tornado.web.Application([
    (r'/auth', AuthHandler),
    (r'/msg', MsgHandler)
])

app.clients = {}


if __name__ == "__main__":
    http_server = tornado.httpserver.HTTPServer(app, ssl_options={
        'certfile': 'tests/keys/server.crt',
        'keyfile': 'tests/keys/server.key'
    })
    http_server.listen(8000)
    tornado.ioloop.IOLoop.instance().start()

After making around 10,000 connections I find I am using around 700MB of memory with SSL compared to 43MB without, and I never get it back unless I kill the process. It seems like the problem is closely tied to the amount of connections made rather than messages sent.

The consumption seems to happen independent of the client (I wrote my own client and tried other clients).

Are secure websockets really that much more memory intensive that plain websockets? Or is my server code not implementing it correctly?

Answer 1

I think the best solution is to use a real webserver (nginx apache) as a proxy and let it manage the ssl layer.