SSL slowness in EC2

Question

We've deployed our rails app to EC2. In our setup, we have two proxies on small instances behind round-robin DNS. These run nginx load balancers for a dynamically growing and shrinking farm of web servers. Each web server also runs nginx with a cluster of mongrels. The nginx here takes care of static content and load balancing the mongrels.

Anyway, our traffic by-and-large is HTTPS. We have the 2 proxies taking care of SSL. I've noticed that our network throughput on those instances caps out at only 60 Mbps or so. To contrast, in testing I am able consistently to get 700+ Mbps on a small instance via regular HTTP. In fact, this is the same as what I can get on a large instance. Similar to what the Right Scale guys got in their testing. (Amazon says a small gets "moderate" network I/O, while a large gets "high". If I had to speculate, I think this is just their way of saying that there are more small instances per physical box sharing one network card. I'm not sure if it means that a large gets a dedicated network interface, but I would doubt it.)

In testing, I was able to get a large instance to get about 250 Mbps SSL. This says to me that the CPU or some other resource is the bottleneck. However, our monitoring graphs don't show the CPU on our proxies being particularly busy.

My questions are:

1. Is my instinct about SSL being slower due to CPU correct and our monitoring graphs are wrong? Or could some other resource be the limiting factor?
2. Should we just take the extra cost and put the proxies on high-CPU instances? Or would it be better to do just add more small instances?
3. Should we offload the SSL termination to the web servers? This introduces one more problem, though: how do we get the client IP address in our application? Right now our proxy sets it in the X-FORWARDED-FOR header, but obviously this wouldn't be possible if it's not decrypting SSL.

I'd love to hear about any similar setups. We tinkered a bit with their Elastic Load Balancer, but I think that basically puts us in the same situation as #3 above. Has anyone else made the switch to ELB and found it to be worth it?

Answer 1

Are you using the SSL session cache that nginx provides? That can help nginx save on cycles constantly re-working-out the encryption. See http://wiki.nginx.org/NginxHttpSslModule#ssl_session_cache

What monitoring are you using to determine your cpu usage? SSL is typically very CPU intensive.

I would keep the SSL proxies as a designated layer, that way you can scale the cost of negotiating ssl separately from other concerns.