Varnish: performance impact large ban list

Question

We were wondering if anyone has experience with a large amount of bans in Varnish. We consider a ban strategy which could result in a couple of hundred (smart) bans each night (on X million cache objects).

Although I am aware that this is highly dependent on environment variables we were wondering if this have a significant performance impact.

Answer 1

Bans are quite CPU intensive so care should be taken not to overuse them. If you do, CPU usage will rise and you'll notice a huge amount of regular expression matches will be executed each second.

In general one ban will match against every object in memory at the point it is entered, so having a million object each ban will result in a million ban evaluation. This might sound like a lot but modern servers are fast and today a modern server is be capable of doing tens of millions of regular expression matches each second. My four year old laptop does something like 15 million regex matches a second running on a single core, just to give you an idea of the scale.

In addition there is another feature of Varnish that comes into play. The ban lurker. The ban lurker is a thread that walks the cache and evaluates bans trying to kill of objects before they are requested, thereby reducing the size of the ban list. If your bans don't use the req object they are candidates for evaluation by the lurker. If you plan on using a few bans you should take care to write your bans in a lurker friendly fashion. So called "smart bans", which you seem to be familiar with.

All in all I think your setup sounds sane. Issuing a couple of hundred smart bans with a few million objects in cache will probably work just fine. There will of course be a bit of CPU load when the bans are deployed and the TTFB will increase somewhat, but I think you'll be fine. You might want to play somewhat with the parameters that control how the ban lurker works, but try the defaults first, they are pretty sane.