nginx req limit & fail2ban & cloudflare large attacks

Question

I started to use as the topic says nginx with req limit + fail2ban + cloudflare. If someone exceeds the requests on any php file the ip gets blocked over the cloudflare api in the "ip firewall" on cloudflares side. Its working fine, but the issue is, that i got attacked today with about 1000~ bots which are able to bypass the javascript check (aka under attack mode) and overloaded my website behind. Fail2ban directly started to block over 350 different ips, but for some reason after that it failed badly. The nginx error log got flooded with "limiting requests, excess:" so fast, that im unable to read anything while i runned tail -f at it. Many ips should be blocked since they also exceeded the limits, but fail2ban didnt blocked any new ips anymore, it just repeated all the time x.x.x.x already banned in the log file. Maybe the issue is, that fail2ban fails to read that much new log entrys, that its unable to work properly.

My current config:

from the nginx.conf:

limit_req_zone  $binary_remote_addr  zone=one:100m  rate=1r/s;

in the sites-enabled php block:

limit_req zone=one burst=5;

jail.config:

[http-dos]
enabled  = true
filter   = http-dos
action   = cloudflare-blacklist
logpath  = /var/log/nginx/error.log
maxretry = 10
findtime = 600
bantime  = 3600

and the failregex:

failregex = limiting requests, excess:.* by zone.*client: <HOST>

And that you see how large the error.log is after the attack with limiting requests:

wc -l /var/log/nginx/error.log
602582 /var/log/nginx/error.log

Another issue which showed up was, that slow bots which are slow and are only able to send like 2-3 requests per second or even less wont get blocked with this config, but if i lower the values on the config it will start to block normal users also. Obviously 2-3 requests per second wont hurt the site, but there are over 100 bots which are slow like that and so it sums up to 300-400 r/s which wont get blocked. Thats also not that much, but since the website behind is a board which runs mysql querys everytime you request the index.php it causes a very high cpu load.

I have no idea how i could fix this issue, maybe one of you knows a solution for this problem. Also im sorry for my english, im teaching it myself and hope you can understand what i wrote.

Answer 1

Many ips should be blocked since they also exceeded the limits, but fail2ban didnt blocked any new ips anymore

I can't see you custom action cloudflare-blacklist, but you have to sure that your API calls was made. It could happen that your calls were rejected, because there are 1200req/5min limit. You can ask Cloudflare support, if your logging doesn't provide such info.

Also I'm not sure what happens when you reach max size of IP rules list. Maybe they reject following attempts to create an entry in the list, and your further violators was not banned at all.

it just repeated all the time x.x.x.x already banned

I believe it's a normal behavior. Some user with IP x.x.x.x has violated your rate limits n times, so n log entries was created. Your jail has maxretry = 10, so after 10 repetitive log entries user have to be banned. After that fail2ban reads remaining n-10 log entries and sees that the user continues to violate your rate-limits, but instead of banning him, it just writes that he is already banned.

Obviously 2-3 requests per second wont hurt the site, but there are over 100 bots which are slow like that and so it sums up to 300-400 r/s which wont get blocked.

According to your allowed rate of 1r/s with burst of 5 they still should be rate-limited periodically and eventually banned. If they're not breaking your limits and it's hard to distinguish bots from actual users, you can define another jail or limit_req_zone for a total number of requests over last hour.

Also Nginx caching could help in certain scenarios. Check these DigitalOcean and Runcloud articles.