Spambots are cluttering my log file [Django]

Question

I have a nice and lovely Django site up and running, but have noticed that my error.log file was getting huge, over 150 MB after a couple of months of being live. Turns out a bunch of spambots are looking for well known URL vulnerabilities (or something) and hitting a bunch of sub-directories like http://mysite.com/ie or http://mysite.com/~admin.php etc.

Since Django uses URL rewriting, it is looking for templates to fit these requests, which raises a TemplateDoesNotExist exception, and then a 500 message (Django does this, not me). I have debug turned off, so they only get the generic 500 message, but it's filling up my logs very quickly.

Is there a way to turn this behavior off? Or perhaps just block the IP's doing this?

Answer 1

Um, perhaps, use logrotate to rotate and compress the logs periodically, if it isn't being done already.

Answer 2

If you can find a pattern in UserAgent string, you may use DISALLOWED_USER_AGENT setting. Mine is:

DISALLOWED_USER_AGENTS = (
    re.compile(r'Java'),
    re.compile(r'gigamega'),
    re.compile(r'litefinder'),
)

See the description in Django docs.