Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

CodeIgniter Session spammed by bots

I have a site with roughly 2000 visitors per day, and the site is spammed with the various search engine bots. I tried reducing the session expire time to 20 minutes, and still I get alot of mysql_slow_queries. So I was looking into the article, Google crawler, cron and codeigniter sessions, to fully ignore the bots from the sessions table, but the the way they do is, ignore the IPs, but as I was analyzing the database I see that the same bot uses different IPs. I noticed that the bots use the same user agent everytime though, so is it safe to ignore the user agents instead? What could be some of the necessary steps to avoid slow queries and ignore the bots?

Some of the SLOW Queries

INSERT INTO `ci_sessions` (`session_id`, `ip_address`, `user_agent`, `last_activity`, `user_data`) VALUES ('619bfd8ef4171480645feb17a15323ee', '219.92.135.144', 'Mozilla/5.0 (Windows; U; Windows NT 6.1; en-US; rv:1.9.2.15) Gecko/20110303 Firefox/3.6.15', 1384875135, '')

INSERT INTO `ci_sessions` (`session_id`, `ip_address`, `user_agent`, `last_activity`, `user_data`) VALUES ('fa48b5168b8e84d90dc9b87ce65dfc89', '66.249.74.112', 'Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)', 1384875522, '')
like image 552
Suthan Bala Avatar asked Dec 19 '22 21:12

Suthan Bala


1 Answers

Edit your user_agent.php in /config, and add the bots you see in your session, adding them to the bot section should eliminate the sessions from logging.

// There are hundreds of bots but these are the most common.
$robots = array(
    'googlebot'     => 'Googlebot',
    'msnbot'        => 'MSNBot',
    'baiduspider'   => 'Baiduspider',
    'bingbot'       => 'Bing',
    'slurp'         => 'Inktomi Slurp',
    'yahoo'         => 'Yahoo',
    'askjeeves'     => 'AskJeeves',
    'fastcrawler'   => 'FastCrawler',
    'infoseek'      => 'InfoSeek Robot 1.0',
    'lycos'         => 'Lycos',
    'yandex'        => 'YandexBot'
);

You can reduce the # of bots, but won't eliminate them. This user-agent process could be used to create a MY_session.php and then exclude session creation for agents matching bots.

EDIT:
I went ahead and created this on github and documented here:
http://blog.biernacki.ca/2014/01/codeigniter-keeping-bots-out-of-your-sessions-table-or-how-i-cleaned-up-my-sessions/

Enjoy

like image 58
Jakub Avatar answered Dec 24 '22 02:12

Jakub