Best methods to clean up a hacked site with no clean version available?

Question

I have been asked to fix a hacked site that was built using osCommerce on a production server.

The site has always existed on the remote host. There is no offline clean version. Let's forget how stupid this is for a moment and deal with what it is.

It has been hacked multiple times and another person fixed it by removing the web shell files/upload scripts.

It is continually hacked often.

What can I do?

Answer 1

Because you cannot trust anything on the web host (it might have had a rootkit installed), the safest approach is to rebuild a new web server from scratch; don't forget to update all the external-facing software before bringing it online. Do all the updating on the happy side of a draconian firewall.

When you rebuild the system, be sure to pay special attention to proper configuration. If the web content is owned by a different Unix user than the web server's userid and the permissions on the files are set to forbid writing, then the web server cannot modify the program files.

Configure your web server's Unix user account so it has write access to only its log files and database sockets, if they are in the filesystem. A hacked web server could still serve hacked pages to clients, but a restart would 'undo' the 'live hack'. Of course, your database contents could be sent to the Yakuza or corrupted by people who think your data should include pictures of unicorns. The Principle of Least Privilege will be a good guideline -- what, exactly, does your web server need to access in order to do its job? Grant only that.

Also consider deploying a mandatory access control system such as AppArmor, SELinux, TOMOYO, or SMACK. Any of these systems, properly configured, can control the scope of what can be damaged or leaked when a system is hacked. (I've worked on AppArmor for ten years, and I'm confident most system administrators can learn how to deploy a workable security policy on their systems in a day or two of study. No tool is applicable to all situations, so be sure to read about all of your choices.)

The second time around, be sure to keep your configuration managed through tools such as as puppet, chef, or at the very least in a revision control system.

Update

Something else, a little unrelated to coming back online, but potentially educational all the same: save the hard drive from the compromised system, so you can mount it and inspect its contents from another system. Maybe there's something that can be learned by doing forensics on the compromised data: you might find that the compromise happened months earlier and had been stealing passwords or ssh keys. You might find a rootkit or further exploit tools. You might find information to show the source of the attack -- perhaps the admin of that site doesn't yet realize they've been hacked.

Be careful when inspecting hacked data -- that .jpg you don't recognize might very well be the exploit that cracked the system in the first place, and viewing it on a 'known good' system might crack it, too. Do the work with a hard drive you can format when you're done. (Virtualized or with a mandatory access control system might be sufficient to confine "passive" data-based hacks, but there's nothing quite like throwaway systems for peace of mind.)

Answer 2

Obtain a fresh copy of the osCommerce version the site was built with, and do a diff between the new fresh osCommerce and the hacked site. Also check for files which exist on the server but not in the osCommerce package.

By manually comparing the differences, you can track down all possible places the hack may have created or modified scripts.