What to do when you’ve been hacked

Initial handling

First of all, don’t panic – it won’t help anything. Personally I want to try and track down the pile of slime that did this. That means trying to preserve all data, logs and possibly sniffing network traffic. However you also need to weigh up the risk of further damage to (or from) your compromised system.

Change all your passwords and any other data that might have been grabbed from the compromised system – e.g. credit card details etc. You should also check any other systems that you use from that machine or that are on the same network (I always assume that the hacker has grabbed the shell history showing which systems I’ve ssh’d to).

I find “chkrootkit” or “rkhunter” very useful for tracking down what rootkits or backdoors (if any) have been installed. Then look for information about these rootkits (via google). Most rootkits will try and hide themselves from process listings. However some use configuration files to do so – if you move those files out of the way the processes become visible once more. Look for network connections (netstat or lsof on processes) and if possible tcpdump them. Check for recently modified files (although it’s trivial to backdate changes so do not rely on only this method to find compromised files).

There are several live CD distros available designed to help you recover from an intrusion. You can also build your own if you are really keen or paranoid. Booting from a clean CD has the advantage that you know the kernel and system binaries are clean. However some rootkit are easier to detect when installed and running in the kernel. For example there is INSERT a DSL/Knoppix based live boot disk, see FrozenTech‘s LiveCD List]] for many more

Rebuilding

If your box has been hacked there is only one thing you should do – rebuild from scratch. Build a brand new machine from scratch and copy over any data that you can. Executables, shell scripts etc should all be carefully checked (or restored from backups – you do have backups don’t you? But don’t restore from the night before’s backup, because you might be restoring the attacker’s files!)

If you use debian you can use dpkg –get-selections on the compromised system to create a list of installed packages. Feeding this to dpkg –set-selections on the new system, followed by aptitude dist-upgrade should then install these packages.

Prevention

Prevention is better than a cure – particularly in this case. Some of these ideas are easier to implement than others:

• use strong passwords.
  • force others to use strong passwords by using a PAM module such as libpam-passwdqc.
• never use cleartext protocols for passwords. If you have to then use individual passwords.
• disable remote root access (for ssh).
• if you can, use SSH keys rather than password based logins.
• install minimal services on a server – e.g. if you don’t use NFS, remove it (and portmap). use nmap (check both TCP and UDP ports) and “netstat -pan” to check what is running.
• keep programs up to date – automatically if possible.
  • be careful that automated patching doesn’t break anything – it may be better to write a script to alert you of new updates so you can review them.
• sign up to security mailing lists, such as the security-announce list of your distrubition.
• run chkrootkit or rkhunter regularly and automatically to detect compromised systems early.
  • make sure these are kept up to date, just like antivirus software.
• set up aide and monitor it for changes.
  • an alternative could be tripwire.
  • put a copy of your aide DB onto a read only disk (e.g. a CD), you can then compare with something you know can’t have been tampered with.
• keep backups.
  • use a good rotation for backups.
• move services away from default ports if possible (e.g. ssh).
• use a firewall and tighten it (inbound and outbound) as much as possible.
• use IDS (intrusion detection systems) such as “snort”. use an active IDS if you can.
• data isolation. servers likely to be compromised should be isolated – private users, chroot()ed services or even better, individual servers.
• use SELinux to restrict permissions.
  • an alternative is grsecurity with PAX.
• use Xen to provide individual servers.