Please first read the LQ FAQ: Security references
, post 1, under "Compromise, breach of security, detection". If you don't know anything but want to take action *now* (your choice), then at least read:
Intruder Detection Checklist (CERT): http://www.cert.org/tech_tips/intrud...checklist.html
Steps for Recovering from a UNIX or NT System Compromise: http://www.cert.org/tech_tips/root_compromise.html
Formatting and Reinstalling after a Security Incident (SF): http://www.securityfocus.com/infocus/1692
...and your choice of threads searching the LQ Linux - Security forum searching for the keyword "compromise" or "hacked".
They compromised root
Clues from the system and application logfiles and files (tarballs, suid binaries)?
I've tried to delete this file and make changes but nothing seems to work.. less, more, cat, vi, vim.. all rendered useless in a read-only environment.
Meaning you remounted all disks readonly?
I was able to login to the server both as my user name
...which means it was still up and running. If it was, and the HD's where not mounted readonly directly after you noticed the compromise, this means running processes corrupted any "evidence".
However, the entire files are read only and issuing an input/output error.
Could be failing HW. Don't reboot or risk loosing data.
I wish to run an fsck
BAD THING TO DO if you want to find out more about the compromise.
however I am unable to umount the drives:
bash:/home/user# umount /dev/hda1
-su: /bin/umount: Input/output error
Dropping to runlevel 1 then remounting the disks readonly.
Is there a way to handle this without going to the datacenter and booting into a cd?
Yes and no, the decision is yours. It depends on what you're willing to trade off. If you want to trade in any forensics for just a quick glance before reinstalling from scratch (restore only if you exactly know what youre doing), then kill all processes that aren't necessary or outright suspected (lsof), disable all unnecessary users, reinstall sshd on a high port (dump the config first on the system) then run Chkrootkit, Rootkit Hunter, package manager in verify mode or file integrity checker. If you're done or if you don't want to learn more, at least (have them) save application version info, the system and application logs, auth files, contents of dirs (accessable by unprivileged users), after running mentioned apps. Then repartition, reformat and reinstall from scratch, unless you can do a restore with a backup which contents can be verified.
If you OTOH want to learn more, your first task will be to (order the colo ppl to) preserve any "evidence" left by any means. In the best circumstances this means killing the box cold (forced shutdown), mounting the disks in another box and making dd images of the full disk. If you're not on a shared host, remounting disks read-only and making dd images of all disks could do. Elif this is a shared hosting setup, just make dd images of all disks and hope for the best. In any case this way you'll have (some form of) disk image to do forensics on.
Boot from a CD (e.g. Knoppix or other Live CD) to have a look around. Be ready to reinstall if you aren't confident that you'v got everything back the way it should be.
How can you make 100 percent sure a server is "restored" "back the way "it should be""?