Hello....


I run a small webserver. Hardly a target for hackers, or so I thought. I had set up what I thought was fairly paranoid security, but I apparently missed something. Yesterday I found that my root password had been changed and a few other things had been screwed with. The website content had not been altered in any way, so I'm assuming the point of the hack was either just for entertainment for the hacker, or to use my server as a launchpad for another hack, making it harder to be traced.


I got the ip addresses from which my server was hacked from /var/log/messages, and forwarded them to the abuse departments of my ISP, and the ISP's of the machines associated. I was careful to not make direct accusations, as it is just as likely as not that the servers from which my server was hacked were just as much victimized as mine.


Anyways, on to the point. I have gone through and found the things that are broken. It looks like my smtp daemon was broken (perhaps exploited, dunno), my FTP daemon was broken, and a few vital programs were broken, along, of course, with the root password being changed. I was able to boot from an emergency recovery disk and fix the accounts so that the root password was no longer compromised, and at the moment, the only port that is open is 80 for Apache.

The other things that are broken include: ifconfig, route, ps, and grep. I've not found anything else. Since this has happened, I'm planning on making some big changes in terms of security, but I need advice as to whether I should try to rebuild and reconfigure from where things are, or if I should back up my content and data, and just start fresh with a new installation of Linux... I got ifconfig, route, and grep back, but ps is returning the error "No such file or directory" despite the fact that I can see it sitting right there and cat it and get output (lovely beeps and stuff, but output nonetheless). I have to figure out where to get ps from if I'm going to pick up the pieces.

Also, my MTA is/was sendmail. I have always been *exceedingly* careful to keep all daemons and software as up to date as I knew how to prevent leaving gaping security holes open. I know that sendmail has been notorious in the past about big security holes, but I found it to be the easiest to configure for my purposes. Should I ditch it and go with something else, like qmail or another MTA?

My FTP daemon was wu-ftpd. Is that a relatively secure daemon to run? I have pure-ftpd, and it's a heck of a lot easier to set up, but I'm worried about security with that. Right now, there are only 4 ip addresses that can get through inetd to use FTP, but I know that ip addresses can be spoofed. Should I go with any particular FTP daemon?


I think that my real vulnerability was that I had SSHd running and left SSH 1.0 open. I'm going to leave that off as I restore things, as I have a dumb terminal that I use for 90% of my administrative tasks anyways.


Any advice will be helpful, even if you don't cover all of my lengthy post! I tend to be wordy at time, and I hope that whoever reads this can wade through the extraneous stuff.


Well, back to studying my security books. Aparently I'm missing quite a bit!

I'm sorry to hear this. For now: get the box off the net. You don't know if you're running a rootshell on some high port. Even after finding out which daemon(s) (is|are) exploited, what rootkit was used, which of the ps-utils have been replaced, etc etc, there should be no other clear answer than
"rebuild your box from scratch".

What you will need to do in the first place (as a lot of other people should do) is get an more active grip on your boxens security. I know it takes discipline, but subscribing to your vendors + a general Linux mailinglist (and actually reading it) :-] , looking at SANS, CERT and SecurityFocus advisories shouldn't take much of your time. Acting on that info can be the difference between having root and being rooted.

Securing your server boxen should start at the base. Don't install development stuff, and compile a new kernel with either GRSecurity or LIDS which will take away a lot of capabilities that can be used against the system. Don't have user accounts,
and when needed use sudo to get root access. Set resource limits on processes and disk quota, and use PAM for authentication. When using GRSecurity, profile your binaries with "grtrace" and group your users as mentioned in the docs for allowing/taking away exec/suid/proc/net_raw and other capabilities, and turn logging on.

Lock down binaries (chattr +iu) when done installing and configuring or consider running em off a "-o ro" mounted partition. Add passive integrity detection to check up on your system's state using Aide or Tripwire (and save the databases offsite), add chkrootkit, and maybe top it off with another checking package like COPS, Tiger, TARA or the like.

Stop and uninstall all unnecessary services. Review any daemon version before installing. Ditch X11, Lpd, all "r" services (rsh, rlogin, rexec, rcp). Restrict access by firewall from obviously spoofed sources and rate-limit in/out traffic like ICMP. Add active intrusion detection using Snort. It will warn you of incoming exploits and other malicious actions against your system. (don't mistake portsentry for an alternative). If possible, set up remote logging so theyll have to go tru a lot to get to those logs, or have a separate log partition to twart DoSsing by filling up disks.

As for an MTA your choices are exim, postfix or qmail. IMO skipping Sshd isn't good, OpenSSH_2.9p2 is considered safe from aprox Feb last year, while the commercial counterpart had flaws. When configured w/o using Protocol 1 tighten security by using Allow|DenyGroups, and compile with TCP Wrappers. As for ftpd's Wu-ftpd is notorious for its flaws. A more secure ftpd could be Muddleftpd or Vsftpd, or if you only need ppl to download stuff you could check out oftpd. Ive been using Muddleftpd for about 3 yrs now, and there still isnt a root exploit to go with it.

For the MTA and ftpd part, if unsure, run as non-root, allow only anonymous read access to ftpd, and chroot the services. I'm using "rootjail" to help me chroot services and it works well.

For docs search (linuxdoc.org) for/check out:
CERT's Techtips, especially the "UNIX Computer Security Checklist",
CERT, root compromise, part F,
LASG: Linux Administrator's Security Guide,
Security Quick-Start HOWTO for Linux,
"Bastille Linux Hardening System"
"Astaro Linux"
SecFocus UNIX,
Xforce,

The rest of my security reference list is in the second reply here: "possibly a dumb(..)".

Thanks... I was afraid that might be the answer... Looks like I've got a lot of work ahead of me.


Which MTA do you think is best, considering the fact that I will be using Majordomo? Or do you know of a listserv software that is better than Majordomo? Thanks again!

Dear Friend,
For recovery first to find out any rootkit installed on your system just go at http://www.chkrootkit.com download chkrootkit software from there and run chkrootkit this will shows you what deamons and what binarys are replaced by hacker and which rootkit installed by hacker on your system and also do a port scan to find out any malicious port open on your system , off all unnessary services use ipchains to protect your system although fresh installation is very best idea


Thanks and Regards

John Lee

Thanks to everyone who posted replies. I rebuilt from scratch, and I'm running more secure software now. I think I should be closer to OK than I was. I'm going to stay up to date on what's going on. I actually just got DDoS'd today...

By some moron who thought I was running IIS!!!! LOL!

Ah well... too many idiots in the world lately, no?


Thanks again!