Hi,
I just finished installing my server again after it had been compromised for the 2nd time in 2 years. I went to login to root this morning and found that it had been compromised again earlier this morning, passwords changed and all. I ran a port scan late yesterday afternoon and found only http 80, https 443, ssh 22, ftp 21 and vnc ports to be open, all the rest being "filtered". Can anyone tell me what the heck is going on. By the way, my os is redhat 9.

-Derek

I just finished installing my server again after it had been compromised for the 2nd time in 2 years.
Sad to hear that. Do you know the point of breach?


I went to login to root this morning and found that it had been compromised again earlier this morning, passwords changed and all. I ran a port scan late yesterday afternoon and found only http 80, https 443, ssh 22, ftp 21 and vnc ports to be open, all the rest being "filtered". Can anyone tell me what the heck is going on.
Unfortunately a portscan usually won't tell much. If the box is local then you can login and check (save) process, network and user activity listings and then run "rpm -Va --noscripts" and Chkrootkit or Rootkit Hunter (if installed) or boot a Live CD (like KNOPPIX, Helix) that includes those tools.
Before you start check out
Intruder Detection Checklist (CERT): http://www.cert.org/tech_tips/intrud...checklist.html
Steps for Recovering from a UNIX or NT System Compromise: http://www.cert.org/tech_tips/root_compromise.html
and the LQ FAQ: Security references for more info.


By the way, my os is redhat 9.
RHL9 is an EOL'ed, deprecated release. If you need it due to legacy applications, be sure to 0) update tru Fedora Legacy while you still can, 1) max out on host and network hardening and 2) prepare migration RSN to CentOS or Fedora Core or Red Hat if you want require RH or alike.

The point of breach was a root attempt on sshd. I was surprised as I had made new passwords since the last compromise last week and the passwords are not simple (either initials, birthdates, and other info that would not be available in a dictionary).

Also I don't see how it could be a root kit as I did a full system install yesterday (disk format and all).

The point of breach was a root attempt on sshd.
How do you know? Logs? Shell history? Did you allow root account access through ssh? What else could it possibly be? What does your httpd provide? Any PHP-based apps? Updated? Do you run a vulnerable version of RealVNC?


I was surprised as I had made new passwords since the last compromise last week and the passwords are not simple (either initials, birthdates, and other info that would not be available in a dictionary).
If you get in with root privileges all of that doesn't matter. Besides all the docs say not to use birthdates and stuff like that.


Also I don't see how it could be a root kit as I did a full system install yesterday (disk format and all).
If you run publicly accessable services w/o proper precautions then a fresh install ain't worth that much.

I suggest you read the docs and perform those steps anyway.

sounds like some hackers have "found" you, and found you to be an easy target for some reason. my guess is that you'll continue to be compromised until you really harden your system more/differently than you have been doing.

changing your passwords to *very* difficult would be a good start (no initials, names, pet names, birthdays, etc.; make it more than 8 characters, include caps, lower case, numbers, and symbols). in my limited experience i've also found that getting daily reports from things like snort, tiger, logwatch (really like that one), etc., are helpful in identifying what's going on on the server, not to mention running chkrootkit and rkhunter daily, as well as checking the /var/logs.

I just have to ask why root was permitted SSH access? If you absolutely must allow root to have remote ssh login access, you'd be better off using public/private key authentication to prevent hackers from taking stabs at your password.

If root doesn't need ssh access, open your sshd_config file and set the following two parameters:

PermitRootLogin no
AllowsUsers anyone, but, root

If you set the AllowUsers parameter, your system will automatically reject ssh access to all users that are NOT in the list.

Well, I know from when I logged in as root, it told me the last ip address that had logged in and it wasn't one I was familiar with. Like I said, the password had been changed. I have not gotten httpd (Apache 2.0.40) up and running (providing anything) yet, so I don't think anything would have gotten in through that. Again, no PHP-based apps up as I am not offering website service yet. I am running the default version of VNC that comes with Redhat 9, but I always start the service when I need it, and stop the service when I am done using it, so it is never available when I am not in contact with the machine.

I will also try this one, as I can always use su as a normal user and get root access to do things, and if I absolutely must have root access, then I can change the settings for the time being.

That's absolutely right! You could even take it a step further and setup sudo.

OK. Unless there's more, let's wrap this up asking the final question: do you have a clear view of the steps you have to take to set up and harden your box?

Well, I have a little more to ask.

I ran nmap from another computer on the network on my server to come up with my port list. I have ftp, ssh, domain, http, https, cadlock (webmin) turned on for a reason, but don't know what rpcbind, kdm, and X11 are doing open. Can you tell me what those three are and if any of the services open are easy doors/windows for hackers to get into.

(The 1651 ports scanned but not shown below are in state: closed)
PORT STATE SERVICE
21/tcp open ftp
22/tcp open ssh
53/tcp open domain
80/tcp open http
111/tcp open rpcbind
443/tcp open https
1000/tcp open cadlock
1024/tcp open kdm
6000/tcp open X11
MAC Address: 00:50:04:A5:5E8 (3com)

Also, I ran chkrootkit this afternoon and found nothing to be a problem. I will continue to run this periodically and check logs.

Thanks for the help,
Derek

For the X11 port open, edit the 'startx' script and replace serverargs="" with the following:
That will stop it from listening on port 6000 (and may fix the kdm port as well, which is the GUI login manager).

As the above posters said, root over ssh is definately a no-no. I'd definately look into a hardening guide for ssh (there are tons around). Basically though disallow root login, setup public / private keys, turn down the number of failed logins allowed, possibly change the port number, etc.

Lots of other things to watch out for related to security, like incorrect permissions on important files / folders, excess user accounts that are allowed login, etc.

Definately try to read up as much as you can, since it is such a broad topic it's rather hard to list out all the "steps" to better security.

btw sad to hear about your continued break ins.

Just another small thing,
If you run a rootkit hunter, remember to run a fresh copy from a floppy disk, if you have been compromised, you can't rely on the installed version being "un-compromised".

I would strongly advocate running iptables in a default-deny mode to protect services on your machine. A good setup, IMHO, is as follows:
More services can be added with lines similar to the last one in the above script.

I ran nmap from another computer on the network on my server to come up with my port list.
Be aware that if you are behind a router there will be a difference in results if it doesn't forward ports. Also be aware Nmap uses a static port-to-name mapping (just like the system uses /etc/services). If you want perfect results use the version/bannerscan option or do a local "netstat -nap".


don't know what rpcbind, kdm, and X11 are doing open. Can you tell me what those three are and if any of the services open are easy doors/windows for hackers to get into.
Rpcbind aka portmap. Stop and remove if you don't provide NFS, NIS or other RPC services. If you remove, make sure you remove services like NFS as well. If unsure stop anyway. If you really need to provide NFS the check here(RHL), here(distro-agnostic) and here(Deb) for details on how to restrict access. Kdm/Gdm/Xdm are Display Managers (DM). Check their configs (/etc/X11/?dm/) for allowing network access which they shouldn't. X11/Xorg has been covered by others.


Also, I ran chkrootkit this afternoon and found nothing to be a problem. I will continue to run this periodically and check logs.
You box was compromised. You can not keep running this setup. Check out Steps for Recovering from a UNIX or NT System Compromise: http://www.cert.org/tech_tips/root_compromise.html and Formatting and Reinstalling after a Security Incident (SF): http://www.securityfocus.com/infocus/1692. Then check out the LQ FAQ: Security references wrt more host and network hardening, choose a maintained O.S. release and reformat, repartition and reinstall from scratch. Do not install network services and applications you don't need in the configuration phase (like FTP, HTTP and MySQL, PHP etc, etc). Basically the only thing you need is SSH. If you need graphical access use TightVNC and tunnel over SSH. Make sure you use different and strong passwords as mentioned before, stop (and remove) unnecessary services, block inbound services in the firewall and disable or remove router port forwarding before you connect the box to the 'net for updates.

Now, with all the config stuff and docs mentioned, do you have a clear view of the steps you have to take to reinitialise, configure and harden your box now?