Centos 5.4 Root Password Changed and or System Comprimised
 

I made a attempt to access my Centos 5.4 system this morning and need to modify a file that required su access. When I attempted to "su" I could not get in with the password that I set. I rebooted the server which made no difference. Either I have been comprimised or I had forgotten my password. How can one reset the admin password to a system and where would I begin looking to see if the system was comprimised?

I figured out how to change my root password after you forget it but I need to figure out of this was a comprimise. Where to begin?

If your server was compromised, nothing to ensure that you can make it clean. Do you have any monitoring tools, IDS, ... installed on the server?

No,

I was thinking about installing tripwire and IDS like snort. Would that be the bare essentials?

One starting point is to start looking for unusual programs running/listening:

lsof -Pwn
netstat -pane
ps -axfwwwe


I'd also have a good look at the log files and root's .bash_history file. The CERT Checklist is always a good place to start as well. And as quanta suggested, if you have any monitoring tools running, now would be a good time to bring them into the picture.

Installing these now would be pretty pointless. Neither of those is going to work particularly well if the machine has been compromised.

As far as tripwire alternatives, take a look at Aide or Samhain. You also might want to think about SELinux since you're running CentOS. Other measures like mod_security might be worth a look, but we'd probably need to have a better idea of what this box is used for and what is currently running.

I am running a chkrootkit on the server as I type this message. I appreciate the advice and you are right that once the system has been compromised what is the worth of putting and IDS and tripwire. I am wondering if an update could have done this. This has never happened to me before so I am very suspicious.

I think auditd would be running by default, perhaps if you were compromised by a script it would forget about sanitizing auditd.

I want to suggest another IDS: OSSEC.

also " su " VS. " su -"????
is the system $PATH 100% the same for the normal user as for root ?
most of the time they are NOT the same -- for security reasons
/sbin & /usr/sbin and NOT normally in the normal user's $PATH

as to the system cracked
did you leave the SEinux default setting set to "enforcing " ? or set it to "permissive" or OFF
SE is not 100% but it will stop 95%+

I looked at the /var/log/secure and I see a gap in entry log that is unusual:

and as you can see there is no data for the tenth! Take a look at the line before that:

I dont like what I am seeing and also auditd is running. Let me dig deeper.

when looking at the audit.log how can one tell the date and time. I dont specifically see anything that specifies that. Also SEliux is not running. I also ran

and

here is the root directory on the system. Do these files and directories look normal?

After thinking about it, why would a smart hacker change the root password and instantly give him or her away? It doesnt make sense.

here is the output of my firewall on the box:

Is this acceptable?

I'm assuming the innocent explanations for that have been ruled out (i.e. someone turned off the computer). If so, then yeah, this is starting to stink.

I agree it doesn't make sense, but there are a couple of explanations. First, maybe this isn't a smart cracker and second, maybe they put themselves into a situation where they needed to change it if they wanted to continue root access. Depending on the machine, it might be some time before someone discovers that the password has changed and depending on the situation, it may take longer to determine it was changed by someone unauthorized. What this does highlight though is the need to spend some quality time developing some facts on the machine.

You would be the better judge of that. Are there things that look out of place? One thing that does strike me is that none of those files have been modified since the 10th. However, if someone is installing stuff, it is probably more likely that they put it someone much less obvious than in /root.

I think at this point it would be useful to do a few things. First, look to see if there is anything unusual running on the box. The ps, lsof and netstat commands I posted earlier would be a good place to start. If there are unusual services, that is not a good sign. If the normal complement of stuff is there, it would be good to verify that the binaries are what is expected using rpm -Vv. Second, since you do have a date, I would examine the machine for files that have been altered or added since the 10th. Since the root password changed, I would also have a good look at root's .bash_history and see if anything jumps out as bizarre. If they have managed to alter the logs, they have probably also altered .bash_history, but it probably doesn't hurt to take a look. Look at your last output and see if any new users crop up or if old users are logging in at unusual times (particularly root).

By the way, feel free to email me any output too big to post here.

Many thanks for all the responses. How is my firewall looking?