Server Compromised?
 

Hello All,

Im using Centos 4 as a linux webserver with LAMP installed. I run yum about once a month and have apf firewall, bfd module installed, root logins not allowed directly, rkhunter and chkrootkit all installed. Basically the full dedicated server list from Crucial Paradigm (http://www.crucialparadigm.com/resou...rver/index.php) !!

With that said, like most that have responded here, I have been getting a lot brute force attempts on my servers from locations in Korea and China. All of a sudden, 6 days ago, one of my servers stopped forwarding me emails from rkhunter, chkrootkit, root login, and brute force attacks. I check the mail log and it says that I dont have a valid email address listed (which doesnt make sense because it was just working and nothing has changed). I dont think the firewall is blocking the port but . . .maybe I accidentally configured something wrong. However, what really is bothering me is that the "last" command only shows records from today and no further history. This server was installed in April and although it has had only a few login attempts (about 20), none of them show. I go to the wtmp log file and it only has about 3 lines there.

Have I been hacked?

If wtmp was recently rotated, then it could very likely be empty without anything malicious happening. What is the last access/modification date on the old rotated /var/log/wtmp.1 file?

Take a look at the config files that are mailing you logs and verify that the mailling address hasn't changed or been altered. Also take a look at your maillogs and see if you can find any messages that explain why you haven't received them. If outgoing mail has somehow been blocked by a firewall misconfiguration, then you should see the failure messages in those logs.

Offtopic, why are you only running yum once a month? You should really be running it nightly, plus having it run automatically via chkconfig/cron is easier.

Is there a way to read the wtmp file? The one that I have is mostly cryptic. I do recognize a few lines because those are the ones that actually show when I run the last command.

As for the mail log, I have noticed the following:

Sep 4 11:11:00 localhost sendmail[4497]: j84GB0cV004497: j84GB0cW004497: DSN: User unknown
Sep 4 11:11:00 localhost sendmail[4498]: j84GB0cg004498: from=<>, size=2164, class=0, nrcpts=1, msgid=<200509041611.j84GB0cW004497@localhost >, proto=ESMTP, daemon=MTA, relay=localhost.localdomain [127.0.0.1]
Sep 4 11:11:00 localhost sendmail[4497]: j84GB0cW004497: to=root, delay=00:00:00, xdelay=00:00:00, mailer=relay, pri=31274, relay=[127.0.0.1] [127.0.0.1], dsn=2.0.0, stat=Sent (j84GB0cg004498 Message accepted for delivery)
Sep 4 11:11:00 localhost sendmail[4499]: j84GB0cg004498: to=<root@localhost >, delay=00:00:00, xdelay=00:00:00, mailer=local, pri=32354, dsn=2.0.0, stat=Sent
Sep 4 11:14:48 localhost sendmail[4553]: j84GEmen004553: from=root, size=250, class=0, nrcpts=1, msgid=<200509041614.j84GEmen004553@localhost>, relay=root@localhost
Sep 4 11:14:48 localhost sendmail[4554]: j84GEmg8004554: <my@emailaddress.com>... User unknown

I have no idea who j84GEmen004553@localhost is. Could this be a hacker?

I run yum manually however, I do see that a yum.conf file is inside my cron.daily folder. is this the file that you are referring to that should run daily?

I have checked with the other servers running the same configuration and everything is the same. The firewall is set to allow port 25 to be used. The failed message that goes to root is returning a 550 - user unknown error.

I have searched and tried to install an IDS on the server but I cannot find tripwire (the free version at least) and I have heard about problems with EIDE. Is there any others that you would recommend?

Is there a way to read the wtmp file?
You can read old wtmp files using last -f /path/to/wtmpfile . Although I was really asking about the date when the wtmp.1 file was created/modified, as that will be the date when it was last rotated. If it was a recent date, then it would make sense that the new wtmp file would have few/no entries.

I have no idea who j84GEmen004553@localhost is. Could this be a hacker?
No, that is just the queue ID that corresponds to the message. That just indicates it's a local mail process (i.e a message being sent locally.

I run yum manually however, I do see that a yum.conf file is inside my cron.daily folder. is this the file that you are referring to that should run daily?
To check if yum is running nightly, do:
chkconfig --list | grep yum

The failed message that goes to root is returning a 550 - user unknown error.
Any reason why that might be happening; does that user exist on the system? What if you try manually sending mail to that user?

I have searched and tried to install an IDS on the server but I cannot find tripwire (the free version at least) and I have heard about problems with EIDE. Is there any others that you would recommend?
Normally I would recommend tripwire, aide or samhain. However, a file integrity IDS is only usefull if you install it before a compromise (preferably immediately after installing the OS and applying updates). If a compromise has occured, then it would be effectively useless.

The wtmp file shows that it was rotated in early September. However, the file wtmp.1 was showing a date of August 4th as the last date. The server has been in operation for several months so this file isnt accurate.

chkconfig --list | grep yum
For this, everything says off. One reason I dont run yum nightly is because I figured that packages would be too new and could cause vulnerabilities. If I want to turn this feature on, if its a good idea, then how can I turn it on?

Failed Username
The email account is on a separate server. Actually, the MX record is pointing to a different server for mail BUT i just remembered that the domain was recently switched to run off of that server. But in my DNS record, the all email accounts are pointing to a different email server. With that said, all other systems pointing to the same email account work correctly.

Aide or Samhain
Is Samhain a better choice than Aide? I realize that a IDS is a good thing to have but didnt know if it was worth the trouble. However, considering that this scare has come about, it may not be such a bad option.

Thanks again.

The wtmp file shows that it was rotated in early September.
If it was rotated recently, then that might explain why wtmp is reletively empty. Chkrootkit should also flag any gross wtmp deletions.

One reason I dont run yum nightly is because I figured that packages would be too new and could cause vulnerabilities.
Pretty doubtfull. The vast majority of packages being upgraded are due to security vulns and I've only seen new packages introduce vulns a handfull of times. So by not upgrading you're allowing your system to have apps with known vulnerabilities for extended periods of time, which isn't good.

If I want to turn this feature on, if its a good idea, then how can I turn it on?
chkconfig yum on

Failed Username
The email account is on a separate server. Actually, the MX record is pointing to a different server for mail BUT i just remembered that the domain was recently switched to run off of that server. But in my DNS record, the all email accounts are pointing to a different email server. With that said, all other systems pointing to the same email account work correctly.
Ok, so should that explain why the mail wasn't being accepted? If you just use: mail that_username@that_mailserver does that work?

Aide or Samhain
Is Samhain a better choice than Aide? I realize that a IDS is a good thing to have but didnt know if it was worth the trouble.
Samhain has some additional features make it more of a true IDS rather than a file integrity scanner like tripwire of aide. So it really depends on whether you want those features or whether you want a leaner file integrity scanner.

Mail Situation
Any mail going to that user has been stopped. The server will not send out any email to that person at all. I have created an account on the system for that username and will check today to see if any of the files are being sent locally.

Samhain
I have installed samhain on the server because based upon what you have mentioned, there may not be a security breach.

History File
Is this a common place for hackers to change information?