OK, so my office has a pretty beefy server running Fedora Core 4. And, as we've undergone some staff changes recently, I'm the one in charge of it. I'm still a bit green behind the ears, but I'm getting there. Anyway, just this morning, I checked out the logwatch report, and there's some interesting stuff. Around the same time, I got a message from the admin who oversees the building's network that we've probably been hacked. Here's the extremely suspicious portion of logwatch:

--------------------- Cron Begin ------------------------

Commands Run:
User <hiding this on purpose>:
/home/<hiding this on purpose>/.../cron.sh > /dev/null 2>&1: 445 Time(s)
personal crontab edited: 1 Time(s)
personal crontab listed: 2 Time(s)
personal crontab replaced: 1 Time(s)
User root:
run-parts /etc/cron.daily: 1 Time(s)
run-parts /etc/cron.hourly: 24 Time(s)
run-parts /etc/cron.weekly: 1 Time(s)

---------------------- Cron End -------------------------

Up until this point, there have only been root jobs running as cron. The other odd thing is that there are no sshd authentication failures for this log. Normally there are plenty, as the server is on a particularly well-sniffed subnet; so I'm guessing whoever got in has managed to clear those logs. I'm guessing this has happened today; as yesterday's logs look about normal. Anyway, the system is now down, and I didn't grab a copy of all the logs first (which was probably a mistake, but oh well), so unfortunately that's about the level of detail I can provide right now.

The root password was unchanged, and I was able to shut down the machine OK. Once I figure out/fix what's going on, I'm guessing I'll have to change that anyway just to be sure; but, yeah. Kinda lost on the "figure out/fix" what's going on bit, though.

Any tips on what I should look for? Anybody had a similar hack?

here's a link that might be useful: http://www.cert.org/tech_tips/win-UN...ompromise.html

just my ...

First of all welcome to LQ. I'm sorry to see that had to be on such an occasion.

win32sux suggestion of using the "Steps for Recovering from a UNIX or NT System Compromise" is a good one (it's also listed in the LQ FAQ: Security references). Some steps can be automated, but some can't. OK, so you're new here and you've got an (perceived) incident. In short the drill goes somewhat like this:
0. quick discovery of processes, connections and rootkits,
1. curbing risks by sealing off subnet or ultimately "pulling the plug", (good to see the box is down already).
2. making backups for forensics,
3. recovery and hardening.


my office has a pretty beefy server running Fedora Core 4
- Does the company has any policies in place wrt information, network, security?
- Did the box have SELinux enabled?
- Was it updated regularly?
- Was it properly firewalled?
- What services where available (OK, except for SSH)?


I got a message from the admin who oversees the building's network that we've probably been hacked.
- What behaviour, unexpected traffic or otherwise out of the ordinary was observed? Was it substantiated by any logs?


Up until this point, there have only been root jobs running as cron.
- Unless PAM stack for cron is hardened (listfile), or /etc/cron.deny is used or if there's a ruleset forbidding this any user can set up his or her own crontab. In absence of measures taken only the validity of the user account and his / her access restrictions and the actual scripts (and their effect) can be basis for "evidence".


The other odd thing is that there are no sshd authentication failures for this log. Normally there are plenty, as the server is on a particularly well-sniffed subnet; so I'm guessing whoever got in has managed to clear those logs.
Don't guess: make sure. Two ways, but whatever you do make sure no data on disk can be altered. Either mount the drive in another box or bring the box up using a LiveCD like KNOPPIX, FIRE, PSK etc, etc. Make a copy of the partitions with dd (or dd_rescue) to another disk or tape for forensic purposes. Now mount drives read-only and run a quick scan with Chkrootkit and or Rootkit Hunter.


Once I figure out/fix what's going on, I'm guessing I'll have to change that anyway just to be sure
Prepare yourself for the fact that getting a box cracked isn't simply mitigated by "fixing things". Next to investing time in how it got cracked and what was looked at the box most likely will have to be repartitioned, reformatted and have the OS reinstalled from scratch. More on that later on.

For Linux... nothing major. Still generally a Windows-based environment. Basically they say "be secure, especially if you have personally identifiable info" which we don't.

I believe it did; will check shortly.

It was updated by a yum cron.daily and cron.weekly job.

We blocked all ports except 22 on the NIC attached to the world; on the local NIC, it was less restricted.

To the world, we had SSH, FreeNX, and VNC (only tunneled through SSH). For the local network, it was connected to a single Orion cluster server and had NFS enabled (but only to the master Orion node).

Here's a copy of the logs I got:

Jun 18 03:54:51 snort[19012]: [1:0:0] code-blue-psyBNC-irc-server {TCP} <my_ip>:42088 -> 64.233.163.27:25
Jun 18 16:42:45 snort[19012]: [1:1000421:1] code-blue-Free_disk_space-SNS#27643 {TCP} <my_ip>:5900 -> 85.138.31.81:64959
Jun 18 20:02:34 snort[19012]: [1:1000421:1] code-blue-Free_disk_space-SNS#27643 {TCP} <my_ip>:5900 -> 86.52.122.216:4060
Jun 18 21:17:20 snort[19012]: [1:1000421:1] code-blue-Free_disk_space-SNS#27643 {TCP} <my_ip>:5900 -> 86.52.122.216:4783

The other IPs don't trace onsite.

OK, thanks for the tip. I'll dig into the script and see what it starts.

I'll do a read-only mount; but using dd is probably not going to work, as the home directories are all mounted on a 1.5 TB RAID.

Yeah, I know... I just wasn't ready to pee in my cheerios just yet.

Here's a copy of the logs I got:
Never seen that ruleset or its names. Where's this from?


I'll dig into the script and see what it starts.
Well, here's a good argument for quick investigation of a live box, you for instance can dump parts of memory and find deleted files that are still used (lsof). Of course you must be able to trust your toolkit or have access to an external one, but the best argument against a quick peek would be it disturbs data on the box.


using dd is probably not going to work, as the home directories are all mounted on a 1.5 TB RAID.
That's kinda prohibitive. If necessary you could carve part of the data out based on MAC times, but let's have a look first at upcoming results of log files, auth data, Chkrootkit and Rootkit Hunter.

Thanks for the help! The panic is now subsiding. From what I've gathered, I believe someone got ahold of a user's password, set up an ftp site running in /home/user/.../, then set up a cronjob to redirect the secure log output to /dev/null every minute. They neglected to clear .bash_history for the user, though. Looks like they tried once to login as root once, and either did a very surgical job of hiding what they did from root's .bash_history (yet forgot to do that for the user, letting me know exactly where they downloaded the ftp server) or else they didn't succeed. By the time I yanked the network card, approximately 9 gigs of porn and pirated video games had ended up there.

I'm still a bit concerned about how they got the user's password. It was a non-obvious username, but that particular uname/password combination had been used quite a bit by this individual, so there's always the possibility that a network sniffer or keylogger was set up on their home computer (which is being checked). I'm planning on wiping the OS partition and reinstalling just to be sure I've closed all the holes; but I'm really hoping my /home partition will be mostly OK. There isn't anything in the logs about doing anything aside from looking up the ip, downloading the ftp server, and starting it. But there is that chunk missing... right now I'm grepping through the logs and looking for everything this particular user ever did, and looking around the time that the file was downloaded for any activity anywhere.

The panic is now subsiding.
Good to hear. Nice job you did reconstructing the timeline. Did the FTPd run on a port below 1024? Have all the logs been redirected to /dev/null? Can you see if there's been changes to wtmp and btmp? (Chkrootkit should show, else run last yourself). Any other files MAC time changed in the period of (suspected) root login?


I'm still a bit concerned about how they got the user's password. It was a non-obvious username, but that particular uname/password combination had been used quite a bit by this individual
It also could be that person using the login/pass combo for other purposes, or traded it in for something else, or was a victim of social engineering. Just thought I'd mention it.


I'm planning on wiping the OS partition and reinstalling just to be sure I've closed all the holes; but I'm really hoping my /home partition will be mostly OK.
Before you nuke the box it would be good to sit back and look at the whole situation again, go over the checklist and see if there's anything left to check. And even if there is no evidence of springboarding it would be good to make sure adjacent boxen on the network are untouched. Mind you, even if nothing points to forced entry, you will not have closed all holes by just re-installing. This really should be an incentive to discuss a "better" password policy (amongst other things): all passes will need to be changed anyway. I'm wondering what system hardening measures you are going to apply... Wrt /home (BTW, 1.5 TB, is that as in huge amount of accounts, or just accumulated stuff like mailboxen and docs?) I would comb it over for scripts and binaries, (plaintext) password/shadow files, virusscan it and make sure it's remounted with appropriate flags. When the O.S. is reinstalled you will need to invest time setting up hardening measures, auditing caps and probably remote logging, but let's handle that after you post a list of what you think is needed wrt hardening.

BTW, did you run Chkrootkit / Rootkit Hunter on the read-only disks?