LinuxQuestions.org
View the Most Wanted LQ Wiki articles.
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - Security
User Name
Password
Linux - Security This forum is for all security related questions.
Questions, tips, system compromises, firewalls, etc. are all included here.

Notices

Reply
 
Search this Thread
Old 06-19-2006, 05:56 PM   #1
BogusTrumper
Member
 
Registered: May 2006
Posts: 77

Rep: Reputation: 15
Unhappy I got hacked; what do I do?


OK, so my office has a pretty beefy server running Fedora Core 4. And, as we've undergone some staff changes recently, I'm the one in charge of it. I'm still a bit green behind the ears, but I'm getting there. Anyway, just this morning, I checked out the logwatch report, and there's some interesting stuff. Around the same time, I got a message from the admin who oversees the building's network that we've probably been hacked. Here's the extremely suspicious portion of logwatch:

--------------------- Cron Begin ------------------------

Commands Run:
User <hiding this on purpose>:
/home/<hiding this on purpose>/.../cron.sh > /dev/null 2>&1: 445 Time(s)
personal crontab edited: 1 Time(s)
personal crontab listed: 2 Time(s)
personal crontab replaced: 1 Time(s)
User root:
run-parts /etc/cron.daily: 1 Time(s)
run-parts /etc/cron.hourly: 24 Time(s)
run-parts /etc/cron.weekly: 1 Time(s)

---------------------- Cron End -------------------------

Up until this point, there have only been root jobs running as cron. The other odd thing is that there are no sshd authentication failures for this log. Normally there are plenty, as the server is on a particularly well-sniffed subnet; so I'm guessing whoever got in has managed to clear those logs. I'm guessing this has happened today; as yesterday's logs look about normal. Anyway, the system is now down, and I didn't grab a copy of all the logs first (which was probably a mistake, but oh well), so unfortunately that's about the level of detail I can provide right now.

The root password was unchanged, and I was able to shut down the machine OK. Once I figure out/fix what's going on, I'm guessing I'll have to change that anyway just to be sure; but, yeah. Kinda lost on the "figure out/fix" what's going on bit, though.

Any tips on what I should look for? Anybody had a similar hack?
 
Old 06-19-2006, 06:35 PM   #2
win32sux
Guru
 
Registered: Jul 2003
Location: Los Angeles
Distribution: Ubuntu
Posts: 9,870

Rep: Reputation: 371Reputation: 371Reputation: 371Reputation: 371
here's a link that might be useful: http://www.cert.org/tech_tips/win-UN...ompromise.html

just my ...
 
Old 06-19-2006, 06:38 PM   #3
unSpawn
Moderator
 
Registered: May 2001
Posts: 27,260
Blog Entries: 54

Rep: Reputation: 2841Reputation: 2841Reputation: 2841Reputation: 2841Reputation: 2841Reputation: 2841Reputation: 2841Reputation: 2841Reputation: 2841Reputation: 2841Reputation: 2841
First of all welcome to LQ. I'm sorry to see that had to be on such an occasion.

win32sux suggestion of using the "Steps for Recovering from a UNIX or NT System Compromise" is a good one (it's also listed in the LQ FAQ: Security references). Some steps can be automated, but some can't. OK, so you're new here and you've got an (perceived) incident. In short the drill goes somewhat like this:
0. quick discovery of processes, connections and rootkits,
1. curbing risks by sealing off subnet or ultimately "pulling the plug", (good to see the box is down already).
2. making backups for forensics,
3. recovery and hardening.


my office has a pretty beefy server running Fedora Core 4
- Does the company has any policies in place wrt information, network, security?
- Did the box have SELinux enabled?
- Was it updated regularly?
- Was it properly firewalled?
- What services where available (OK, except for SSH)?


I got a message from the admin who oversees the building's network that we've probably been hacked.
- What behaviour, unexpected traffic or otherwise out of the ordinary was observed? Was it substantiated by any logs?


Up until this point, there have only been root jobs running as cron.
- Unless PAM stack for cron is hardened (listfile), or /etc/cron.deny is used or if there's a ruleset forbidding this any user can set up his or her own crontab. In absence of measures taken only the validity of the user account and his / her access restrictions and the actual scripts (and their effect) can be basis for "evidence".


The other odd thing is that there are no sshd authentication failures for this log. Normally there are plenty, as the server is on a particularly well-sniffed subnet; so I'm guessing whoever got in has managed to clear those logs.
Don't guess: make sure. Two ways, but whatever you do make sure no data on disk can be altered. Either mount the drive in another box or bring the box up using a LiveCD like KNOPPIX, FIRE, PSK etc, etc. Make a copy of the partitions with dd (or dd_rescue) to another disk or tape for forensic purposes. Now mount drives read-only and run a quick scan with Chkrootkit and or Rootkit Hunter.


Once I figure out/fix what's going on, I'm guessing I'll have to change that anyway just to be sure
Prepare yourself for the fact that getting a box cracked isn't simply mitigated by "fixing things". Next to investing time in how it got cracked and what was looked at the box most likely will have to be repartitioned, reformatted and have the OS reinstalled from scratch. More on that later on.

Last edited by unSpawn; 06-19-2006 at 06:40 PM.
 
Old 06-19-2006, 07:22 PM   #4
BogusTrumper
Member
 
Registered: May 2006
Posts: 77

Original Poster
Rep: Reputation: 15
Quote:
Originally Posted by unSpawn
- Does the company has any policies in place wrt information, network, security?
For Linux... nothing major. Still generally a Windows-based environment. Basically they say "be secure, especially if you have personally identifiable info" which we don't.

Quote:
Originally Posted by unSpawn
- Did the box have SELinux enabled?
I believe it did; will check shortly.

Quote:
Originally Posted by unSpawn
- Was it updated regularly?
It was updated by a yum cron.daily and cron.weekly job.

Quote:
Originally Posted by unSpawn
- Was it properly firewalled?
We blocked all ports except 22 on the NIC attached to the world; on the local NIC, it was less restricted.

Quote:
Originally Posted by unSpawn
- What services where available (OK, except for SSH)?
To the world, we had SSH, FreeNX, and VNC (only tunneled through SSH). For the local network, it was connected to a single Orion cluster server and had NFS enabled (but only to the master Orion node).

Quote:
Originally Posted by unSpawn
- What behaviour, unexpected traffic or otherwise out of the ordinary was observed? Was it substantiated by any logs?
Here's a copy of the logs I got:

Jun 18 03:54:51 snort[19012]: [1:0:0] code-blue-psyBNC-irc-server {TCP} <my_ip>:42088 -> 64.233.163.27:25
Jun 18 16:42:45 snort[19012]: [1:1000421:1] code-blue-Free_disk_space-SNS#27643 {TCP} <my_ip>:5900 -> 85.138.31.81:64959
Jun 18 20:02:34 snort[19012]: [1:1000421:1] code-blue-Free_disk_space-SNS#27643 {TCP} <my_ip>:5900 -> 86.52.122.216:4060
Jun 18 21:17:20 snort[19012]: [1:1000421:1] code-blue-Free_disk_space-SNS#27643 {TCP} <my_ip>:5900 -> 86.52.122.216:4783

The other IPs don't trace onsite.

Quote:
Originally Posted by unSpawn
- Unless PAM stack for cron is hardened (listfile), or /etc/cron.deny is used or if there's a ruleset forbidding this any user can set up his or her own crontab. In absence of measures taken only the validity of the user account and his / her access restrictions and the actual scripts (and their effect) can be basis for "evidence".
OK, thanks for the tip. I'll dig into the script and see what it starts.

Quote:
Originally Posted by unSpawn
Don't guess: make sure. Two ways, but whatever you do make sure no data on disk can be altered. Either mount the drive in another box or bring the box up using a LiveCD like KNOPPIX, FIRE, PSK etc, etc. Make a copy of the partitions with dd (or dd_rescue) to another disk or tape for forensic purposes. Now mount drives read-only and run a quick scan with Chkrootkit and or Rootkit Hunter.
I'll do a read-only mount; but using dd is probably not going to work, as the home directories are all mounted on a 1.5 TB RAID.

Quote:
Originally Posted by unSpawn
Prepare yourself for the fact that getting a box cracked isn't simply mitigated by "fixing things". Next to investing time in how it got cracked and what was looked at the box most likely will have to be repartitioned, reformatted and have the OS reinstalled from scratch. More on that later on.
Yeah, I know... I just wasn't ready to pee in my cheerios just yet.
 
Old 06-19-2006, 08:20 PM   #5
unSpawn
Moderator
 
Registered: May 2001
Posts: 27,260
Blog Entries: 54

Rep: Reputation: 2841Reputation: 2841Reputation: 2841Reputation: 2841Reputation: 2841Reputation: 2841Reputation: 2841Reputation: 2841Reputation: 2841Reputation: 2841Reputation: 2841
Here's a copy of the logs I got:
Never seen that ruleset or its names. Where's this from?


I'll dig into the script and see what it starts.
Well, here's a good argument for quick investigation of a live box, you for instance can dump parts of memory and find deleted files that are still used (lsof). Of course you must be able to trust your toolkit or have access to an external one, but the best argument against a quick peek would be it disturbs data on the box.


using dd is probably not going to work, as the home directories are all mounted on a 1.5 TB RAID.
That's kinda prohibitive. If necessary you could carve part of the data out based on MAC times, but let's have a look first at upcoming results of log files, auth data, Chkrootkit and Rootkit Hunter.

Last edited by unSpawn; 06-19-2006 at 08:21 PM.
 
Old 06-20-2006, 04:58 PM   #6
BogusTrumper
Member
 
Registered: May 2006
Posts: 77

Original Poster
Rep: Reputation: 15
Thanks for the help! The panic is now subsiding. From what I've gathered, I believe someone got ahold of a user's password, set up an ftp site running in /home/user/.../, then set up a cronjob to redirect the secure log output to /dev/null every minute. They neglected to clear .bash_history for the user, though. Looks like they tried once to login as root once, and either did a very surgical job of hiding what they did from root's .bash_history (yet forgot to do that for the user, letting me know exactly where they downloaded the ftp server) or else they didn't succeed. By the time I yanked the network card, approximately 9 gigs of porn and pirated video games had ended up there.

I'm still a bit concerned about how they got the user's password. It was a non-obvious username, but that particular uname/password combination had been used quite a bit by this individual, so there's always the possibility that a network sniffer or keylogger was set up on their home computer (which is being checked). I'm planning on wiping the OS partition and reinstalling just to be sure I've closed all the holes; but I'm really hoping my /home partition will be mostly OK. There isn't anything in the logs about doing anything aside from looking up the ip, downloading the ftp server, and starting it. But there is that chunk missing... right now I'm grepping through the logs and looking for everything this particular user ever did, and looking around the time that the file was downloaded for any activity anywhere.
 
Old 06-21-2006, 06:37 AM   #7
unSpawn
Moderator
 
Registered: May 2001
Posts: 27,260
Blog Entries: 54

Rep: Reputation: 2841Reputation: 2841Reputation: 2841Reputation: 2841Reputation: 2841Reputation: 2841Reputation: 2841Reputation: 2841Reputation: 2841Reputation: 2841Reputation: 2841
The panic is now subsiding.
Good to hear. Nice job you did reconstructing the timeline. Did the FTPd run on a port below 1024? Have all the logs been redirected to /dev/null? Can you see if there's been changes to wtmp and btmp? (Chkrootkit should show, else run last yourself). Any other files MAC time changed in the period of (suspected) root login?


I'm still a bit concerned about how they got the user's password. It was a non-obvious username, but that particular uname/password combination had been used quite a bit by this individual
It also could be that person using the login/pass combo for other purposes, or traded it in for something else, or was a victim of social engineering. Just thought I'd mention it.


I'm planning on wiping the OS partition and reinstalling just to be sure I've closed all the holes; but I'm really hoping my /home partition will be mostly OK.
Before you nuke the box it would be good to sit back and look at the whole situation again, go over the checklist and see if there's anything left to check. And even if there is no evidence of springboarding it would be good to make sure adjacent boxen on the network are untouched. Mind you, even if nothing points to forced entry, you will not have closed all holes by just re-installing. This really should be an incentive to discuss a "better" password policy (amongst other things): all passes will need to be changed anyway. I'm wondering what system hardening measures you are going to apply... Wrt /home (BTW, 1.5 TB, is that as in huge amount of accounts, or just accumulated stuff like mailboxen and docs?) I would comb it over for scripts and binaries, (plaintext) password/shadow files, virusscan it and make sure it's remounted with appropriate flags. When the O.S. is reinstalled you will need to invest time setting up hardening measures, auditing caps and probably remote logging, but let's handle that after you post a list of what you think is needed wrt hardening.

BTW, did you run Chkrootkit / Rootkit Hunter on the read-only disks?
 
  


Reply


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off


Similar Threads
Thread Thread Starter Forum Replies Last Post
Have i been hacked ? Fcm Linux - Security 2 06-21-2006 11:16 AM
Hacked !!?? rusty105 Linux - Security 30 06-15-2006 08:03 AM
I'm probably being HACKED, help !! jmr0311 Linux - Security 2 08-26-2004 04:16 PM
Have I been hacked? Take a look at this... Ausar Linux - Security 7 08-25-2004 03:37 PM
help! i've been hacked geo_serban Linux - Newbie 3 07-29-2004 07:24 AM


All times are GMT -5. The time now is 10:43 AM.

Main Menu
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
identi.ca: @linuxquestions
Facebook: linuxquestions Google+: linuxquestions
Open Source Consulting | Domain Registration