LinuxQuestions.org
Go Job Hunting at the LQ Job Marketplace
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - Security
User Name
Password
Linux - Security This forum is for all security related questions.
Questions, tips, system compromises, firewalls, etc. are all included here.

Notices

Reply
 
Search this Thread
Old 04-12-2010, 09:58 AM   #1
Kevin Joubert
LQ Newbie
 
Registered: Apr 2010
Location: Johannesburg, South Africa
Distribution: Redhat & Centos
Posts: 3

Rep: Reputation: 0
Tracing unauthorised (or incompetent) user activity


Our company has recently experienced a spate of attacks at some of our customers that have resulted in the root file system being trashed. Although all our servers run a raid controller in mirror mode that allows us to get the customers back up again quite quickly, I am suspicious about the nature of these attacks. Our root passwords are quite strong yet they always get cracked within about 1200 attempts. This makes me think its an inside job that is being disguised to look like an external "brute force" attack. Also the way that the system is crashed, leaving the whole root file system corrupted, is very unsubtle and it occurs to me that this may be done to hide an industrial espionage attack.

Is there any way we can log the activities of everyone who log's onto one of our servers as root user so that we can see what they did before the system crashed. Our support staff very rarely need root access so it won't result in a lot of data to wade through. Also, we have about 100 customers so if we have to purchase a tool to do this it can't be too expensive.

I am using this forum because I'm not sure if the attacker is one of our own support staff, so any advice on how to track the person down would also be greatly appreciated.

Many thanks, Kevin Joubert.
 
Old 04-12-2010, 11:40 AM   #2
TB0ne
Guru
 
Registered: Jul 2003
Location: Birmingham, Alabama
Distribution: SuSE, RedHat, Slack,CentOS
Posts: 14,419

Rep: Reputation: 2532Reputation: 2532Reputation: 2532Reputation: 2532Reputation: 2532Reputation: 2532Reputation: 2532Reputation: 2532Reputation: 2532Reputation: 2532Reputation: 2532
Quote:
Originally Posted by Kevin Joubert View Post
Our company has recently experienced a spate of attacks at some of our customers that have resulted in the root file system being trashed. Although all our servers run a raid controller in mirror mode that allows us to get the customers back up again quite quickly, I am suspicious about the nature of these attacks. Our root passwords are quite strong yet they always get cracked within about 1200 attempts. This makes me think its an inside job that is being disguised to look like an external "brute force" attack. Also the way that the system is crashed, leaving the whole root file system corrupted, is very unsubtle and it occurs to me that this may be done to hide an industrial espionage attack.

Is there any way we can log the activities of everyone who log's onto one of our servers as root user so that we can see what they did before the system crashed. Our support staff very rarely need root access so it won't result in a lot of data to wade through. Also, we have about 100 customers so if we have to purchase a tool to do this it can't be too expensive.

I am using this forum because I'm not sure if the attacker is one of our own support staff, so any advice on how to track the person down would also be greatly appreciated.

Many thanks, Kevin Joubert.
You can look at everyone's .bash_history files, to see what they were doing. Mirror your syslogs to another server, so even if your system log files are edited/deleted, all the events are still getting 'teed' to a remote system, where they DON'T have access. Look at the output from the last command, to see who logged in from where.

Bear in mind that ANY of this can be circumvented by someone who knows what they're doing. In the event of stupidity being the root cause, that's easily fixed.
 
Old 04-12-2010, 11:55 AM   #3
devnull10
Member
 
Registered: Jan 2010
Location: Lancashire
Distribution: Slackware Stable
Posts: 547

Rep: Reputation: 115Reputation: 115
Surely anyone with half a brain would remove the .bash_history file if they were trying to trash something!?! In any case, it's the root password being cracked, so all you would see is the history in roots log. And there is no way to stop root from removing anything!
Perhaps you can restrict who knows the root password to an absolute minimum set of people and have them all use sudo for everything?
 
Old 04-12-2010, 12:05 PM   #4
TB0ne
Guru
 
Registered: Jul 2003
Location: Birmingham, Alabama
Distribution: SuSE, RedHat, Slack,CentOS
Posts: 14,419

Rep: Reputation: 2532Reputation: 2532Reputation: 2532Reputation: 2532Reputation: 2532Reputation: 2532Reputation: 2532Reputation: 2532Reputation: 2532Reputation: 2532Reputation: 2532
Quote:
Originally Posted by devnull10 View Post
Surely anyone with half a brain would remove the .bash_history file if they were trying to trash something!?! In any case, it's the root password being cracked, so all you would see is the history in roots log. And there is no way to stop root from removing anything!
Perhaps you can restrict who knows the root password to an absolute minimum set of people and have them all use sudo for everything?
Yep...that's why I said it's easily circumvented in my original post.

SUDO will work, but only if you're very careful to limit the commands it can run, to be absolutely safe, and especially don't grant SUDO shell access, since it then will only log that userx ran a bash shell at time X...but won't log what they DID after that. Got to restrict vi, chmod, chown, mv, cp, and rm too. If they have those commands via sudo, they can then cp the sudoers file to their home directory, chmod/chown, then edit it, and move it back to the 'real' location, modify the attributes, and they're back in. Easier to designate a VERY small subset of commands that SUDO can run, and exclude everything else.
 
Old 04-12-2010, 12:05 PM   #5
unSpawn
Moderator
 
Registered: May 2001
Posts: 27,118
Blog Entries: 54

Rep: Reputation: 2787Reputation: 2787Reputation: 2787Reputation: 2787Reputation: 2787Reputation: 2787Reputation: 2787Reputation: 2787Reputation: 2787Reputation: 2787Reputation: 2787
A slightly more elaborate reply ;-p First of all I'm having a few reservations since no hard data is made available with respect to setup, state of security and auditing, types of attacks, co-occurrence, timeline et cetera. (But then again I'll prolly have to contain myself since an investigation is not what you ask for.) Secondly, but probably more important, you'll have to have a plan and an explanation in place before you execute your plan. As staff has admin rights they will notice (or be tipped off by) any changes in server configuration, software or infrastructure. As a third point I strongly suggest you consider consulting with legal council and professional investigative services with respect to Privacy Laws, evidence gathering and possible submission and other issues that may arise from whatever is proposed. (In some cases it might be required to have personnel involved in the investigation to sign an NDA and in some jurisdictions one is required to publicize information about an ongoing investigation. IANAL and I don't know your local Law, so please consider it.) Also since thrashing a fs repeatedly isn't your average cracker modus operandi you'll have to make certain you can rule out hardware errors.


I've edited your OP to filter the facts you posted:
Quote:
Originally Posted by Kevin Joubert View Post
- (..) we have about 100 customers
- Our company has recently experienced a spate of attacks at some of our customers
- (..) resulted in the root file system being trashed.
- the way that the system is crashed, leaving the whole root file system corrupted, (..)
- Our root passwords (..) always get cracked within about 1200 attempts.
- (..) support staff very rarely need root access
- I'm not sure if the attacker is one of our own support staff
Before you start drawing a plan you should take stock of how staff logs in or performs tasks like physical logins, using any "web panels", network services (SSH, telnet, et cetera), through terminal servers. These need to be evaluated for continued use (telnet?) and if used their logging needs to be not too terse. Less methods available means a more narrow auditing scope. Review login methods (root directly, wheel group access, via an intermediate unprivileged user account). Make sure no network root access can occur other than by first logging in through an intermediate unprivileged user account. Each member with root access rights must have a separate account. (Physical logins I leave up to you. If you've got a quality colo then proper and accurate access records are kept, right?) Change all root passwords to be unique per server. Distribute them on a need to know basis. Set up each intermediate to run 'rootsh' (Bash wrapper). Enable SE Linux and Auditd where possible (or any other MAC you use). Set up a syslog server with Syslog-NG (that only a select group has admin access to, preferably you and one trustworthy admin, and preferably over another 'net). Make all machines duplicate syslog there. Deny su transitions and force all intermediates to transition to root only using 'sudo rootsh -i -u root'. Require all staff to keep a log of which servers they logged into. Corellate logs against syslog server entries. I probably did miss or didn't address something (like for instance verifying all automated and scheduled tasks) but hey, it's a start...

Last edited by unSpawn; 04-12-2010 at 12:07 PM.
 
Old 04-12-2010, 12:28 PM   #6
devnull10
Member
 
Registered: Jan 2010
Location: Lancashire
Distribution: Slackware Stable
Posts: 547

Rep: Reputation: 115Reputation: 115
Another thing, whilst not foolproof and a bit messy, would be to set up roots .bash_login script to send an email to an address such as unixsecurity@yourdomain.com which your security admins have access to. Then as soon as someone logs in as root you get an email sent to you with the details. If you pipe the "who" command to it, you may get some additional information such as the address of the PC they are logged in from.
So in your .bash_login for root, have something like:


Code:
who | grep root | mail -s "*** SYSTEM ALERT *** - USER HAS LOGGED IN AS ROOT! - $(date)" unixsecurity@mydomain.com
Then say if anyone intends at all to log in using root they MUST email the security group first to let them know - fingers get lost, people get tar and feathered etc for failing to do so!! This is similar a setup to what we use in our org for root access to the unix servers.


It doesn't stop the attack, but it at least gives you a little bit of an audit trail, somewhere to start from! And if it happens during the day you have as chance of finding out.

In any case, root access should be strictly limited. As said above, use sudo and tie it down!

Last edited by devnull10; 04-18-2010 at 02:51 PM.
 
Old 04-12-2010, 03:39 PM   #7
anomie
Senior Member
 
Registered: Nov 2004
Location: Texas
Distribution: RHEL, Scientific Linux, Debian, Fedora, Lubuntu, FreeBSD
Posts: 3,930
Blog Entries: 5

Rep: Reputation: Disabled
Quote:
Originally Posted by Kevin Joubert
Our support staff very rarely need root access...
Time to start limiting attack vectors:
  • Another vote for enabling very limited sudoers (i.e. which can only execute scripts that are properly vetted, and perform specific actions -- no editors or pagers)
  • No one has root's password except you and the boss (or similar, as per your documented security policies)
  • Direct root logins over ssh are disabled
  • Services are reviewed on a case-by-case basis for weak configurations and known exploits
  • The strength of your strong passwords are objectively tested by pam_passwdqc
  • Logs (/var/log/security, etc.) are regularly reviewed for warnings signs
  • Latest security and bugfixes are applied

Last edited by anomie; 04-12-2010 at 03:41 PM.
 
Old 04-12-2010, 03:49 PM   #8
frndrfoe
Member
 
Registered: Jan 2008
Distribution: RHEL, CentOS
Posts: 375

Rep: Reputation: 38
You could setup a remote syslog server and make sure that you are sending all the appropriate logs.

Mount /home directories, /tmp, and /var/tmp with noexec assuming they are partitioned accordingly, in case they are running a rootkit.
 
Old 04-16-2010, 05:26 AM   #9
Kevin Joubert
LQ Newbie
 
Registered: Apr 2010
Location: Johannesburg, South Africa
Distribution: Redhat & Centos
Posts: 3

Original Poster
Rep: Reputation: 0
Thumbs up Thanks for the help

I can't for the life of me find the "Thanks button" that some posts refer to so here's a manual thanks very much to frndrfoe , anomie , devnull10 , unSpawn and TB0ne. I have already issued instructions to implement some of the measures you guys have suggested and I have decided to hire an external guru to help dramatically beef up our security and auditing protocols.

We managed to glean some information from the firewall vendor at one of the sites and it seems that the attacks originated from an ISP in Romania, so, unless there is a way to spoof IP addresses, it looks like my inside job suspicions were unfounded. Apart from that almost useless bit if info I was surprised at how little help, or protection for that matter, the ISP firewalls provide in these cases. It's a far cry from the picture they paint when they're doing their sales pitches!

Thanks again, Kevin Joubert.
 
Old 04-16-2010, 11:16 AM   #10
unSpawn
Moderator
 
Registered: May 2001
Posts: 27,118
Blog Entries: 54

Rep: Reputation: 2787Reputation: 2787Reputation: 2787Reputation: 2787Reputation: 2787Reputation: 2787Reputation: 2787Reputation: 2787Reputation: 2787Reputation: 2787Reputation: 2787
Quote:
Originally Posted by Kevin Joubert View Post
I have decided to hire an external guru to help dramatically beef up our security and auditing protocols.
Good one.


Quote:
Originally Posted by Kevin Joubert View Post
I have already issued instructions to implement some of the measures you guys have suggested
Can you be more specific?
 
Old 04-16-2010, 03:32 PM   #11
Jim Bengtson
Member
 
Registered: Feb 2009
Location: Iowa
Distribution: Ubuntu 9.10
Posts: 164

Rep: Reputation: 38
Unhappy

Quote:
...so, unless there is a way to spoof IP addresses...
Ummm...I've got some bad news for you...
 
Old 04-21-2010, 04:03 AM   #12
Kevin Joubert
LQ Newbie
 
Registered: Apr 2010
Location: Johannesburg, South Africa
Distribution: Redhat & Centos
Posts: 3

Original Poster
Rep: Reputation: 0
Hi Unspawn,

Although my support staff should have the skills, to make sure it gets done without a backdoor (I take note of the comment about spoofing IP addresses) I hired a consultant to set up an e-mail alert system for all login's to root and other high privilege accounts. He is also creating a syslog mirror server system for our high risk accounts. An analysis of the attacked sites showed that one thing they all have in common is a permanent, high speed internet connection, so we've designated those approx 30 sites as 'high risk'.

Creating, testing and installing the various scripts is going to take a few weeks but at least we're doing something positive now, instead of hanging around the coffee room waiting for the next attack.

I also tasked the consultant with reviewing all our security processes and tensioning them up as much as possible. I used the full text of the replies from you and Anomie in my brief to them, so thanks once again for the detailed responses. If you ever end up in South Africa I'll buy you a beer or three. And, on that note, this time when I logged in I saw large, clearly marked "thanks" buttons in every response box. I must be losing my marbles??

Regards, Kevin Joubert.
 
Old 04-21-2010, 04:42 PM   #13
unSpawn
Moderator
 
Registered: May 2001
Posts: 27,118
Blog Entries: 54

Rep: Reputation: 2787Reputation: 2787Reputation: 2787Reputation: 2787Reputation: 2787Reputation: 2787Reputation: 2787Reputation: 2787Reputation: 2787Reputation: 2787Reputation: 2787
Thanks for the Joburg beer offer. I think was a good decision to hire a consultant and I hope it works out OK. If anything comes up that's worth sharing I'm all ears. Wrt the Thanks button I don't known. There's been some small changes but that shouldn't affect this apart from local caching. If problems persist though please consider a thread in the LQ Suggestions forum.
 
  


Reply

Tags
redhat, security, tracing


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off


Similar Threads
Thread Thread Starter Forum Replies Last Post
unauthorised file erase pingvina Linux - Software 28 02-21-2006 12:28 PM
monitor user activity alagenchev Linux - Security 5 01-20-2006 10:02 PM
monitoring user activity buehler Linux - Security 1 09-02-2005 03:46 PM
Tracing which user logs onto which PC kenji1903 Linux - Networking 13 09-13-2004 10:04 AM
User action tracing atlantislim Linux - Security 6 08-26-2002 08:57 PM


All times are GMT -5. The time now is 03:21 PM.

Main Menu
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
identi.ca: @linuxquestions
Facebook: linuxquestions Google+: linuxquestions
Open Source Consulting | Domain Registration