Our company has recently experienced a spate of attacks at some of our customers that have resulted in the root file system being trashed. Although all our servers run a raid controller in mirror mode that allows us to get the customers back up again quite quickly, I am suspicious about the nature of these attacks. Our root passwords are quite strong yet they always get cracked within about 1200 attempts. This makes me think its an inside job that is being disguised to look like an external "brute force" attack. Also the way that the system is crashed, leaving the whole root file system corrupted, is very unsubtle and it occurs to me that this may be done to hide an industrial espionage attack.

Is there any way we can log the activities of everyone who log's onto one of our servers as root user so that we can see what they did before the system crashed. Our support staff very rarely need root access so it won't result in a lot of data to wade through. Also, we have about 100 customers so if we have to purchase a tool to do this it can't be too expensive.

I am using this forum because I'm not sure if the attacker is one of our own support staff, so any advice on how to track the person down would also be greatly appreciated.

Many thanks, Kevin Joubert.

You can look at everyone's .bash_history files, to see what they were doing. Mirror your syslogs to another server, so even if your system log files are edited/deleted, all the events are still getting 'teed' to a remote system, where they DON'T have access. Look at the output from the last command, to see who logged in from where.

Bear in mind that ANY of this can be circumvented by someone who knows what they're doing. In the event of stupidity being the root cause, that's easily fixed.

Surely anyone with half a brain would remove the .bash_history file if they were trying to trash something!?! In any case, it's the root password being cracked, so all you would see is the history in roots log. And there is no way to stop root from removing anything!
Perhaps you can restrict who knows the root password to an absolute minimum set of people and have them all use sudo for everything?

Yep...that's why I said it's easily circumvented in my original post.

SUDO will work, but only if you're very careful to limit the commands it can run, to be absolutely safe, and especially don't grant SUDO shell access, since it then will only log that userx ran a bash shell at time X...but won't log what they DID after that. Got to restrict vi, chmod, chown, mv, cp, and rm too. If they have those commands via sudo, they can then cp the sudoers file to their home directory, chmod/chown, then edit it, and move it back to the 'real' location, modify the attributes, and they're back in. Easier to designate a VERY small subset of commands that SUDO can run, and exclude everything else.

A slightly more elaborate reply ;-p First of all I'm having a few reservations since no hard data is made available with respect to setup, state of security and auditing, types of attacks, co-occurrence, timeline et cetera. (But then again I'll prolly have to contain myself since an investigation is not what you ask for.) Secondly, but probably more important, you'll have to have a plan and an explanation in place before you execute your plan. As staff has admin rights they will notice (or be tipped off by) any changes in server configuration, software or infrastructure. As a third point I strongly suggest you consider consulting with legal council and professional investigative services with respect to Privacy Laws, evidence gathering and possible submission and other issues that may arise from whatever is proposed. (In some cases it might be required to have personnel involved in the investigation to sign an NDA and in some jurisdictions one is required to publicize information about an ongoing investigation. IANAL and I don't know your local Law, so please consider it.) Also since thrashing a fs repeatedly isn't your average cracker modus operandi you'll have to make certain you can rule out hardware errors.


I've edited your OP to filter the facts you posted:
Before you start drawing a plan you should take stock of how staff logs in or performs tasks like physical logins, using any "web panels", network services (SSH, telnet, et cetera), through terminal servers. These need to be evaluated for continued use (telnet?) and if used their logging needs to be not too terse. Less methods available means a more narrow auditing scope. Review login methods (root directly, wheel group access, via an intermediate unprivileged user account). Make sure no network root access can occur other than by first logging in through an intermediate unprivileged user account. Each member with root access rights must have a separate account. (Physical logins I leave up to you. If you've got a quality colo then proper and accurate access records are kept, right?) Change all root passwords to be unique per server. Distribute them on a need to know basis. Set up each intermediate to run 'rootsh' (Bash wrapper). Enable SE Linux and Auditd where possible (or any other MAC you use). Set up a syslog server with Syslog-NG (that only a select group has admin access to, preferably you and one trustworthy admin, and preferably over another 'net). Make all machines duplicate syslog there. Deny su transitions and force all intermediates to transition to root only using 'sudo rootsh -i -u root'. Require all staff to keep a log of which servers they logged into. Corellate logs against syslog server entries. I probably did miss or didn't address something (like for instance verifying all automated and scheduled tasks) but hey, it's a start...

Another thing, whilst not foolproof and a bit messy, would be to set up roots .bash_login script to send an email to an address such as unixsecurity@yourdomain.com which your security admins have access to. Then as soon as someone logs in as root you get an email sent to you with the details. If you pipe the "who" command to it, you may get some additional information such as the address of the PC they are logged in from.
So in your .bash_login for root, have something like:


Then say if anyone intends at all to log in using root they MUST email the security group first to let them know - fingers get lost, people get tar and feathered etc for failing to do so!! This is similar a setup to what we use in our org for root access to the unix servers.


It doesn't stop the attack, but it at least gives you a little bit of an audit trail, somewhere to start from! And if it happens during the day you have as chance of finding out.

In any case, root access should be strictly limited. As said above, use sudo and tie it down!

Time to start limiting attack vectors:

• Another vote for enabling very limited sudoers (i.e. which can only execute scripts that are properly vetted, and perform specific actions -- no editors or pagers)
• No one has root's password except you and the boss (or similar, as per your documented security policies)
• Direct root logins over ssh are disabled
• Services are reviewed on a case-by-case basis for weak configurations and known exploits
• The strength of your strong passwords are objectively tested by pam_passwdqc
• Logs (/var/log/security, etc.) are regularly reviewed for warnings signs
• Latest security and bugfixes are applied

You could setup a remote syslog server and make sure that you are sending all the appropriate logs.

Mount /home directories, /tmp, and /var/tmp with noexec assuming they are partitioned accordingly, in case they are running a rootkit.

I can't for the life of me find the "Thanks button" that some posts refer to so here's a manual thanks very much to frndrfoe , anomie , devnull10 , unSpawn and TB0ne. I have already issued instructions to implement some of the measures you guys have suggested and I have decided to hire an external guru to help dramatically beef up our security and auditing protocols.

We managed to glean some information from the firewall vendor at one of the sites and it seems that the attacks originated from an ISP in Romania, so, unless there is a way to spoof IP addresses, it looks like my inside job suspicions were unfounded. Apart from that almost useless bit if info I was surprised at how little help, or protection for that matter, the ISP firewalls provide in these cases. It's a far cry from the picture they paint when they're doing their sales pitches!

Thanks again, Kevin Joubert.

Good one.


Can you be more specific?

Ummm...I've got some bad news for you...

Hi Unspawn,

Although my support staff should have the skills, to make sure it gets done without a backdoor (I take note of the comment about spoofing IP addresses) I hired a consultant to set up an e-mail alert system for all login's to root and other high privilege accounts. He is also creating a syslog mirror server system for our high risk accounts. An analysis of the attacked sites showed that one thing they all have in common is a permanent, high speed internet connection, so we've designated those approx 30 sites as 'high risk'.

Creating, testing and installing the various scripts is going to take a few weeks but at least we're doing something positive now, instead of hanging around the coffee room waiting for the next attack.

I also tasked the consultant with reviewing all our security processes and tensioning them up as much as possible. I used the full text of the replies from you and Anomie in my brief to them, so thanks once again for the detailed responses. If you ever end up in South Africa I'll buy you a beer or three. And, on that note, this time when I logged in I saw large, clearly marked "thanks" buttons in every response box. I must be losing my marbles??

Regards, Kevin Joubert.

Thanks for the Joburg beer offer. I think was a good decision to hire a consultant and I hope it works out OK. If anything comes up that's worth sharing I'm all ears. Wrt the Thanks button I don't known. There's been some small changes but that shouldn't affect this apart from local caching. If problems persist though please consider a thread in the LQ Suggestions forum.