Hello everyone,

I'm a long time follower of LQ, but just registered because now I have my own problem with which I need help.

Yesterday I discovered that my server security has been breached on Jun 16 at 2:53 AM and somebody gain root on it, deleted my /var/www and /home and then powered off the server.

Several question marks arise mainly because he did not use a brute-force attack.
Anyway, about my server:

- Debian 5.0.8, Lenny. Linux version 2.6.26-2-686, x86
- services that run on it and are exposed to the internet:
haproxy 1.4.11
apache 2.2.9
bind 9.6
postfix 2.5.5-1.1
ssh 1:5.1p1-5 - only Protocol 2
The rest is firewalled, with a DROP policy.
-PHP (5.2.17) has disable_functions = passthru,system,proc_open,popen,curl_exec,curl_multi_exec,parse_ini_file,show_source
- fail2ban allows 4 tries on SSH, and the bans the ip for 2 days.
- 90% of the accounts in /etc/passwd have /bin/false as shell, exceptions are: root, daemon, nobody, and 3 user accounts which are disabled with passwd -l.

Logs:

last:
reboot system boot 2.6.26-2-686 Fri Jun 17 02:38 - 11:18 (08:40)
root pts/0 77.211.97.166 Thu Jun 16 02:53 - down (00:11)
Here, I don't understand what happened because from Thursday it jumped to Fri. And I'm seeing this jump of 24hours in system logs too; then it comes back to the current date (at that time), ie Thursday 16th.

auth.log
Jun 16 02:53:23 storage sshd[3206]: Bad protocol version identification 'NICK idioten' from 77.211.97.166
Jun 16 02:53:27 storage sshd[3209]: Bad protocol version identification 'NICK idioten' from 77.211.97.166
Jun 16 02:53:31 storage sshd[3216]: Bad protocol version identification 'NICK idioten' from 77.211.97.166
Jun 16 02:53:34 storage sshd[3219]: Bad protocol version identification 'NICK idioten' from 77.211.97.166
Jun 16 02:53:38 storage sshd[3222]: Bad protocol version identification 'NICK idioten' from 77.211.97.166
Jun 16 02:53:40 storage sshd[3212]: Accepted password for root from 77.211.97.166 port 49777 ssh2
Jun 16 02:53:40 storage sshd[3212]: pam_unix(sshd:session): session opened for user root by (uid=0)
Jun 16 02:53:41 storage sshd[3231]: Bad protocol version identification 'NICK idioten' from 77.211.97.166
-same message repeats every 3-4 seconds until 02:59:18-

syslog:
Jun 16 03:05:23 storage shutdown[3731]: shutting down for system halt
Jun 16 03:05:26 storage init: Switching to runlevel: 0

messages:
Jun 16 03:05:23 storage shutdown[3731]: shutting down for system halt
-nothing new-

One bitter irony is that on the same day I finished configuring Nagios and used a plugin to warn me if somebody is logged in and from where. It worked, but I never got the alert (anyway, even if I did, I was sleeping at that time).
Jun 16 03:04:36 storage nrpe[3690]: Host is asking for command 'check_who' to be run...
Jun 16 03:04:36 storage nrpe[3690]: Running command: /usr/bin/sudo /usr/lib/nagios/plugins/check_users.sh -w 1
Jun 16 03:04:36 storage nrpe[3690]: Command completed with return code 1 and output: USERS WARNING: [users: root(1)] --- root pts/0 2011-06-16 02:53 (77.211.97.166)

bash history is untouched. whatever he did, nothing was logged. my previous commands are all there, but not his.

My root password is quite complex - lower case letters with upper case letters with numbers and with punctuation marks. passwordmeter showed 100% strength on it. I doubt he'd crack it.

So this is it. He stayed 11 minutes logged in, deleted my /home and my /var/www. I don't understand why he didn't delete /root (I had quite a few valuable things there), or even a system dir, or backdoor my server.

Lessons learned:
I know, never allow root access. And now I won't. Disable root login from ssh, use other users and sudo.
My next critical mistake was allowing everyone to ssh (I previously didn't).

Now if anyone has any idea what was exploited..please do share.
I'm currently trying to recover my deleted files from /var/www. If you have any tips, I'd appreciate it.

Let me first officially welcome you to LQ. If you have been following the security forum for a while you have a pretty good idea of how we operate. It looks like you have already begun the investigative process. One thing I might suggest is that you hold off on the attempt to restore your system, until you find out how they gained entrance. Based upon what you have posted I think it is pretty clear that they did.

1) The first thing to do would be to secure the server as best you can. Ideally, you don't want to turn things off, restart, etc so please avoid that as much as possible. If you have physical access to the machine, leave the network cable unplugged, otherwise put up the firewall to everything but a secured SSH connection.

2) Once you have secured the machine, take the time to review the CERT check list. It will give you an idea of the things to look for, which is where we will begin the investigation. Here is a link to the CERT Checklist.

3) Next, please run and obtain the output of the following commands: ps axfwwwwe, netstat -pane and lsof -Pwn. You will want to capture the output to a file. You will need to run these commands as root. The files will be large, so once you have obtained them we can work with you to make arrangements to upload them. I would also suggest that you make copies of all the log files. They will need to be analyzed.

4) Make copious notes, which you have started, of the server processes you have running, including version information and patch level. We can investigate the CVE records for known exploits.

1 members found this post helpful.

will help with the CVE queries.

szboardstretcher, the user indicated that they are running Debian 5.0.8, Lenny. Linux version 2.6.26-2-686, x86. Debian systems do not use RPM.

Everyone, please wait for the OP to provide the requested information before we start jumping to conclusions.

Hello Noway2,

A first step I made was to DROP all SSH access and allow only several trusted IP's.
Then I did a dd after the partition and transferred it on my PC (so I have access to all the logs and /).
I do have physical access to the machine; I will be moving it to my room this night for better access.

Currently I have powered off the machine to keep it from writing any information to disk, thus blowing away any chance to recover my files.
But before I powered off, I took a look at the process list - nothing was unusual. lsof -Pni showed there was no hidden listening connection.

I'll have to start up the machine to get outputs from "ps axfwwwwe, netstat -pane and lsof -Pwn".

About the patch level, I did an apt-get upgrade a few days before. So everything that was patched in the repo, I had it installed.

I will review the CERT list and make the next steps within the next hours, as I am not on site right now.

Thank you!

I would also take a list of what you have running in apache.

@theblah,

You said that fail2ban is supposed to shut down SSH access after four failed attempts. Could you look at your logs and see if you can tell if fail2ban is actually working? Could you also look for that IP address and/or similar ssh error messages? And lastly, is there anyone else besides you who knows the root password?

@nomb - well, there's nothing left to run anymore. he deleted everything.
@Hangdog42 and Noway2 - I have to put off for two days this operation - got to get out of town tomorrow and I'll be returning on Sunday. We'll resume then.

Thank you for your support! BBL

is this a server you have physical control of?
if it is, a few suggestions to prevent such a thing from happening again

ideally a server SHOULD NOT be the same machine you use as your desktop for day to day messing around, too much security risk as hackers can get in through CLIENT software running on the machine as well as server software, in fact that is even MORE likely to happen

1) install a second nic in the server and configure firewall rules to only allow remote management such as ssh to listen to that interface and not the one attached to the public facing internet (see diagrams)
2) make sure ONLY the specified ports are forwared to that machine
3) install a second router and have only it and the public facing nic of the server attached to the internet server
4) if possible
--- A) separate some of the functions provided into either separate real machines or virtual machines (in short, try not to put all your eggs in one basket)
--- B) make sure the servers (real or virtual) are running a stripped down version of the LINUX OS with ONLY the software necessary to run the server software and nothing more, no unnecessary software that a hacker can gain control of
--- C) NEVER run servers on a machine you use as for your day to day computer usage, even if the other machine is a virtual machine on that machine

and above all, NEVER run as root unless you need to be root to perform a task, and even then run the task with either
sudo or su -c

the computer shop I work at had a setup like this

that kind of setup is overkill perhaps for your purposes but something like this should work
note in both instances, eth0 does not talk to eth1 on any of the servers

this is assuming the server is a separate physical entity
the other option would simply to make the server(s) virtual machines, not as good but works

Ping: theblah,

Just following up with you on this situation to see if you have been able to obtain any of the information for the analysis yet?

Back again.
For the past day I have been dealing with inodes, blocks, debugfs and ext3grep.
Unfortunately I haven't had so much success, partly because I made the mistake of running
photorec (from testdisk) and it made some files in /var/www, thus overwriting some inodes.
Alas, I did my best, and I also made a clone, so there's no point of keeping the server down anymore.

@Hangdog42 - Only I know the root password.
As far as fail2ban goes, it didn't kick in because the password was entered correctly from the first try.
I can confirm the fail2ban works and it was working at that time (it bans and unbans ips).
The messages produced in auth log
Jun 16 02:53:23 storage sshd[3206]: Bad protocol version identification 'NICK idioten' from 77.211.97.166
Jun 16 02:53:27 storage sshd[3209]: Bad protocol version identification 'NICK idioten' from 77.211.97.166
are, as I understand, due to protocol mismatch. He didn't go as far as logging in with various users/passwords, in which case fail2ban would have banned him.

@frieza - Thank you for the diagrams. Unfortunately, I think they're kind of an overkill for me. I'm not that enterprise-level.

@Noway2 - I have the outputs for ps, netstat and lsof.
I just remembered a critical change I made 2 days before this happened.
I wanted to enforce a password policy on user accounts, so I changed in /etc/pam.d/common-password:

password required pam_cracklib.so retry=5 minlen=7 lcredit=0 ucredit=-1 dcredit=-1 ocredit=1 difok=3 debug
password required pam_unix.so use_authtok nullok md5

Before the "correct" configuration, I remember having trouble with it. I could su as an user, run passwd and then put any password, despite the password requirements I had set up. Passwd would throw in an error saying the tokens haven't been updated (something like that), but when logging in as the user, it actually had the password I just entered.
In the current configuration I have, it seemes to comply with my requirements: 5 retries, minimum length 7, no credit for lowercase chars, one uppercase char required, one digit required, one credit for other chars.
Before this change I had:

password required pam_unix.so nullok obscure md5

The reason I was asking about this was that protocol mismatches like this are a way of doing log poisoning, which is a weird form of DDOS. If there were a lot of these, it also may have been a way to try and get some information from openssh. There is an old bug against that version of SSH that may allow attackers to gain information. It was considered pretty much a long-shot, but it is there.

Which of course brings up the question, how do you handle patching and what was the state of patching on this machine?

My guess is that the protocol mismatches are due to IRC connections being sent to your SSH service that the attacker didn't configure correctly. "NICK idioten" is an IRC command and according to Google, "idioten" is German for idiot.

1 members found this post helpful.

This is obviously incorrect as clearly the perpetrator knows your root password. The question is how did they get it in the first place?
Did they perform a brute force password guessing and cover up their tracks after success? The problem with line of approach is that fail2ban should have picked up traffic from this attempt, which may have occurred in the days, weeks, or moths prior to your becoming aware of a problem. One thing that I have found is that if you use an application that monitors logs for suspect traffic is that an application like fail2ban may prevent alerts because it stops short term events before they reach the trip point. Granted, it would have slowed them down to the point that statistically it is unlikely that they would have success. This too leads me to think that there is a different vector.

OlRoy brings up a good point about the "NICK idioten" messages. It also reminds us of the important of not jumping to conclusions and not being lead astray by the glaringly obvious. This is why a fact based investigation is so important. While I can certainly appreciate your desires to recover your lost(?) data and restore your operation, I have serious concerns about how entry was gained in the first place and whether or not you will be subject to a repeat occurrence.

Message #66 in the list says: "Err, hang on. As I said, I backported it in openssh 1:5.1p1-5, which is
the version in stable. That means there's nothing to do, right?"
My version is: OpenSSH_5.1p1 Debian-5, OpenSSL 0.9.8g 19 Oct 2007 - so this *should* mean I have the patched version.

In sources.list I have:
security.debian.org/ lenny/updates main contrib
volatile.debian.org/debian-volatile lenny/volatile main contrib non-free
packages.dotdeb.org lenny all

I don't have a policy for upgrades, I do them when I mostly feel/remember to. But as it happens, I just upgraded a few days before the hack:
history | grep apt-get
6608 10/06/11-14:03:31 apt-get update
6609 10/06/11-14:03:47 apt-get upgrade
upgraded on 10, hacked on 16...
I have the log of that upgrade and SSH is not on the list.

Wasn't it /nick $name ? He's missing the /. As for the idioten...yeah, I guessed..