Hello Everyone,

Fighting a bit of a nasty morning... anyone seen this before?

We have a number of servers that have password authentication disabled as well as shell access disabled for all users except those whom have keys. These servers run cPanel and have been updated to the following specs:

2.6.18-164.el5PAE #1 SMP Thu Sep 3 04:10:44 EDT 2009 i686 i686 i386 GNU/Linux
openssh-4.3p2-36.el5_4.2

Early (around midnight-1am CST) this morning we had a widespread attack via an unknown vector. In the attack, the only thing that I can find is the following (IP blacked out, although it is the attackers' address):

Nov 12 04:31:22 sharedserver/sharedserver sshd[16083]: Received disconnect from 100.100.100.100: 11: No supported authentication methods available
Nov 12 04:32:14 sharedserver/sharedserver sshd[11265]: Received signal 15; terminating.
Nov 12 04:32:14 sharedserver/sharedserver sshd[16570]: Server listening on :: port 2.
Nov 12 04:32:14 sharedserver/sharedserver sshd[16570]: error: Bind to port 2 on 0.0.0.0 failed: Address already in use.
Nov 12 04:32:27 sharedserver/sharedserver sshd[16611]: Accepted password for root from 100.100.100.100 port 3630 ssh2
Nov 12 04:32:27 sharedserver/sharedserver sshd[16611]: pam_unix(sshd:session): session opened for user root by (uid=0)


The concerning part is that it obviously appears that there is someone reloading SSHD, but there is no successful login (at all) via shell prior to this.

This time corresponds with a modified sshd_config that then allows password authentication, whereby the user then logs in as root and has a good time, so to speak.

I know that the following vulnerability is out in the wild:

http://www.securityfocus.com/bid/36038/discuss

However, since the user never actually logged into the server from what I can see, I'm still searching for the real way that this occurred.

I have logs from these servers, if you need other information to possibly help track this down that is possible. I'm having a hard time finding the vector for this attack though...

Any assistance would be greatly appreciated.

That actually looks like a post logrotate(8) SIGTERM and restart.

As for root logging in directly...? That's a problem.

Unfortunately this is not a logrotate... which is what I first suspected. Most of the logs on these servers rotate weekly on Sunday.

- You say "number of servers". How many?
- Does the same sequence show on all servers?
- At the same time?
- Same source address?

- Any other activity in other system or daemon logs?
- Can you read back and see if there is an abnormal amount of recon done or unusual recon patterns?

- What web stack services do these server share except the web-based control panel?
- Is all software of the latest version?
- What does verifying the whole system show?
- And of the files which are not part of RPM packages?

- Any unusual processes?
- Anything else you think would be useful to add?

- Did you already mitigate the situation?
- If you did: how?
- If you didn't then you know replacing sshd_config requires elevated privileges so I'd suggest you also get familiar with the Intruder Detection Checklist (CERT): http://web.archive.org/web/200801092...checklist.html and perform those checks. Before you do best save all process, open files, network and user details, raise the firewall to deny access except from your management IP (range), kill sshd and ensure the binary is not subverted before running it with a known clean config.

Uh, did anyone else notice this:

That does seem a bit odd since 100.100.100.100 is a reserved IP address.

I think he is saying a hacker restarted ssh configured to accept password logins, and then logged in. He probably replaced the hacker's real IP with 100.100.100.100

But if the hacker logged in though SSH, he either has this guy's root pass, or the hacker replaced the hash in /etc/shadow

I agree, there is some serious potential for trouble here. Unfortunately unless the OP decides to cooperate, we're kind of stuck.

There are far too many OP's here wanting an easy solution to a complicated problem.

...and what's worse is that the OP is simply sticking his head in the sand trying to ignore the problem, see: http://www.linuxquestions.org/questi...on-tmp-770006/

It sounds like he thinks SElinux is a fix all for security, including a fix all for his already hacked box, and a fix all so he never has to apply another patch again.

This guy also posted the same on thing as his other post on the cpanel forums, and on web hosting talk.

As I read his other thread I understand he never deployed SE Linux in the first place...


Sure. In terms of maximizing ones chances for the topic being eyeballed that's good. The downside is a lot of OP's will forget to revisit and update their threads.

Actually, I hadn't forgotten, I was fighting this.

Ultimately this issue stemmed from a feature that I missed in cPanel, where you can enable/disable password authentication via the interface. Same for additions to hosts.allow/deny, and in some cases where you have something such as APF or CSF installed via a plugin that allows management of iptables - the same there as well.

Simply chattr +i stopped this particular issue. In the meantime though I have been fighting through the multiple issues that have arisen from this. Since I have prevented access to the box now, I've been more concerned with the heart of the issue - getting off the rooted boxes that, no matter what I add TO the boxes, are still technically rooted.

While I could go through the trouble of configuring and building a new kernel, I thought my time during the day was better geared towards moves. I still stand by that decision.

I have no desire to spend time securing that which may, or may not, be impossible to secure simply due to the nature of the original attack. I can't trust those boxes. Why waste that time, when I can instead dedicate my time to the new boxes that are secure?

This attack turned out to be a compromise of some of our WHMs, which was propagated through a few PHP functions that some of our other, now deeply berated coworkers, re-enabled without consulting us.

I think he is talking about the setting:
"SSH Password Auth Tweak"
Which is in the Security Center in WHM.
Which can only be changed as root.

Is he trying to say someone at the user level changed that setting as root through a php function that he though he had disabled?

And then SSH'd in afterwards using the root password?

Fighting on your own, then. We're here to help, hence the many replies, but because you didn't update your thread but only started one (I think might be related) there is no way for us to know, right?


And while I think it's admirable you're fighting this on your own, judging from the questions in your other thread and lack of response, I'm starting to wonder if you have a complete overview of main cause(s) (vs symptoms, collateral) and how to guard / harden against those. Or am I mistaken?