You are of course right, mase... There's no question he DID have root access. But I see no way to figure out how or when he got it. The files involved in this attack date clear back to 2006. My server didn't even come into existence until 2008. So, it's pretty much impossible to determine when the initial penetration actually occured.

That means if I do nothing else for the next 60 days but rebuild the server and all 25 of the sites hosted there and do everything possible to tighten security, he could STILL be back 5 minutes after the server is up and running on the net again. I hate to admit that. But it's true.

Under those circumstances, what's the incentive and where's the justification to rebuild? I personally watched a huge multi-million dollar corporate webhosting firm with thousands of active servers deliberately and systematically ignore (and even hide) widespread server vulnerabilities and penetrations for years because there was NO way to remove them without admitting to thousands of clients that their server and network security sucked dead bears and wasn't worth the cost of the gunpowder it would take to blow it straight to hell.

That's why I went out on my own. I figured I couldn't do any worse than those guys were doing while I was paying them to do much better.

You're right, Wim. But let's face it we humans are eternal optimists. That's why we keep getting up every time we fall. ;-)

He may well get in again. The only system that is 100% secure is not plugged in, encased in cement, and dropped to the bottom of the Marianas Trench...and then only because it's such a bother to go down and retrieve it.

So you can't make it 100% secure and still use it. But you can make the intruder respect you...

Well it is pretty clear I think that he got
in through a security hole, because you didn't upgrade.
(rather than maybe a configuration problem)
And since he got root rights and you can track the debian security
mailing list back and search for privilege escalation bugs as well as
bugs in one of your services,
that should be enough to know what it was.
I think there where only a few bugs of that kind.
And that's why I strongly believe, that installing a fresh and fully updated system will make you secure again.

Here is the mailing list, one of the thinks you should do is
subscribe to it:

http://lists.debian.org/debian-security/


Also you have to differentiate, up until this point the attack might
as well have been scripted. A script that runs somewhere with no user attention that automatically infects hosts with a specific security hole and installs shv5. At that point in time you might as well be able
to completely clean the infection.

But if an attacker actually accessed your server then you don't know what happened since he could have done anything. And in that case you will stay infected and none of your programs / services can be trusted anymore.

Thanks, Jim. The reference to Tripwire is helpful. I'm also hopeful about COPS (ftp://ftp.cerias.purdue.edu/pub/tool...ps.1.02.README) and Tiger (http://packages.debian.org/lenny/tiger and http://www-arc.com/tara/) both of which I discovered at the very useful www.yolinux.com (see: http://www.yolinux.com/TUTORIALS/Lin...rityTools.html).

Tripwire has been added to my server security evaluation list.

Thanks.

Amen to that, Jim!

Once again, you're right. Blunt as a blackjack, perhaps... but right! With THAT said, I think I'll go have a left over lollipop and flip a coin... heads, we rebuild... tails we have another lollipop...

Before helping you with suggestions on how to assess the situation better I think I should first be allowed to change your perspective a bit here. Be warned this may come across as me lecturing you. That is not the case. Please try to see through that and just focus on the aspects I point at. I'd like to present you with three arguments that should lead to mitigating the situation as I asked you to consider before, looking at the situation from the vantage point of:

0. You
Running GNU/Linux is all about performance, protecting assets and providing services in a continuous, stable and secure way. Trustworthy, stable and secure enough for many companies and institutions to rely on for critical processes. What most people forget is that running a GNU/Linux machine is also a responsibility. What you have is a root compromise. This simply means the machine is no longer under your control. You are responsible for mitigating that situation swiftly and decisively.

1. Us
Again: you have a root compromise. Without you controlling the machine it may be or may have been used to attack other machines over the 'net. As such the machine not only poses a threat to you but also to us.

2. Your business
Best practices state a compromised machine remains untrusted and must not be used. In a root compromise scenario any communications may be sniffed and any data on the server may have been siphoned off. This includes not only databases containing commercial and private information but also personal or company account details which may be sold or abused or otherwise used to damage business. If you run a company then you (may) have (paying) clients who trust you. Not timely alerting them of the compromise, not allowing them the opportunity to stop using the server or take measures will kill your business once the word gets out because trust will be lost. You know very well that trust is easier lost than regained.


If after reading this you feel that you still have good reasons to keep the server in use as-is, postpone any mitigating measures or otherwise stalling seems sensible please let me know.

More to the point, the OP is responsible for what his server does, even if the hacker initiates it. How understanding will his customers be if the server gets blackballed because it was used in a DDOS attack, or as a launching point on an attack on some other server?

Imagine the lawyers getting involved...?

Thanks. (Not to promote things too much but for the upcoming release of Rootkit Hunter 1.3.6 I'm revisiting all known rootkits and adding in missing details. I've always thought of doing that as just making RKH as comprehensive as possible (as rootkit compromises have nearly died out), but your compromise shows me this wasn't a waste of effort. If you want to run RKH from CVS with all the new details you can get it with 'mkdir /tmp/rkh && cd /tmp/rkh && cvs -Q -z3 -dserver:anonymous@rkhunter.cvs.sourceforge.net:/cvsroot/rkhunter co -P rkhunter'.)


Root compromise: unless you have safe and independent means of verifying integrity (Aide, Samhain or even tripwire, package management) all bets are off.


SHV5 installs SSHd and keys but not with the default names. SHV4 does but places it in /lib/security/.config/ssh/.

This is the Linux Security forum. When doing proper incident response we should react to facts, not fiction. (For those who have done actual forensic investigations you know what I mean, you know what happens if an investigator starts to think or think up things. He gets biassed, starts to interprete things, is prone to dismissing clues or tries to match up available evidence to what he wants to see.) Please just be careful.


That's what I thought of as well. Unpatched Debian has seen its fair share of SSL/SSH problems in the past 1-2 years.


No it won't. Recovery takes more than that ranging from ensuring adjacent systems are clean to changing all auth to properly hardening. And especially in his case most of the default hardening won't even depend on the outcome of an investigation.


Even if you have a complete timeline of events and a full understanding of the compromise that is really, really bad advice. Please don't.

A system installed from scratch and updated without any services or anything yet configured is secure as in there won't be any traces of the rootkit left that's what I mean.

How to harden the system would have been another topic in itself.

I didn't advice him to do it, I just told him that there is a possibility. He can't do what I said anyway since he has no means
of knowing if the attacker actually accessed the server.
That's why my advice would be to reinstall from scratch as I've told
him already.

I didn't feel lectured at all, Unspawn. Indeed, by the time you had made this post I was already in the process of developing a plan for how to move my existing web clients off this server while I rebuild it and then move them back again after that's done. In fact, when I spotted and read this post I was already talking with my clients about what a full-scale server evacuation and a rebuild and move-back strategy. So far, everyone has agreed to stick with me and has accepted my plan.

Frankly, I knew all along what the decision would probably be. But before I tossed out both the baby and the bathwater in a mad rush to get off the server, I needed to think the problem through and come up with a short a plan for how to make it work.

Thanks for the comments and feedback. You were right on the money.

chkrootkit and rkhunter at least hinted at the presence of both the SHV4 and SHV5 rootkits. From what I saw, I believe that's right. I've concluded both rootkits are present on my server. I've been in the profession a long time. I studied the impact of these 2 beasts fairly carefully. There are several relatively easy ways to identify them and I'd be willing to bet the files and directories they create are standard in name too. So, yes, I could probably offer some ideas about what to look for.

But if I posted those details here, they might well change tommorrow...

Screw the damned lawyers... Frankly, Jim, while you and I may take this seriously so do the huge hosting companies and their entire staffs from the CEO to the lowest support techs who collectively conspire to ignore, obfuscate and look the other way when their own dedicated and shared hosting servers and their vast corporate revenues are threatened.

Let's face it. I may be morally obligated to try to find a solution that works for my hosting customers; but no government agency is going to fund me and there are no prison sentences for small web hosts who choose to look the other way as they drive by either. The fact is there are hundreds (if not thousands) of compromised servers and millions of PCs around the world that have been similarly compromised by rootkits or zombieware. And ALL could be aimed at federal, state and local governments, police and media sites and financial institutions around the world and there'd be NOTHING short of pulling the plug on every computer on the planet that anyone could do to stop it. If you don't believe that's true, ask the financial institutions of Uzbekistan that were attacked and brought to their knees 3 years ago for over 2 weeks after the government approved the removal of a Soviet era Russian statue from an obscure municpal park or plaza.

There's no question the same thing could happen in America and if it did, it would eventually be proved that legions of zombies recruited from our own pc and server infrastructure had been used to mount such an attack. We've certainly seen lots of evidence for years now that those of us on the internet's front lines have been fighting the initial skirmishes of a cyber war. Do you see Homeland Security doing anything about it? Hell no! They'd much rather grope their way through suitcases filled with the panties of dangerous-looking grandmas at airports instead!

We've been lucky. So far we haven't experienced a large enough, long enough or devastating enough cyber-attack to provoke an arrogant and tech-naive president like George Dubyah - who had a John Wayne complex and advanced testosterone poisoning - into declaring hacking to be an act of war and starting a global war on hackers yet. But God knows that day could come tomorrow!

Meanwhile, I plan to keep slugging away here. Fighting off the bad guys as long as this tired old bod draws breath. I may take a cyber bullet or an rpg or two, but I'm a tough old buzzard. And the truth is I'd MUCH rather die at the front, wearing my boots and fighting a cyber war than to do it in bed in a nursing home somewhere!

Who knows? Maybe I'll be the first to be posthumously awarded the Congressional Medal of Cyber-Honor! Come on you ayrab-commie-hacker-bastards... come right-on down here to my old adobe computer room with the 18" thick walls and dig me out!! Ratatatatatat!! KABOOM!!!