LinuxQuestions.org
Review your favorite Linux distribution.
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - Security
User Name
Password
Linux - Security This forum is for all security related questions.
Questions, tips, system compromises, firewalls, etc. are all included here.

Notices


Reply
  Search this Thread
Old 11-02-2009, 03:14 PM   #1
websissy
Member
 
Registered: Jul 2008
Posts: 49

Rep: Reputation: 15
I've been hacked... NOW what?


I've been trying to upgrade my copy of debian etch to prepare for an upgrade to lenny. The server installation is 1 year old and it hasn't received any updates installed until today.

In the course of trouble shooting issues with the upgrade, I noticed an error message in the boot logs that said:

Quote:
Searching for splash image ... none found, skipping ...
/bin/ls: invalid option -- v
Try `/bin/ls --help' for more information.
And when I posted that message seeking an explanation, I was asked to do this:

/bin/ls -version

That resulted in the following message:

Quote:
ls - GNU fileutils-3.13
and the person advising me responded with

Quote:
You have a rootkit. Probably shv5.

This guy had the same problem:

http://www.void.gr/kargig/blog/2009/...ecurity-issue/

I suggest backing up "user data", with that I mean everything that is not binary.

Configuration files for example. But check them.

Reinstall from clean media, preferably lenny.
In addition to being in total shock, I'm lost here and not sure how to proceed. Frankly, since I'm not sure how (or when) this happened to begin with I'm not sure if I know enough to avoid letting it happen again.

Any advice here from someone with experience in these situations would be GREATLY appreciated!

Can I rely on tools like chkrootkit and rkhunter to help me determine what rootkit is involved and how long it has been here. Or am I trying to stand up in a quicksand pool?

Thanks!

Last edited by websissy; 11-02-2009 at 03:17 PM.
 
Old 11-02-2009, 03:28 PM   #2
MS3FGX
LQ Guru
 
Registered: Jan 2004
Location: NJ, USA
Distribution: Slackware, Debian
Posts: 5,852

Rep: Reputation: 357Reputation: 357Reputation: 357Reputation: 357
Quote:
Frankly, since I'm not sure how (or when) this happened to begin with I'm not sure if I know enough to avoid letting it happen again.
Well, I will give you one hint:

Quote:
The server installation is 1 year old and it hasn't received any updates installed until today.
I think this is a pretty good indication of what happened. If no security updates have ever been applied, clearly that is the first suspect. I don't know Debian well enough to determine the number and severity of the security updates you missed, but certainly missing any security updates is already getting off to a bad start.

As your adviser said, backup only the server's content. None of the programs, or even the settings. He mentioned that you could copy the configuration files if you "check them", but that would assume you know what to look for, and no offense, but it doesn't sound like you do; so I wouldn't risk it.

Copy everything to known clean media, run a few checks on it to be sure it is clean, and then wipe the server. Install your OS, and keep it updated. Of course, go through the security basics before the new server goes online. Use strong passwords, disable anything you aren't actively using, and make sure you are using sane permissions.

What role does this server play? WWW, DNS, DHCP? That would be nice to know, as there are service-specific tips to help secure your installation as well.
 
Old 11-02-2009, 03:48 PM   #3
unSpawn
Moderator
 
Registered: May 2001
Posts: 29,409
Blog Entries: 55

Rep: Reputation: 3582Reputation: 3582Reputation: 3582Reputation: 3582Reputation: 3582Reputation: 3582Reputation: 3582Reputation: 3582Reputation: 3582Reputation: 3582Reputation: 3582
Quote:
Originally Posted by websissy View Post
Code:
ls - GNU fileutils-3.13
Finding 'ls' in your bootup messages is kind of odd itself but 'ls' has moved from the "fileutils" to the "coreutils" package in or around 2003 so that is cause for concern. If this is really a root compromise involving the SHV5 rootkit then you'd also find references to libsh.so, ttymon and ttyload binaries, and the latter two also in some of the SysV startup files.

I agree the server should be reformatted and have the OS reinstalled from scratch but before doing that it would be good to investigate, come up with evidence of a breach of security and create a timeline of events. This may take some time and given the state of (mis) management the server is in right now might not give you much more facts than starting from scratch. While you ponder your choices in the meanwhile save detailed process, open files, network connection and auth data off-site, then make the server unavailable by stopping public services, raising the firewall except for traffic from/to your management IP (or range), if the machine is remote you'll only need SSH.
 
Old 11-02-2009, 05:55 PM   #4
websissy
Member
 
Registered: Jul 2008
Posts: 49

Original Poster
Rep: Reputation: 15
Okay, MS3FGX. You're telling me I deserve to be beaten. I think I just HAVE been. But thanks for the reminder that even after 42 years in the IT biz I'm STILL not perfect.

By the way, I mis-spoke when I said the ls -v command was in a bootlog. That was wrong. I actually ran a debian script (dpkg-reconfigure) and captured the output to check it. I noticed the ls error in that file and questioned it. That's how the rootkit was discovered.

Frankly, I felt I WAS being attentive to security. But I admit this is my first time admin-ing a "www" server where I assumed full admin responsibilities (as opposed to admin-ing a strictly internal departmental server with NO web access or a dedicated www server that was supposedly being admined by a dedicated server hosting company.) At my last hosting company I had a server penetration within weeks of receiving my server. I reported the problem immediately. It took 2 years to convince their admins there WAS something wrong and get them to fix it! But I was forced to do the user account rebuilds myself.

I never forgot that.

That's why I decided a year ago to become the admin for my server. If I was paying them to admin and they weren't doing it, what was I paying them for? The way I saw it, if I failed, I had no one to blame but me and in the process I'd learn a lot. At age 59 I admit I'm still learning.

My rationale for not installing Debian updates was I didn't want to be running a constantly churning (unstable) server. So, I decided I didn't want to be constantly applying updates. In deciding that I didn't realize updates released by Debian would include security patches I SHOULD BE applying. At my last hosting company my server ran the same OS release for 2 years without them ever applying ANY security updates. I had NO idea a linux server could be so easily (and invisibly) compromised from outside. I've learned an invaluable lesson.

But ours is a profession where there are no excuses for ignorance. Yes, I screwed-up. I'll do my best to not make that mistake again. But I'm determined to try again anyway.

BTW, I did download and run rkhunter and chkrootkit. They agreed the server is infected with SHV5.

So, I'm forced to rebuild. Mea Culpa! What more can I say? There's no sense beating me. I'm doing a fine job of that myself.

Last edited by websissy; 11-02-2009 at 06:47 PM.
 
Old 11-02-2009, 06:42 PM   #5
websissy
Member
 
Registered: Jul 2008
Posts: 49

Original Poster
Rep: Reputation: 15
Quote:
Originally Posted by unSpawn View Post
Finding 'ls' in your bootup messages is kind of odd itself but 'ls' has moved from the "fileutils" to the "coreutils" package in or around 2003 so that is cause for concern. If this is really a root compromise involving the SHV5 rootkit then you'd also find references to libsh.so, ttymon and ttyload binaries, and the latter two also in some of the SysV startup files.

I agree the server should be reformatted and have the OS reinstalled from scratch but before doing that it would be good to investigate, come up with evidence of a breach of security and create a timeline of events. This may take some time and given the state of (mis) management the server is in right now might not give you much more facts than starting from scratch. While you ponder your choices in the meanwhile save detailed process, open files, network connection and auth data off-site, then make the server unavailable by stopping public services, raising the firewall except for traffic from/to your management IP (or range), if the machine is remote you'll only need SSH.
As I said above, I was wrong when I said the ls -v command was in a bootlog. I had run a debian script (dpkg-reconfigure) and captured the output to check it. I noticed the ls error in the log and questioned it. That's how the rootkit was discovered.

I realize the server will need to be rebuilt; but before I start that, I'd like to try to figure out how long this compromise has been present and determine if there is any way at all to lobotomize it or keep it from doing any further harm. If it has been there for months (I suspect it may have been), then rather than take 25 client sites down for weeks while I rebuild, I might conclude it's best to leave the compromised system running and keep client sites running while I work on building a completely new server or building a completely new drive on THIS server.

I know I may be whistling in the dark here. But most of my clients are small businesses and their www server needs are not complex - a few html pages, a web form or two and some streaming music or videos. Only two of them involve significant databases and both of those sites are mine. But despite their size, my small business customers really do NEED their sites to be up and accessible on the web. So, my goal here would be to avoid putting them completely out of the web business while I tackle this rebuild.

Thanks a lot for your comments and any feedback you can offer.

Last edited by websissy; 11-02-2009 at 06:50 PM.
 
Old 11-02-2009, 07:08 PM   #6
chrism01
LQ Guru
 
Registered: Aug 2004
Location: Sydney
Distribution: Centos 6.10, Centos 7.5
Posts: 17,655

Rep: Reputation: 2484Reputation: 2484Reputation: 2484Reputation: 2484Reputation: 2484Reputation: 2484Reputation: 2484Reputation: 2484Reputation: 2484Reputation: 2484Reputation: 2484
The problem is, now your server is affected, your clients might pick up something nasty from your server. Note that I don't know the shv5 kit, but I'd be worried about my clients.... you're in a tricky position.
 
Old 11-02-2009, 09:04 PM   #7
nigelc
Member
 
Registered: Oct 2004
Location: Sydney, Australia
Distribution: Mageia 7
Posts: 374
Blog Entries: 4

Rep: Reputation: 65
This is what google showed try this:

http://sourceforge.net/projects/rkhunter/
 
Old 11-02-2009, 09:45 PM   #8
Wim Sturkenboom
Senior Member
 
Registered: Jan 2005
Location: Roodepoort, South Africa
Distribution: Slackware 10.1/10.2/12, Ubuntu 12.04, Crunchbang Statler
Posts: 3,786

Rep: Reputation: 282Reputation: 282Reputation: 282
There is a sticky with security references on the main page of this subforum.

And I recently stumbled over 20 Linux Server Hardening Security Tips; it might be of help.
 
Old 11-02-2009, 11:42 PM   #9
websissy
Member
 
Registered: Jul 2008
Posts: 49

Original Poster
Rep: Reputation: 15
Quote:
Originally Posted by chrism01 View Post
The problem is, now your server is affected, your clients might pick up something nasty from your server. Note that I don't know the shv5 kit, but I'd be worried about my clients.... you're in a tricky position.
Yep, Chris. You're right. I am in a tricky position. Frankly I've spent the last few hours looking HARD at this rootkit and what others have to say about it. It wouldn't be easy, but I've almost convinced myself it could be lobotomized if not totally exorcised.

Last edited by websissy; 11-02-2009 at 11:44 PM.
 
Old 11-02-2009, 11:59 PM   #10
websissy
Member
 
Registered: Jul 2008
Posts: 49

Original Poster
Rep: Reputation: 15
Quote:
Originally Posted by Wim Sturkenboom View Post
There is a sticky with security references on the main page of this subforum.

And I recently stumbled over 20 Linux Server Hardening Security Tips; it might be of help.
Thanks, Wim. This is a very useful set of tips. I've browsed through about 7 of them. Then I put it in my "read and study when I haven't been at the keyboard for 21 hours" pile!

Thanks again.
 
Old 11-03-2009, 12:13 AM   #11
websissy
Member
 
Registered: Jul 2008
Posts: 49

Original Poster
Rep: Reputation: 15
Quote:
Originally Posted by nigelc View Post
This is what google showed try this:

http://sourceforge.net/projects/rkhunter/
Thanks Nigel. I installed & ran rkhunter & chkrootkit this evening. They hinted at the existence of shv4 and confirmed the presence of shv5.

I then researched and read everything I could about the shv5 rootkit and found 2 or 3 others who claim to have successfully removed this rootkit without a server rebuild.

I then carefully studied it on my server and believe it probably can be manually defeated and removed. It has certain characteristics that make it "relatively easy" to identify, isolate and remove.

I also tried to determine the original date of the compromise but was unable to do that based on the dates of the files involved. They date back 2 or 3 years before my server even existed.
 
Old 11-03-2009, 12:41 AM   #12
websissy
Member
 
Registered: Jul 2008
Posts: 49

Original Poster
Rep: Reputation: 15
Quote:
Originally Posted by unSpawn View Post
... If this is really a root compromise involving the SHV5 rootkit then you'd also find references to libsh.so, ttymon and ttyload binaries, and the latter two also in some of the SysV startup files.

...

While you ponder your choices in the meanwhile save detailed process, open files, network connection and auth data off-site, then make the server unavailable by stopping public services, raising the firewall except for traffic from/to your management IP (or range), if the machine is remote you'll only need SSH.
For the record, I did confirm direct replacement or impacts on ls, ps, top, dir, ifconfig, ttymon, ttyload, netstat, pstree, lsof, find, md5sum and other system components.

Now can you please explain what you're referring to when you mention SysV startup files? That one went right over my head. Are you referring to changes in the /etc/init.d/ startup files? If so, I'd say there's no question THAT could have happened.

From what I saw, I'd say it's likely both the openssh and openssl packages on my server have also been compromised by this rootkit. In fact, I'm convinced shv5 contains files designed to replace the server's public (and private?) ssh and ssl key files. (cue ominous music...)

With THAT damage assessment in my head, I'm going to go sleep on this.

Thanks for the feedback, unSpawn. You gave me food for thought.

Last edited by websissy; 11-03-2009 at 10:38 AM.
 
Old 11-03-2009, 07:54 AM   #13
mase
LQ Newbie
 
Registered: Jul 2008
Posts: 22

Rep: Reputation: 15
Problem is: An attacker might have had access to a root shell on your server, that means he could have done anything.

And it is easier to just reinstall from fresh media
than trying to track down what might have been changed,
especially on a running server.
 
Old 11-03-2009, 08:05 AM   #14
Wim Sturkenboom
Senior Member
 
Registered: Jan 2005
Location: Roodepoort, South Africa
Distribution: Slackware 10.1/10.2/12, Ubuntu 12.04, Crunchbang Statler
Posts: 3,786

Rep: Reputation: 282Reputation: 282Reputation: 282
@mase
Easier yes, and the attacker will, with the same ease, get in again if you don't know how he/she got in.
 
Old 11-03-2009, 09:54 AM   #15
Jim Bengtson
Member
 
Registered: Feb 2009
Location: Iowa
Distribution: Ubuntu 9.10
Posts: 164

Rep: Reputation: 38
Quote:
For the record, I did confirm direct replacement or impacts on ls, ps, top, dir, ifconfig, ttymon, ttyload, netstat, pstree, lsof, slocate, find, md5sum and other system components.
And that's only the ones you know about. I don't think you have any alternative but a full wipe and reinstall. This time, after you do the install but before you open it for business, try running Tripwire (or commercial version here) to make a baseline record of what your server looks like (file names, sizes, etc.). Then you can take regular snapshots and compare the current data to the original data...any files that have been altered will be listed, and if you can't explain the discrepancy then you know there may be a hacker at work.
 
  


Reply


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off



Similar Threads
Thread Thread Starter Forum Replies Last Post
think i got hacked phatbastard Linux - Security 7 12-01-2007 01:36 PM
hacked? WRSpithead Linux - Security 2 08-30-2006 03:21 AM
Was my PC hacked? the-iguana Linux - Security 4 04-07-2006 08:38 AM
Have I been hacked? af_dave Linux - Security 3 07-14-2004 02:02 PM
HELP I think i got hacked spank Linux - Newbie 5 03-24-2004 08:59 AM

LinuxQuestions.org > Forums > Linux Forums > Linux - Security

All times are GMT -5. The time now is 12:13 PM.

Main Menu
Advertisement
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
Facebook: linuxquestions Google+: linuxquestions
Open Source Consulting | Domain Registration