LinuxQuestions.org
Support LQ: Use code LQ3 and save $3 on Domain Registration
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - Security
User Name
Password
Linux - Security This forum is for all security related questions.
Questions, tips, system compromises, firewalls, etc. are all included here.

Notices

Reply
 
Search this Thread
Old 08-11-2003, 01:54 PM   #1
complus
Member
 
Registered: Aug 2003
Distribution: Red Hat 9
Posts: 76

Rep: Reputation: 15
Unhappy boot hangs after freeing kernel memory


Hi all. This is my first time posting to the forum ... I'm in a bit of a serious bind, and am not sure where to turn to for help. I tried contacting Linux, but apparently they don't offer support (not even support I can pay for) for RedHat 7.3 ...

My production server may have been victim of a DOS attack, but I'm not sure. This morning I could not access any of my secure pages. After doing some trouble shooting, I decided to try rebooting the server (probably a newbie mistake). As the system was going down this is the message that was displayed:

"FUCK: caught signal ll while manipulating kernel"

(in case that didn't display right, the first word is the 4-letter "F" word... I'm sure thats not supposed to be there ). So the system goes through its normal process but hangs at "Freeing kernel memory: 280k freed". The only thing that works at this poing is Ctrl-Alt-Del. I booted from the installation CD and typed in linux rescue at the boot prompt, the files get copied to /mnt/sysimage and i get to a shell prompt ... but I'm not really sure what to do from there. Any help, or a point in the direction to help would be GREATLY appreciated.
 
Old 08-11-2003, 03:29 PM   #2
codecruncher
Member
 
Registered: Aug 2003
Location: Austria
Distribution: Fedora Core release 2 (Tettnang)
Posts: 37

Rep: Reputation: 15
Man, I would copy the data off that sucker and re-install...

But be aware! That is just me! Maybe there is a solution...I don't think so......
 
Old 08-11-2003, 04:29 PM   #3
jailbait
Guru
 
Registered: Feb 2003
Location: Blue Ridge Mountain
Distribution: Debian Wheezy, Debian Jessie
Posts: 7,472

Rep: Reputation: 154Reputation: 154
It seems that you have been hacked. You cannot trust any binary in your system because you do not know where the malicious code is hidden. So you will probably have to disconnect from the network and reinstall like codecruncher says.


If you want to chase the intruder then backup your entire system now. After you straighten out the mess you can offer the evidence to the FBI here:

http://www1.ifccfbi.gov/index.asp
 
Old 08-11-2003, 04:48 PM   #4
complus
Member
 
Registered: Aug 2003
Distribution: Red Hat 9
Posts: 76

Original Poster
Rep: Reputation: 15
Question ... how can I back up the server if I can't get it to boot all the way? Is there something I can do when I use linux rescue and get to the shell prompt?

Thanks.
 
Old 08-11-2003, 06:29 PM   #5
jailbait
Guru
 
Registered: Feb 2003
Location: Blue Ridge Mountain
Distribution: Debian Wheezy, Debian Jessie
Posts: 7,472

Rep: Reputation: 154Reputation: 154
"how can I back up the server if I can't get it to boot all the way?"

You could use knoppix or tomsrbrt. Boot the CD or floppy, mount your / partition, and access your backup programs through the mountpoint.

"Is there something I can do when I use linux rescue and get to the shell prompt?"

I don't know what linux rescue is. If it is a bootable CD then mount your / partition, etc.
 
Old 08-11-2003, 08:17 PM   #6
unSpawn
Moderator
 
Registered: May 2001
Posts: 26,988
Blog Entries: 54

Rep: Reputation: 2743Reputation: 2743Reputation: 2743Reputation: 2743Reputation: 2743Reputation: 2743Reputation: 2743Reputation: 2743Reputation: 2743Reputation: 2743Reputation: 2743
Is there something I can do when I use linux rescue and get to the shell prompt?
Follow Jailbait's advice and boot your RH recue CD, tomsrtbt or some live CD. DO NOT TRY to load the "standard" rescue bootup choice: this will get you running the box in runlevel 1 and could well try to start whatever took over your box. This goes as well for accessing information on the mounted disks: DO NOT RUN ANYTHING from the disks. It may not infect or harm anything, but if you're trying to gather evidence you will destroy it.

If you have another server with spare space, and you want to try and get a grip on what has been going on, then make a network copy of the partitions with "dd" for later examination using netcat, ftp, or whatever. Do that before you mount the disks to do a backup. If you want help with trying to recreate the events that led to the (possible) compromise, post in the Linux - Security forum please.

If you have no spare box, but still want to get sort of an idea of what's on the box: make sure all regular partitions are mounted and issue
"find / >/mnt/fd0/find.log 2>/mnt/fd0/find.log.err ". This will just show a list of files on the system. /mnt/fd0 is where you mount a floppy when you mount a CD, cuz you can't save it there.
Now issue "find / | xargs -ix md5sum "'x'" >/mnt/fd0/find.md5 2>/mnt/fd0/find.md5.err". This will just show a list of files on the system again, but now with the md5sums.
Now do a quick search for a short string (or try "manipulating"): " grep /* -aHe "FUCK" >/mnt/fd0/string.log 2>/mnt/fd0/string.log.err"
If you have no intention to know whats been goin on, at least search for the string and report back those results if you will.
//Of course the above would be better run tru the Biatchux/FIRE forensics CD, but who know how to operate it?...

As for backing up stuff, please DO NOT backup binaries: only HUMAN READABLE stuff. Also include your shadowed password/group files (or just the whole of /etc), logfiles (or /var), u and wtmp and the rpm databases in the backup. Mark the backup as UNSAFE and put it somewhere safe where it cant be reused easily.

Please reformat the box before reinstalling, do not reuse passwords, and ask in the Linux - Security forum about proper security and hardening measures if unsure.


HTH
 
Old 08-12-2003, 09:13 AM   #7
complus
Member
 
Registered: Aug 2003
Distribution: Red Hat 9
Posts: 76

Original Poster
Rep: Reputation: 15
Thank you all for your posts.

This is what I am being told happened, by our Network admin ...a new worm got to our network, and from there it got to my server through a port that must have been open and it basically caused a DOS attack on my server. The worm is W32/Blaster worm and info on it can be found at http://www.cert.org/advisories/CA-2003-20.html. I'm not understanding how a worm that affects MSFT systems could have gotten to my Linux box, but I'm being told it is possible since they are on the same network and if the port was open. Seeing as I am a newbie, my install was a straight-out-of-the-box install, and it is very possible these ports were open by default.

I'm finding out that there are issues with our network, and the newbie mistake was to have my web server on the same network as everything else. We've hired a consultant to come in and perform analysis, restructuring of the network, etc...

unSpawn, I will do as you posted. I will also post whatever I find.

In the meantime, I've been told that the Linux OS should be "built securly" and not installed "out-of-the-box". Can anyone point me in the right direction to learn about this? Should I post in the Linux-security forum for that?

Thanks again.
 
Old 08-12-2003, 04:22 PM   #8
Mathieu
Senior Member
 
Registered: Feb 2001
Location: Montreal, Quebec, Canada
Distribution: RedHat, Fedora, CentOS, SUSE
Posts: 1,403

Rep: Reputation: 46
By default, Linux is secure.
What you need is a firewall that will protect you from the Microsoft Network.

You can take a look at the documentation and tutorials on
netfilter / iptables
http://www.netfilter.org/

Or if you want GUIs:
FireStarter
http://firestarter.sourceforge.net/
Shorewall
http://www.shorewall.net/

Securing and Optimizing Linux
http://en.tldp.org/LDP/solrhe/Securi...-Edition-v1.3/


If you need a rescue disk, use Tomsrtbt
http://www.toms.net/rb/

Last edited by Mathieu; 08-12-2003 at 04:24 PM.
 
Old 08-12-2003, 09:54 PM   #9
complus
Member
 
Registered: Aug 2003
Distribution: Red Hat 9
Posts: 76

Original Poster
Rep: Reputation: 15
Unfortunately, our network was attacked by the new worm that was in the news today - so I spent all day dealing with that and our clients and left my server dead in the water for now. I have my UNIX guy helping me out on this, and we are going to follow all of your suggestions first thing tomorrow, and I will certainly post any findings.

The question that my UNIX admin wants me to ask is in the "normal" sequence of events in the boot process of RH 7.3 professional, what would "normally" come after the following:

Freeing unused kernel memory: 280k freed.

He's convinced my server is not lost, and he said knowing what would be next will certainly help him troubleshoot whats going on.

Thanks again for all your posts.
 
Old 08-13-2003, 01:52 PM   #10
complus
Member
 
Registered: Aug 2003
Distribution: Red Hat 9
Posts: 76

Original Poster
Rep: Reputation: 15
Can someone please tell me what would be next in the normal sequence of boot events after:

Freed unused kernel memory: 280k freed

Thanks,
 
Old 08-13-2003, 02:19 PM   #11
Mathieu
Senior Member
 
Registered: Feb 2001
Location: Montreal, Quebec, Canada
Distribution: RedHat, Fedora, CentOS, SUSE
Posts: 1,403

Rep: Reputation: 46
I believe Linux executes /etc/rc.d/rc.sysinit
The rc.sysinit scripts initializes the services, mount points, modules, etc...
 
Old 08-13-2003, 02:25 PM   #12
complus
Member
 
Registered: Aug 2003
Distribution: Red Hat 9
Posts: 76

Original Poster
Rep: Reputation: 15
Thats great, thanks! So if that is next, then would it be safe to assume that the system hangs as its executing that script and that script could be causing the problem? Or is it more likely that the kernel is corrupt.

Thanks again.
 
Old 08-13-2003, 02:35 PM   #13
TheOneAndOnlySM
Member
 
Registered: Jul 2003
Location: Dallas, TX
Distribution: Ubuntu 10.04 LTS
Posts: 987

Rep: Reputation: 30
after that "freeing unused kernel memory" here's what i get (note that this is from an scsi slackware distro)

INIT verision 2.84 booting
scsi0 tagged queueing now active for target 0
Adding swap
/etc/rc.d/rc.S Testing file system status: Readonly
checking root file system
fsck......
remounting in read-write mode
::mounts other things::checks other systems::
INIT: entering runlevel 3

so ya, basically some stuff from /etc/rc.d init scripts get done
 
Old 08-13-2003, 02:36 PM   #14
TheOneAndOnlySM
Member
 
Registered: Jul 2003
Location: Dallas, TX
Distribution: Ubuntu 10.04 LTS
Posts: 987

Rep: Reputation: 30
well, the original post said something about messing with the kernel, so try compiling a new one (but do check out the init scripts and look for something suspicious)
 
Old 08-13-2003, 04:46 PM   #15
unSpawn
Moderator
 
Registered: May 2001
Posts: 26,988
Blog Entries: 54

Rep: Reputation: 2743Reputation: 2743Reputation: 2743Reputation: 2743Reputation: 2743Reputation: 2743Reputation: 2743Reputation: 2743Reputation: 2743Reputation: 2743Reputation: 2743
"By default, Linux is secure."
Untrue. Linux posesses only a small set of differences in architecture like privilege separation that help protect it when a default install is done. Unless a custom install, initial upgrades and proper hardening are done Linux isn't that much more secure "out of the box" compared to an install of any Pitiful Operating System (abbrev.: POS, aka "the MICROS~1 Game Platform").

"What you need is a firewall (...)"
A firewall can be regarded as a Single Point of Failure if no other restrictive features are introduced. Also a Netfilter-based firewall can't do packet inspection, which means it has no knowledge of packet contents so it won't detect malicious activity within allowed traffic channels.

The question that my UNIX admin wants me to ask is in the "normal" sequence of events in the boot process of RH 7.3 professional, what would "normally" come after (...)
At TLDP, read the "From Power Up To Bash Prompt" HOWTO. if you want gory kernel details read the "Linux 2.4.x Initialization for IA-32 HOWTO".
Smack your "UNIX admin" over the head real good. Relying on textual representation of what happens on bootup is as stupid as cleaning a gun and leaving a bullet in the chamber. Basic filesystem integrity checking and running chkrootkit would have been a better place to start at.

He's convinced my server is not lost, and he said knowing what would be next will certainly help him troubleshoot whats going on.
Ask him based on what evidence he came to that conclusion.

So if that is next, then would it be safe to assume that the system hangs as its executing that script and that script could be causing the problem? Or is it more likely that the kernel is corrupt.
You're about to take an answer from a different distro, booted under controlled (I hope) circumstances as a guideline for your situation?

Or is it more likely that the kernel is corrupt.
Code:
]$ cd /big/home/unspawn/kernel-2.4.21
]$ find . -type f | xargs -ix grep "x" -aHe "FUCK"
]$ find . -type f | xargs -ix grep "x" -aHe "manipulating kernel"
Ooooohhhh, weird innit? No results.... Tsk.

well, the original post said something about messing with the kernel, so try compiling a new one (but do check out the init scripts and look for something suspicious)
Hi-la-rious. Sorry.

Uh. No. Nooooooh. Can't be true.... Maybe...
Is it.... A ^%$#@& rootkit?
Lemme search my stash...
Code:
]$ gfind rk/ "while manipulating kernel"
./rk/sk-1.3b/src/main.c: printf("\nFUCK: Got signal %d while manipulating kernel!\n", ret & 0x7f);
Sk. Or SuckIT, more likely: "The SucKIT is easy-to-use, Linux-i386 kernel-based rootkit. The code stays in memory through /dev/kmem trick, without help of LKM support nor System.map or such things."


From LQ's FAQ: Security
references, see post #1: Compromise, breach of security, detection.
 
  


Reply


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off


Similar Threads
Thread Thread Starter Forum Replies Last Post
'Freeing Unused Kernel Memory: 224k Freed - Laptop Hangs nutnut Linux - Laptop and Netbook 1 10-21-2005 07:09 PM
'Freeing unused kernel memory' isn't solved yet VertX Linux - Software 2 02-19-2005 06:49 AM
Hang after Freeing unused Kernel memory.. junjem0702 Linux - Software 16 01-03-2005 07:12 PM
stops at freeing unused kernel memory rtr Linux - General 1 12-09-2003 10:42 AM
Hang after freeing unused kernel memory darkcloud Linux - General 9 07-07-2003 08:12 PM


All times are GMT -5. The time now is 10:35 AM.

Main Menu
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
identi.ca: @linuxquestions
Facebook: linuxquestions Google+: linuxquestions
Open Source Consulting | Domain Registration