|
boot hangs after freeing kernel memory
Hi all. This is my first time posting to the forum ... I'm in a bit of a serious bind, and am not sure where to turn to for help. I tried contacting Linux, but apparently they don't offer support (not even support I can pay for) for RedHat 7.3 ...
My production server may have been victim of a DOS attack, but I'm not sure. This morning I could not access any of my secure pages. After doing some trouble shooting, I decided to try rebooting the server (probably a newbie mistake). As the system was going down this is the message that was displayed: "FUCK: caught signal ll while manipulating kernel" (in case that didn't display right, the first word is the 4-letter "F" word... I'm sure thats not supposed to be there ). So the system goes through its normal process but hangs at "Freeing kernel memory: 280k freed". The only thing that works at this poing is Ctrl-Alt-Del. I booted from the installation CD and typed in linux rescue at the boot prompt, the files get copied to /mnt/sysimage and i get to a shell prompt ... but I'm not really sure what to do from there. Any help, or a point in the direction to help would be GREATLY appreciated. |
Man, I would copy the data off that sucker and re-install...
But be aware! That is just me! Maybe there is a solution...I don't think so...... |
It seems that you have been hacked. You cannot trust any binary in your system because you do not know where the malicious code is hidden. So you will probably have to disconnect from the network and reinstall like codecruncher says.
If you want to chase the intruder then backup your entire system now. After you straighten out the mess you can offer the evidence to the FBI here: http://www1.ifccfbi.gov/index.asp |
Question ... how can I back up the server if I can't get it to boot all the way? Is there something I can do when I use linux rescue and get to the shell prompt?
Thanks. |
"how can I back up the server if I can't get it to boot all the way?"
You could use knoppix or tomsrbrt. Boot the CD or floppy, mount your / partition, and access your backup programs through the mountpoint. "Is there something I can do when I use linux rescue and get to the shell prompt?" I don't know what linux rescue is. If it is a bootable CD then mount your / partition, etc. |
Is there something I can do when I use linux rescue and get to the shell prompt?
Follow Jailbait's advice and boot your RH recue CD, tomsrtbt or some live CD. DO NOT TRY to load the "standard" rescue bootup choice: this will get you running the box in runlevel 1 and could well try to start whatever took over your box. This goes as well for accessing information on the mounted disks: DO NOT RUN ANYTHING from the disks. It may not infect or harm anything, but if you're trying to gather evidence you will destroy it. If you have another server with spare space, and you want to try and get a grip on what has been going on, then make a network copy of the partitions with "dd" for later examination using netcat, ftp, or whatever. Do that before you mount the disks to do a backup. If you want help with trying to recreate the events that led to the (possible) compromise, post in the Linux - Security forum please. If you have no spare box, but still want to get sort of an idea of what's on the box: make sure all regular partitions are mounted and issue "find / >/mnt/fd0/find.log 2>/mnt/fd0/find.log.err ". This will just show a list of files on the system. /mnt/fd0 is where you mount a floppy when you mount a CD, cuz you can't save it there. Now issue "find / | xargs -ix md5sum "'x'" >/mnt/fd0/find.md5 2>/mnt/fd0/find.md5.err". This will just show a list of files on the system again, but now with the md5sums. Now do a quick search for a short string (or try "manipulating"): " grep /* -aHe "FUCK" >/mnt/fd0/string.log 2>/mnt/fd0/string.log.err" If you have no intention to know whats been goin on, at least search for the string and report back those results if you will. //Of course the above would be better run tru the Biatchux/FIRE forensics CD, but who know how to operate it?... As for backing up stuff, please DO NOT backup binaries: only HUMAN READABLE stuff. Also include your shadowed password/group files (or just the whole of /etc), logfiles (or /var), u and wtmp and the rpm databases in the backup. Mark the backup as UNSAFE and put it somewhere safe where it cant be reused easily. Please reformat the box before reinstalling, do not reuse passwords, and ask in the Linux - Security forum about proper security and hardening measures if unsure. HTH |
Thank you all for your posts.
This is what I am being told happened, by our Network admin ...a new worm got to our network, and from there it got to my server through a port that must have been open and it basically caused a DOS attack on my server. The worm is W32/Blaster worm and info on it can be found at http://www.cert.org/advisories/CA-2003-20.html. I'm not understanding how a worm that affects MSFT systems could have gotten to my Linux box, but I'm being told it is possible since they are on the same network and if the port was open. Seeing as I am a newbie, my install was a straight-out-of-the-box install, and it is very possible these ports were open by default. I'm finding out that there are issues with our network, and the newbie mistake was to have my web server on the same network as everything else. We've hired a consultant to come in and perform analysis, restructuring of the network, etc... unSpawn, I will do as you posted. I will also post whatever I find. In the meantime, I've been told that the Linux OS should be "built securly" and not installed "out-of-the-box". Can anyone point me in the right direction to learn about this? Should I post in the Linux-security forum for that? Thanks again. |
By default, Linux is secure.
What you need is a firewall that will protect you from the Microsoft Network. :D You can take a look at the documentation and tutorials on netfilter / iptables http://www.netfilter.org/ Or if you want GUIs: FireStarter http://firestarter.sourceforge.net/ Shorewall http://www.shorewall.net/ Securing and Optimizing Linux :study: http://en.tldp.org/LDP/solrhe/Securi...-Edition-v1.3/ If you need a rescue disk, use Tomsrtbt http://www.toms.net/rb/ |
Unfortunately, our network was attacked by the new worm that was in the news today - so I spent all day dealing with that and our clients and left my server dead in the water for now. I have my UNIX guy helping me out on this, and we are going to follow all of your suggestions first thing tomorrow, and I will certainly post any findings.
The question that my UNIX admin wants me to ask is in the "normal" sequence of events in the boot process of RH 7.3 professional, what would "normally" come after the following: Freeing unused kernel memory: 280k freed. He's convinced my server is not lost, and he said knowing what would be next will certainly help him troubleshoot whats going on. Thanks again for all your posts. |
Can someone please tell me what would be next in the normal sequence of boot events after:
Freed unused kernel memory: 280k freed Thanks, |
I believe Linux executes /etc/rc.d/rc.sysinit
The rc.sysinit scripts initializes the services, mount points, modules, etc... |
Thats great, thanks! So if that is next, then would it be safe to assume that the system hangs as its executing that script and that script could be causing the problem? Or is it more likely that the kernel is corrupt.
Thanks again. |
after that "freeing unused kernel memory" here's what i get (note that this is from an scsi slackware distro)
INIT verision 2.84 booting scsi0 tagged queueing now active for target 0 Adding swap /etc/rc.d/rc.S Testing file system status: Readonly checking root file system fsck...... remounting in read-write mode ::mounts other things::checks other systems:: INIT: entering runlevel 3 so ya, basically some stuff from /etc/rc.d init scripts get done |
well, the original post said something about messing with the kernel, so try compiling a new one (but do check out the init scripts and look for something suspicious)
|
"By default, Linux is secure."
Untrue. Linux posesses only a small set of differences in architecture like privilege separation that help protect it when a default install is done. Unless a custom install, initial upgrades and proper hardening are done Linux isn't that much more secure "out of the box" compared to an install of any Pitiful Operating System (abbrev.: POS, aka "the MICROS~1 Game Platform"). "What you need is a firewall (...)" A firewall can be regarded as a Single Point of Failure if no other restrictive features are introduced. Also a Netfilter-based firewall can't do packet inspection, which means it has no knowledge of packet contents so it won't detect malicious activity within allowed traffic channels. The question that my UNIX admin wants me to ask is in the "normal" sequence of events in the boot process of RH 7.3 professional, what would "normally" come after (...) At TLDP, read the "From Power Up To Bash Prompt" HOWTO. if you want gory kernel details read the "Linux 2.4.x Initialization for IA-32 HOWTO". Smack your "UNIX admin" over the head real good. Relying on textual representation of what happens on bootup is as stupid as cleaning a gun and leaving a bullet in the chamber. Basic filesystem integrity checking and running chkrootkit would have been a better place to start at. He's convinced my server is not lost, and he said knowing what would be next will certainly help him troubleshoot whats going on. Ask him based on what evidence he came to that conclusion. So if that is next, then would it be safe to assume that the system hangs as its executing that script and that script could be causing the problem? Or is it more likely that the kernel is corrupt. You're about to take an answer from a different distro, booted under controlled (I hope) circumstances as a guideline for your situation? Or is it more likely that the kernel is corrupt. Code:
]$ cd /big/home/unspawn/kernel-2.4.21well, the original post said something about messing with the kernel, so try compiling a new one (but do check out the init scripts and look for something suspicious) Hi-la-rious. Sorry. Uh. No. Nooooooh. Can't be true.... Maybe... Is it.... A ^%$#@& rootkit? Lemme search my stash... Code:
]$ gfind rk/ "while manipulating kernel"From LQ's FAQ: Security references, see post #1: Compromise, breach of security, detection. |
unSpawn, while I greatly appreciate your input, I don't get your sense of humour. Are you saying that you do not think its a corrupt kernel, and instead you think its a rootkit?
Well, I've searched for the "evidence" of a corrupt kernel, and the grep returned nothing. I ALSO tried checking for the rootkit, by running your command, and checking somethings suggested by a document I found from your post of security links... and it would "appear" that a rootkit has not been installed. I'm at a loss here... |
I don't get your sense of humour.
...just means you been properly desensitized. Cool by me. Are you saying that you do not think its a corrupt kernel, and instead you think its a rootkit? Until someone proves me otherwise it is a strong option for me. Well, I've searched for the "evidence" of a corrupt kernel, and the grep returned nothing. I ALSO tried checking for the rootkit, by running your command, and checking somethings suggested by a document I found from your post of security links... and it would "appear" that a rootkit has not been installed. May I ask what you ran the check from? Did you boot a cdrom of floppy, and not the kernel on the harddisk? Actually my commands contain an error, for instance where it says "find / " it should of course say "find /mnt/partition", I hope you saw that. While you're at it, run these two w/o outer quotes, both from Chkrootkit. Next one should return nothing: " strings (/mnt/partition!!!)/sbin/init | grep HOME " And this one should return "0": " strings -a /bin/login | grep -c "^root$" " If you scanned from a floppy or cdrom (that is not booting a kernel from the harddisk), and any irregular results show up, it's a good indication of a rootkit. |
Ok, let me please start from the beginning because I am thoroughly confused....Please forgive me cause my brain thinks in Windows....
At first I booted from a floppy boot disk and entered into rescue mode (this was before I saw the replies). From there I could get to a shell prompt. From here I could navigate around the system and saw all my files, but many commands were not available ( I couldn't even pull up a man entry), and I could not see my tape drive. What I did next was boot from a System Administrator CD that came with the RH software. THis allowed me to run linux single user mode. Here, I can see my tape drive, navigate around my file system, etc. I am currentlyhaving a problem restoring data from tape, but that is under a separate post). Actually my commands contain an error, for instance where it says "find / " it should of course say "find /mnt/partition", I hope you saw that. While you're at it, run these two w/o outer quotes, both from Chkrootkit. Now maybe what I'm not understanding is that there is apparently a difference between the root directory I am in at this point, and the actual root directory of the server?? (the mounted partition??) According to the info on the CD, my filesystem gets mounted to mnt/chroot. Here is where I am severely lost, and no I did not catch the mistake in your code because of this... which may be why I'm not getting any results on the following: back to the beginning... I tried the following: Code:
find / >/mnt/fd0/find.log 2>/mnt/fd0/find.log.err Now what I can say is that I am not booting from the kernel on the hard disk, because my system will not boot passed a certain point. Basically, I'm panicking because I'm running out of time, and I'm missing two important files in my existing backup (I did a backup a few weeks ago by pulling the files to my PC via SSH and put on zip disk). I guess what it comes down to is this: Is there anyway I can recover my system, or am I going to have to reinstall Linux? I really wanted to find out what happened, but I'm feeling the pressure from the big guys... Lastly, unspawn .. I'm definitely not desensitized, and I wasn't insulting your humor... I just was having a hard time understanding it ... i.e. you said Hi-la-ri-ous. Sorry" to recompiling the kernel ... I didn't understand if that meant "nope, that wouldn't work" or "bad suggestion" (and if so why - I'm a newbie remember :confused: ). I loved the comments on my UNIX admin, and might soon be taken into consideration... Thanks again.... |
Basically, I'm panicking because I'm running out of time, (...) Is there anyway I can recover my system, or am I going to have to reinstall Linux?
Let's get your priorities straight and cut the Gordian knot: you're driven by a lack of time, forced to rebuild the system. Your only option is to reformat the drives and install from scratch. This will make sure no harmfull remains are left or can be introduced back by restoring an untrusted backup. If you are forced to restore data, do not restore binaries. Install from scratch and only fill in gaps you have a chance at verifying. Never restore authentication like certificates or password databases. Do not connect restored system to the network until you are done verifying. If you have no external means of verification, like filesystem integrity database on readonly media (or a safe copy of rpm database if we're talking system binaries only) then in essence this slashes your whole backup sequence. *Whatever I wrote above ain't rules. For each corner you cut try to responsably weigh shortterm against longterm benefits, and use it as an argument to gain the necessary time. I really wanted to find out what happened, but I'm feeling the pressure from the big guys... Let's cut this one short too. I offered you to start with either dd'ing disk images to a safe place OR rummage tru the real data. You choose the last option. With that you lost your chance for doing proper forensics on the compromised the system. This does not mean you should not make a backup of th disks for forensics purposes, only the chance of a successfull outcome will be significantly lower. Let's try and recap this thread. - A production server malfunctioned. The only clue was a message saying "FUCK: caught signal ll while manipulating kernel". The server would not reboot properly. - The preliminary claim was this had anything to do with Msblaster.exe (which, apart from killing network services that cannot cope with unexpected input, could not have had any effect on the system). Disk images where not copied to another server. A first manual check of the filesystem did not yield any evidence. A suggestion was made this could be due to infection with the SuckIT rootkit. /bin/login and /sbin/init where not checked for proof so this remains inconclusive. No external verification methods where tried. - Preferred approach should be to detach box from network, boot removable media (rescue cd/fd) and perform initial scan of readonly mounted partitions with filesystem integrity scanner (Aide, Samhain tripwire), followed by scan with Chkrootkit. If unsure, dd out images of harddisks and images of partitions, start manual inspection of auth and log data. If no external verification methods exist and the distributions package management system does not hold an md5sum database, and knowngoods.org (or alike) server does not hold an md5sum database for the system, then manual inspection remains. (If network/server policies permit it, logging in/outbound traffic and deploying a sniffer are additional) If unsure, make list of events, offer young goat's intestines and try to summon/lure (local) N*X security expert from beyond the Styx. If positive, alert users of system and network, take appropriate actions to limit access to network, check other boxen and perform forensics on dd images. Rebuild server from scratch. Please see the CERT site or LQ Security references for more info. |
Hacked!
I'm getting the same message on a box that's been hacked. The source of the message is /sbin/init ... or more specifically, an altered version of /sbin/init.
It appears to be a badly written, or perhaps badly installed, rootkit. |
I'm getting the same message on a box that's been hacked.
Well, if you've read the thread you know the drill. |
| All times are GMT -5. The time now is 02:18 AM. |