unSpawn, while I greatly appreciate your input, I don't get your sense of humour. Are you saying that you do not think its a corrupt kernel, and instead you think its a rootkit?

Well, I've searched for the "evidence" of a corrupt kernel, and the grep returned nothing. I ALSO tried checking for the rootkit, by running your command, and checking somethings suggested by a document I found from your post of security links... and it would "appear" that a rootkit has not been installed.

I'm at a loss here...

I don't get your sense of humour.
...just means you been properly desensitized. Cool by me.

Are you saying that you do not think its a corrupt kernel, and instead you think its a rootkit?
Until someone proves me otherwise it is a strong option for me.

Well, I've searched for the "evidence" of a corrupt kernel, and the grep returned nothing. I ALSO tried checking for the rootkit, by running your command, and checking somethings suggested by a document I found from your post of security links... and it would "appear" that a rootkit has not been installed.

May I ask what you ran the check from? Did you boot a cdrom of floppy, and not the kernel on the harddisk?
Actually my commands contain an error, for instance where it says "find / " it should of course say "find /mnt/partition", I hope you saw that. While you're at it, run these two w/o outer quotes, both from Chkrootkit. Next one should return nothing:
" strings (/mnt/partition!!!)/sbin/init | grep HOME "
And this one should return "0":
" strings -a /bin/login | grep -c "^root$" "

If you scanned from a floppy or cdrom (that is not booting a kernel from the harddisk), and any irregular results show up, it's a good indication of a rootkit.

Ok, let me please start from the beginning because I am thoroughly confused....Please forgive me cause my brain thinks in Windows....

At first I booted from a floppy boot disk and entered into rescue mode (this was before I saw the replies). From there I could get to a shell prompt. From here I could navigate around the system and saw all my files, but many commands were not available ( I couldn't even pull up a man entry), and I could not see my tape drive.

What I did next was boot from a System Administrator CD that came with the RH software. THis allowed me to run linux single user mode. Here, I can see my tape drive, navigate around my file system, etc. I am currentlyhaving a problem restoring data from tape, but that is under a separate post).

Actually my commands contain an error, for instance where it says "find / " it should of course say "find /mnt/partition", I hope you saw that. While you're at it, run these two w/o outer quotes, both from Chkrootkit.

Now maybe what I'm not understanding is that there is apparently a difference between the root directory I am in at this point, and the actual root directory of the server?? (the mounted partition??) According to the info on the CD, my filesystem gets mounted to mnt/chroot. Here is where I am severely lost, and no I did not catch the mistake in your code because of this... which may be why I'm not getting any results on the following:

back to the beginning... I tried the following:

This returned nothing. I tried to run: gfind rk/ "while manipulating kernel" but my system is not finding the gfind tool. Then I tried to run a few commands (i.e. netstat, ps, ...) with the -/ option which I read in the documentation would be a sure sign of a rootkit attack. I've tried the greps on "FUCK" and "while manipulating kernel" and all are not returning anything.

Now what I can say is that I am not booting from the kernel on the hard disk, because my system will not boot passed a certain point.

Basically, I'm panicking because I'm running out of time, and I'm missing two important files in my existing backup (I did a backup a few weeks ago by pulling the files to my PC via SSH and put on zip disk). I guess what it comes down to is this:

Is there anyway I can recover my system, or am I going to have to reinstall Linux? I really wanted to find out what happened, but I'm feeling the pressure from the big guys...

Lastly, unspawn .. I'm definitely not desensitized, and I wasn't insulting your humor... I just was having a hard time understanding it ... i.e. you said Hi-la-ri-ous. Sorry" to recompiling the kernel ... I didn't understand if that meant "nope, that wouldn't work" or "bad suggestion" (and if so why - I'm a newbie remember :confused: ). I loved the comments on my UNIX admin, and might soon be taken into consideration...

Thanks again....

Basically, I'm panicking because I'm running out of time, (...) Is there anyway I can recover my system, or am I going to have to reinstall Linux?
Let's get your priorities straight and cut the Gordian knot: you're driven by a lack of time, forced to rebuild the system. Your only option is to reformat the drives and install from scratch. This will make sure no harmfull remains are left or can be introduced back by restoring an untrusted backup.

If you are forced to restore data, do not restore binaries. Install from scratch and only fill in gaps you have a chance at verifying. Never restore authentication like certificates or password databases. Do not connect restored system to the network until you are done verifying. If you have no external means of verification, like filesystem integrity database on readonly media (or a safe copy of rpm database if we're talking system binaries only) then in essence this slashes your whole backup sequence.
*Whatever I wrote above ain't rules. For each corner you cut try to responsably weigh shortterm against longterm benefits, and use it as an argument to gain the necessary time.


I really wanted to find out what happened, but I'm feeling the pressure from the big guys...
Let's cut this one short too. I offered you to start with either dd'ing disk images to a safe place OR rummage tru the real data. You choose the last option. With that you lost your chance for doing proper forensics on the compromised the system. This does not mean you should not make a backup of th disks for forensics purposes, only the chance of a successfull outcome will be significantly lower.


Let's try and recap this thread.
- A production server malfunctioned. The only clue was a message saying "FUCK: caught signal ll while manipulating kernel". The server would not reboot properly.

- The preliminary claim was this had anything to do with Msblaster.exe (which, apart from killing network services that cannot cope with unexpected input, could not have had any effect on the system). Disk images where not copied to another server. A first manual check of the filesystem did not yield any evidence.
A suggestion was made this could be due to infection with the SuckIT rootkit. /bin/login and /sbin/init where not checked for proof so this remains inconclusive. No external verification methods where tried.

- Preferred approach should be to detach box from network, boot removable media (rescue cd/fd) and perform initial scan of readonly mounted partitions with filesystem integrity scanner (Aide, Samhain tripwire), followed by scan with Chkrootkit. If unsure, dd out images of harddisks and images of partitions, start manual inspection of auth and log data.

If no external verification methods exist and the distributions package management system does not hold an md5sum database, and knowngoods.org (or alike) server does not hold an md5sum database for the system, then manual inspection remains. (If network/server policies permit it, logging in/outbound traffic and deploying a sniffer are additional) If unsure, make list of events, offer young goat's intestines and try to summon/lure (local) N*X security expert from beyond the Styx.

If positive, alert users of system and network, take appropriate actions to limit access to network, check other boxen and perform forensics on dd images. Rebuild server from scratch.

Please see the CERT site or LQ Security references for more info.

Hacked!
 

I'm getting the same message on a box that's been hacked. The source of the message is /sbin/init ... or more specifically, an altered version of /sbin/init.

It appears to be a badly written, or perhaps badly installed, rootkit.

I'm getting the same message on a box that's been hacked.
Well, if you've read the thread you know the drill.