Post hack investagation help...
 

Please bear with me while I fill in the background...

I noticed our web (RH7.2) server was generating quite a bit of traffic and started to investigate. I found it had been compromised via a LKM hack and a rootkit had been installed. Obviously no longer being able to trust the installed drive I installed RH9 on a spare drive and attempted to boot using this then mounting the 7.2 so I could do further investigation. The problem is that no matter how I try the 7.2 install boots and it is driving me bananas !

I have tried combinations of master/slave, IDE0 and IDE 1 (combinations of both - drives are a couple of Segate ATA's), I have tried grub and lilo loaders but anthough they "boot" off the RH9 install the kernal loaded is the 7.2. I think that maybe the rootkit may have played havok with the system but I don't understand how it could do this.

My final straw (not sure of my methodology here) was to rename the /boot on the 7.2 but hey presto it still loads.

Appreciate any help on this....

Thanks.

couple of questions:

did you build the RH 9 disk on the same machine w/ the old drive installed?

and

do you really need it for forensics?

First, (on a good machine) make a boot disk from rh9 and set its root device to be the rh9 disk. I've copied kernels to floppy and used rdev to change the root device, although there are sure to be other ways.

# cp vmlinuz /dev/fd0
# rdev /dev/fd0 /dev/hda999

I agree, you can't trust anything (including grub) from the hacked system. And your new rh9 install may be suspect now, if the old system has run with the rh9 partitions mounted.

If you really want to stop the RH7.2 kernel booting, in the RH7.2 boot directory, delete or rename the kernel to something else. If it's the one installed by redhat it's called something like vmlinux-2.4.etc.etc.

Course this doesn't achieve much if the RH7.2 root partition gets used - all the RH7.2 startups and applications are there.

The problem is that no matter how I try the 7.2 install boots and it is driving me bananas !
The 7.2 disk is your master, and most likely has the bootloader is in the MBR. Changing this would change the contents of the disk, which is something you DO NOT WANT when doing basic forensics...

I agree offered solutions should work, but Cyph3r7's first question should draw your attention first. In should be "easier" to detach the 7.2 disk, attach the 9.2 disk as master, install and reattach the 7.2 as readonly slave. If it was already part of the 7.2 box *before* the compromise, you should have backupped that one first, and don't forget to nuke it before installing.

If you're still unable to get 9.2 going have a look at the FIRE/Biatchux (http://biatchux.sourceforge.net/) forensics cd. Kinda cool, because now you can use the 9.2 disk (since you prolly didnt back it up before messing with it) as dump to hold the image of the 7.2 disk. Remember to work on A COPY of the 7.2 image, and if you don't keep logs of what you do to the image, at least try working under "script".

I found it had been compromised via a LKM hack and a rootkit had been installed.
If all of this didn't work you can boot the FIRE cdr or 9.2 cdr (in rescue mode) and you may be able to mount the 7.2 disk (readonly!) and list the contents. Just to satisfy my curiosity:
What anomalies have you found in the logs?
Are the contents of the autentication files changed?
If you run rpm in verify mode, what md5sums changed?
Which rootkit was installed? If unknown, what's the listing of visible files?
What LKM was installed? If unknown, what's the listing of visible files?

Some more info..

After the hack I plugged in the RH9 hdd and built it from scratch without the 7.2 connected. So the RH9 should think it is all by itself. I have tried the following combos.

RH9 ans master IDE0
with
RH7.2 as SLAVE IDE0
...............MASTER IDE1
...............SLAVE IDE1

I have fdisk'd the 7.2 to ensure the /boot partition in no active and even set the machine bios to boot from the 9 hdd - which it does. Then for some reason lilo swithes to the 7.2 disk and loads from there. I have tried both lilo and grub on the 9 hdd but it makes no diffrence.

I will try the solution posted by soob - thx.

If anyone is interested I will make my report to managment available. Let you know how it goes tomorrow.....

thx again

If anyone is interested I will make my report to managment available.
No, I'm NOT interested in a mgmnt report. I'm interested in your approach, method, tools used and all gory details of the compromise and forensics done.
[edit]
Ah, well, OK. If you can't post details, then of course a management report (shudder) is welcome...
[/edit]

touché

I in no way meant to insult the good people here by in any way inferring that the IQ of those here is anything remotley eqated to that of management.

Of course I will have to take out any jargon, technical detail and words with more that 3 syllable/initials which would therefore render it useless to those who frequent this site, but under previous stated level of the office food chain will make for some intense head nodding around the board room.

:D

Stay tuned....for the all the gore....

Any ETA?