So this
only happens if the second eth is plugged in there? Without it it's OK? If this is happening, they (disk controller and ethernet card) may not getting set the correct IRQ's. You can set them, as in this example.
Code:
alias eth0 3c501
alias eth1 3c501
options eth0 -o 3c501-0 io=0x280 irq=5
options eth1 -o 3c501-1 io=0x300 irq=7
See sections 2.4 and 2.5 of the Ethernet-HOWTO (/usr/doc/Linux-HOWTOs on my system, just the bundle off
www.tldp.org ), and man 7 bootparam to find out about setting card options specifically. Usually the kernel will handle this itself, but this is one possibility that came to mind about what may be wrong.
If this is happening regardless of whether the card is in or out now, you may be lacking support for your disk controller in-kernel, which you should compile in. There's a few posts around here about compiling in disk controller support.
I don't use grub, but setting a root device is just
root=/dev/hda1 ro (for example) on the kernel parm line. Lilo's is like
root=/dev/hda1 also, you can even use
rdev bzImage /dev/whatever and don't pass a root= line. Probably grub's man page will tell its config option for the root device.