grub not booting the correct device
 

Hi,

Well, first a bit of background:
I got an old HP DL380 G4 to run a minecraft server on, but I mostly got it to learn about linux server architecture, so don't tell me it's old and slow, because believe me, I know. First off, I tried installing Fedora 18, Centos 6.4, archlinux, and Ubuntu. Fedora, Centos, and Ubuntu couldn't resolve the display driver properly and the monitor kicked an error which I spent several days trying to fix. With Archlinux, I was able to boot to the disk prompt, and from there I could install the system and chroot in. However, when I tried to boot the the drive itself, I got the monitor error again. Finally I tried an older version of CentOS (5.9), and that worked all the way through the graphical install with one important caveat. When I had installed Arch, I used GPT for the partition tables, but CentOS 5.9 doesn't support that, so I had to go and change it to MBR. After CentOS was installed, I booted the system, and got a
So, eventually, I figured out how to see and change the boot device from the rescue prompt. the set command returned that grub was looking for (hd0, gpt1) as the root device, but I don't think that's valid. My partitions are like so:
#1 4GB Swap /dev/cciss/c0d0p1
#2 10GB ext3 /dev/cciss/c0d0p2 (this is where the grub files are)
#3 340GB ext3 /dev/cciss/c0d0p3 (This is where /boot, /home, /var etc. are) Also this is flagged as the boot device.
In /grub/device.map it shows hda0 /dev/cciss/c0d0, which is the main raid controller device. I tried to set the root partition from the grub rescue prompt with set root=(hda0,2) but it was not persistent through a reboot. It just went back to (hda0, gpt1). It seems like grub is leftover from the arch install when the partition table was GPT, and it hasn't changed the device map. I tried from the arch disk prompt, but I got an error (something like cannot create file x, sorry I don't remember the actual message). Any idea how I might re-install grub to get it to see the proper partitions for boot?

How did you do that ?.
Let's see (as root) the output from "parted -l" (use Arch, not the old CentOS).

Boot from rescue disk and chroot into installation.
Double check your parttion schema to detect /boot.
Check that /boot have kernel, initrd, grub, grub/*stage*, grub/grub.conf

Use (hd1,0) if your /boot is first partition on cciss, (hd1,1) if second; see below
Now update grub record like:
# grub
grub> device (hd1) /dev/cciss/c0d0

grub> root (hd1,0)
Filesystem type is ext2fs, partition type 0x83

grub> setup (hd1)
Checking if "/boot/grub/stage1" exists... no
Checking if "/grub/stage1" exists... yes
Checking if "/grub/stage2" exists... yes
Checking if "/grub/e2fs_stage1_5" exists... yes
Running "embed /grub/e2fs_stage1_5 (hd1)"... failed (this is not fatal)
Running "embed /grub/e2fs_stage1_5 (hd1,0)"... failed (this is not fatal)
Running "install /grub/stage1 (hd1) /grub/stage2 p /grub/grub.conf "... succeeded
Done.

grub> quit

Ha !
Just paid attention, you have first SWAP partition 4G.
Grub cannot jump to partition above 2G from disk start.

You have to remove first swap partition, create small /boot instead, copy all content to it,
then follow my previous post.

As you've create 10G /boot, looks like you have a lot free space.

Just format your first partition and use it instead /boot
/boot rarely runs out of 100m.

Once boot will be successfull, format secong 10g partition as swap.

Well, I re-partitioned the drive with parted like so:

p1 /boot 4GB exf3 /dev/cciss/c0d0p1
p2 10GB swap /dev/cciss/c0d0p2
p3 / 350GB exf3 /dev/cciss/c0d0p3

Then I re-installed CentOS 5.9 to p3 and selected "install grub on first sector of /dev/cciss/c0d0p1". Am I correct in thinking that the grub files should be on the /boot partition? Anyway, after doing this, I booted and the partition error is gone, but it just sits at "Grub Loading" forever. I also tried installing CentOS with the default option of installing grub on /dev/sda, but no dice. Same grub loading screen. I can't chroot (arch-chroot) into p3 (mounted on /mnt) because it says /mnt/run doesn't exist. I tried grub-install on /dev/cciss/c0d0p1, but it says Any ideas what might be the issue now? Also I should mention that I can't figure out how to get to the grub prompt to execute the setup command like you mentioned. There's no "grub" command, if that's what you mean.

at the grub prompt what is the output of set,set will show what partition grub is looking for /boot/grub on and the root partition among other things. ls will list the partitions present and ls /, ls /boot, and ls /boot/grub will display what is in the directories of the partition that grub has for root.

Ahh, but therein lies the problem. I never get to a grub prompt, it hangs at "grub loading". Is there a way to get the grub prompt from the arch DVD command line? I can't chroot into the system because there's no /mnt/run mount point (with the CentOS drive being mounted at /mnt), so I can't actually get to a local system terminal. How is the grub prompt normally reached?

create a new directory mount the boot partition to the new directory then run
grub-install --root-directory=<new directory>

Well, I tried "grub-install --root-directory=/mnt/boot" (That's where I mounted partition 1), but I got an error "ext2 file system does not support embedding". So, I tried it with the --force option and it seemed to work, but the boot sequence still stops at "grub loading".

I think you need to install to mbr, instead of first partition.

Sorry for my ignorance, but how exactly would I target MBR from grub-install?

It may be /dev/cciss I don't use raid.

Ok, so I did what you said:
After I did that, the system boots into the grub prompt. How do you boot to linux from the prompt? Also, since there's only one install, why would grub not boot to it automatically?

My recepie (one of first replies) was relevant for inital situation, when you had rescued
chrooted environment. Then in this environment, you have command grub.

Now, you reinstall over and over, and have no even rescue system.

I had no problem with G4 and RH4 and RH5 on it. I think RH6 will run also.
So CentOS6, probably, good.

Make fresh install, choose text mode, accept wipe all disks and default layout.
Everything should work.

find out what kernel and and initrd with . may have to change the partition that grub is looking at if your kernels aren't located on your boot partition. ls will give you the root device names. Then you can also use ls to find a /boot/grub/grub.cfg file if one exist. if it does then you can load it with this
then run update-grub to create a /boot/grub/grub.cfg file and/or copy the one from you root partition to your boot partition.

It's like this machine is possessed, I went and re-installed the OS and selected "delete partitions and set up default scheme", and it seemed to work. However, when I rebooted , it gave this error: Any idea what the issue might be now?

Looks like grub didn't install correctly. Try installing grub from install disk again.

Ok, so I re-installed grub, but now it's just back to the grub prompt, no default OS. I can boot using the linux, initrd, and boot commands, which I suppose is OK, but since there's only one OS, it should boot CentOS automatically. I looked at the menu.lst file, and everything looks fine; kernel and initrd are specified correctly and the default=0 line is there. At this point it's not a huge issue, as I won't be turning it off very often, but I would like to be able to start right into the OS.

Do you install grub legacy? If I'm not mistaken grub legacy won't do uefi gpt parititions. You need grub2, which what was installed before you reinstalled the Os.

To be honest I'm not completely sure, it's never said grub2, it's always said just grub. However, it's my understanding that legacy used menu.lst, and grub2 uses grub.cfg. Both files are there, so if I had to guess I would say it's the new one with a linked menu.lst for compatibility. But you're the expert, so if I do have grub legacy installed, how would I update it to grub2?

EDIT: I installed grub like so:
EDIT 2:

When I run
I get this:
So it looks like I've got grub legacy installed. How do I update?

If your using linux at the grub prompt for your kernel then you are using grub2. Look in your /boot and see if you have a efi directory. If so go down the /boot/efi/??/?? until you find grubx64.efi. Copy /boot/grub/grub.cfg to this directory and see if this fixes the grub menu problem.

I think it's grub legacy, it doesn't seem to recognize normal linux commands. The only ones I've used are these:
Also, there's no EFI directory in /boot.

If this is working then your using grub2. kernel /vmlinuz-2.6.18-348.el5PAE is what you would be using if it was grub legacy.

Did you try and copy the grub.cfg file to the /boot/efi directory? If it doesn't exist mount your first partition, it shoud be a fat32 partition and then see if you can find the directory with grubx64.efi in it. That is where you want to copy your grub.cfg to.

Well, I mounted /dev/cciss/c0d0p1 to /mnt, but still no dice. There's no efi directory, and there's no grubx64.efi anywhere. I'm labeling this thread as solved, because at this point it's a minor annoyance rather than a real issue. Anyway, thanks for all the help, if you have any more ideas about the grub issue please post.

I thought we had done this, but anyways download and run bootinfoscript and post the results.
http://bootinfoscript.sourceforge.net/

Ok, so here are the results. To my unpracticed eye there seems to be less data here than there should be.

You don't have any more info because of mounting problems. However from what I can tell if you have a /boot/grub files also located here Then maybe copy the grub.cfg file from your will fix the problem. Using at the grub command prompt should tell you what hard drive/partition that grub is looking for grub files and therefore grub.cfg. Look for the line that starts with prefix. Another thought is does a /boot/grub/grub.cfg even exist, if not you may need to run update-grub or create one by hand.

Ok, so I mounted /dev/mapper/VolGroup00-LogVol00 and there is a /boot directory, but the boot folder is empty. Also, there is no grub.cfg in /boot/grub of c0d0p1, but there is a grub.conf, which might fulfill the same function. at the grub prompt says that it's booting from /dev/cciss/c0d0p1.

what does grub.conf look like, if it had then it is for grub legacy. If it had then it is grub2. grub2 looks for grub.cfg not grub.conf

Yep it's like so:
So, how do I update to grub2?

This should create a grub.cfg and put it in the right location.
Not sure which grub update-grub will try and use.

Sorry repeating advise.

Since you reinstall with default layout, it is follows:
First partition is mounted as /boot
Second partition is LVM; It may include more than one LV, but all of them resides
on same second partition hidden by LVM.

Look on my very first advise, do copy-paste, and be happy

Ok, so I finally got it working. I decided to re-install CentOS because I had installed 5.9 i386, but because it was 32 bit, it wouldn't let me allocate more than 2GB of memory to my minecraft server, so I downloaded and burned a liveDVD of CentOS 5.9 x86_64 (6.4 gives a blank screen at the gui installer). I installed the new one, but this time, I switched the drive order so that c0d0 was first, and lo and behold there was an option to install grub on the MBR. So I selected it, and everything went fine, grub now automatically loads CentOS. Anyway, if anyone is wondering, I think the issue is that I originally installed grub from the liveDVD, so it's system analysis incorrectly detected the DVD as the main OS, and when that wasn't there at boot, it just went to a grub prompt. Thanks for all the assistance, this seems like a very helpful forum.