This error magically persists across repartitioning and all other disk changes.

Mountain · 03-02-2009, 08:39 PM

How do I make this disappear?
"Incorrect metadata area header checksum"

I am willing to wipe out all data on the disk. In fact, I have deleted all partitions and started fresh several times, but I still get the error "Incorrect metadata area header checksum" when I run vgscan as soon as any new partition is put on the disk -- even non-LVM partitions.

I suspect using dd to zero out the whole disk might work, but I want a short cut... something faster than waiting a day or two for dd to zero a 1.5 TB disk.

Where is this LVM metadata persisted on the disk that enables it to reappear even after deleting all partitions?

mostlyharmless · 03-03-2009, 01:22 PM

Did you use dmraid at some point? You can use dmraid -E to erase metadata.

Mountain · 03-03-2009, 02:10 PM

Quote:

Originally Posted by mostlyharmless

Did you use dmraid at some point? You can use dmraid -E to erase metadata.

I did not use dmraid, but I did partition these disks in a wide variety of ways while I was trouble shooting a boot up problem. I put LVM volume groups and logical volumes on them in a variety of ways.

However, I also used LVM to remove everything (or so I thought).

I just do this:

Code:

dd if=/dev/zero of=/dev/sdX bs=446 count=1

maybe it will work?

mostlyharmless · 03-03-2009, 02:19 PM

Well, dd should work, but it is a blunderbuss to kill a mosquito, as they say.

dmraid metadata can also be written by on-board firmware raid controllers. You might have written some without knowing it if you were experimenting or got the disks used.

You could check for dmraid style metadata, (stored at the end of the disk and typically not erased by LVM or repartitioning), with dmraid -r

Mountain · 03-03-2009, 02:43 PM

Quote:

Originally Posted by mostlyharmless

Well, dd should work, but it is a blunderbuss to kill a mosquito, as they say.

dmraid metadata can also be written by on-board firmware raid controllers. You might have written some without knowing it if you were experimenting or got the disks used.

You could check for dmraid style metadata, (stored at the end of the disk and typically not erased by LVM or repartitioning), with dmraid -r

Thank you.

I'm pretty sure my problem is on sde or sde1, which was indeed part of a raid array on the Areca 1220 controller.

Here's the section from vgscan:

Code:

vgscan      /dev/sde: size is 1953525168 sectors
vgscan      /dev/sde1: size is 690732 sectors
vgscan      /dev/sde1: size is 690732 sectors
vgscan      /dev/sde1: lvm2 label detected
vgscan  Incorrect metadata area header checksum
vgscan      /dev/sdf: size is 1953525168 sectors
vgscan      /dev/sdf: size is 1953525168 sectors
vgscan      /dev/sdf: No label detected

I am assuming the "Incorrect metadata area header checksum" refers to the entries immediately above it (/dev/sde1).

But this isn't the result I expected here:

Code:

sudo dmraid -r /dev/sde1/
No RAID disks and with names: "/dev/sde1/"

Code:

sudo dmraid -r /dev/sde/
No RAID disks and with names: "/dev/sde/"

And I assume that's why using the -E arg returns an error:

Code:

 sudo dmraid -E /dev/sde1/
ERROR: option missing/invalid option combination with -E

Or am I doing something wrong?

UPDATE: maybe this helps:

Code:

o dmraid -r
No RAID disks

I appreciate the next suggestion.

Mountain · 03-04-2009, 09:55 AM

Detailed update:

Background on the problematic drive (and a spare):
Drive: sde

Code:

Disk /dev/sde: 1000.2 GB, 1000204886016 bytes
255 heads, 63 sectors/track, 121601 cylinders, total 1953525168 sectors
Units = sectors of 1 * 512 = 512 bytes
Disk identifier: 0x000377d3

Partition  Boot         Start           End          Size  Id System

/dev/sde1    *             63       690,794       690,732  83 Linux

Drive: sdf

Code:

Disk /dev/sdf: 1000.2 GB, 1000204886016 bytes
255 heads, 63 sectors/track, 121601 cylinders, total 1953525168 sectors
Units = sectors of 1 * 512 = 512 bytes
Disk identifier: 0x00000000

Partition  Boot         Start           End          Size  Id System

Steps
duplicate the boot partition (and the 62 sectors preceding it) to the other drive (sdf):

Code:

$ sudo dd bs=512 if=/dev/sde of=/dev/sdf count=690794

see if the problem was duplicated to /dev/sdf

Code:

vgscan      /dev/sde: size is 1953525168 sectors
vgscan      /dev/sde1: size is 690732 sectors
vgscan      /dev/sde1: size is 690732 sectors
vgscan      /dev/sde1: lvm2 label detected
vgscan  Incorrect metadata area header checksum
vgscan      /dev/sde2: size is 1952829270 sectors
vgscan      /dev/sde2: size is 1952829270 sectors
vgscan      /dev/sde2: No label detected
vgscan      /dev/sdf: size is 1953525168 sectors
vgscan      /dev/sdf1: size is 1953520002 sectors
vgscan      /dev/sdf1: size is 1953520002 sectors
vgscan      /dev/sdf1: lvm2 label detected
vgscan  Found duplicate PV xxxxxxxxxxxxxxxxxxx: using /dev/sdf1 not /dev/sde1
vgscan  Incorrect metadata area header checksum

Yes, the warning now exists on sdf1 too.

Code:

$ sudo tune2fs -U c1b9d5a2-f162-11cf-9ece-0020afc76f16 /dev/sdf1

This may not have been necessary...

Wanted to test this:

Code:

$ sudo dd bs=62 if=/dev/zero of=/dev/sdf count=1

Did not resolve the current issue.

Test pvremove on the backup drive:

Code:

$ sudo pvremove /dev/sdf1
$ sudo vgscan -vv

It works.

Do it for real.

Code:

$ sudo pvremove /dev/sde1

Check results:

Code:

vgscan      /dev/sde: size is 1953525168 sectors
vgscan      /dev/sde1: size is 690732 sectors
vgscan      /dev/sde1: size is 690732 sectors
vgscan      /dev/sde1: No label detected
vgscan      /dev/sde2: size is 1952829270 sectors
vgscan      /dev/sde2: size is 1952829270 sectors
vgscan      /dev/sde2: No label detected
vgscan      /dev/sdf: size is 1953525168 sectors
vgscan      /dev/sdf1: size is 1953520002 sectors
vgscan      /dev/sdf1: size is 1953520002 sectors
vgscan      /dev/sdf1: No label detected

The error is gone from both sde and sdf.

FYI => Grub0.97 is installed in the MBR of /dev/sde and looks on the same drive in partition #1 for /grub/stage2 and /grub/menu.lst.

But there are a few new problems:

Quote:

Unknown BootLoader on sde2
Unknown BootLoader on sdf2
hexdump: /dev/sdf2: No such file or directory
hexdump: /dev/sdf2: No such file or directory

See if I can fix sdf first:

Code:

$ sudo dd if=/dev/zero of=/dev/sdf bs=446 count=1

No change.

Code:

$ sudo dd if=/dev/zero of=/dev/sdf2 bs=1kB count=1

That worked.

Now, how to fix sde2?

Code:

$ sudo dd if=/dev/zero of=/dev/sde2 bs=1kB count=1

check:

Code:

$ sudo bash boot_info_script27.sh

Good. No errors. But will it boot? I don't expect a problem because I didn't mess with sde1, which is the boot partition. The MBR of sde is still in place and untouched as well. If I still have problems, I'll report back.

mostlyharmless · 03-04-2009, 12:53 PM

Very interesting, I'm glad you got it fixed, I'll try to remember this one the next time someone asks about a "metadata" error.

archtoad6 · 03-05-2009, 07:24 AM

Quote:

Originally Posted by Mountain

If I still have problems, I'll report back.

Actually, please report period. You've given too much to not close the issue. -- It would be nice to have you confirm that the solution works.

TIA

Mountain · 03-05-2009, 10:31 AM

Let me update by saying that yesterday I started as I had before and put new partitions on the disks, created new LVM physical volumes, groups and logical volumes, make encrypted volumes in those LVs, and created file systems and formatted them.

After doing all that (and at stages in between), I ran all the diagnostics I ran before. Now I am not getting any of the errors I saw previously. So I think that pretty much concludes this issue.

Previously, any time I deleted the partitions and then recreated them, the errors came back. I'm not seeing that now. I'm happy

archtoad6 · 03-07-2009, 01:50 PM

Any idea of what was different between before & now?

Mountain · 03-08-2009, 12:56 PM

Quote:

Originally Posted by archtoad6

Any idea of what was different between before & now?

Yes, this whole long thread could be boiled down to this:

Code:

sudo pvremove /dev/sdf1

I learned that removing all the partitions and even using

Code:

dd if=/dev/zero of=/dev/sdX bs=1k count=1

is not enough to remove the LVM metadata.

I do not know exactly where LVM stores this metadata, but it apparently isn't in the first 1k of the disk.

Most of what I did in this thread ended up being useful for me to learn and to verify my results, but in the future, if I get that error under similar circumstances, I will simply use

Code:

sudo pvremove /dev/sdX

Hopefully, this thread serves a useful supplement to the information that Google usually serves up when one enters the error message "Incorrect metadata area header checksum".

The info I found in sources such as the LVM How-To did not seem to apply to my situation (although in hindsight I can see the relationship). But I wasn't trying to recover physical volume metadata, I was trying to get rid of it.

EDIT: and the reason it was so important to me to remove this error message is that I am building a file server. I reasoned that if I couldn't start with an error-free storage state, it certainly did not make sense to begin putting all my important data on this server. So I wanted to understand and remove any and all errors I was seeing before putting this server to use.