LinuxQuestions.org
Visit Jeremy's Blog.
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - General
User Name
Password
Linux - General This Linux forum is for general Linux questions and discussion.
If it is Linux Related and doesn't seem to fit in any other forum then this is the place.

Notices


Reply
  Search this Thread
Old 04-15-2006, 02:19 PM   #1
jjlune
LQ Newbie
 
Registered: Apr 2006
Location: California
Posts: 8

Rep: Reputation: 0
EXT3-fs:group descriptors corrupted! (GNU/RH9)


Whole error message reads:
*************

Mounting root filesystem
EXT3-fs error (device sd(8,17)): ext3_check_descriptors: Block bitmap for group 0 not in group (block 33188)!
EXT3-fs: group descriptors corrupted!
mount: error 22 mounting ext3
pivotroot: pivot_root(/sysroot,/sysroot/initrd) failed: 2
umount /initrd/proc failed: 2
Freeing unused kernel memory: 156k freed
Kernel panic: No init found. Try passing init- option to kernel.

*************

It is our mailserver and the guy who installed it 3 yrs ago, closed the shop and long gone. As a TOTAL-newby to Linux, I've spent the Good Friday and night here at the office - going 33 hours since walked into the office and some...

I so far, was able to download RH9 iso files to make 3 installation CDs; tried many different ways to revive the ext3 fs with no avail - besides, most of the recommendations on web were at least 2 years old!
I am sure you Linux gurus out there have answers to my agonizing (at least to me) suffering by now.

Please, make me spend the Easter with my dear family, if you could.

Thank you very much and more in advance.

For your info:

Gnu/RH9 is installed on Compaq Proliant 5500 (i386) with multiple disks vary from 4.3Gb to 18Gb HDDs.

Mailserver: Kerio
 
Old 04-15-2006, 02:28 PM   #2
btmiller
Senior Member
 
Registered: May 2004
Location: In the DC 'burbs
Distribution: Arch, Scientific Linux, Debian, Ubuntu
Posts: 4,284

Rep: Reputation: 371Reputation: 371Reputation: 371Reputation: 371
What happens when you try to run fsck on the affected partition (using CD-1 in rescue mode)? Hopefully it should be able to fix the problem. If not, then try booting off a Knoppix CD and looking at your partitions (Knoppix is a Linux system bootable off of CD). With and luck, your mail is not stored on the root partition but on some other partition that is unaffected. In this case, I'd suggest backing up the mail and reinstalling (you might want to consider an OS upgrade -- RH9 is ancient). If the mail spools are on a different partition, backing up is not strictly necessary so long as you don't reformat that partition, but given the circumstances I'd strongly suggest it anyway.

If this stuff is mission critical, and it sounds like it might be, you should seriously consider hiring a Linux consultant if there's no one in house who knows how to deal with this system. It is possible to destroy all the data on the server with one false move. If you are not comfortable with this, please get someone to help.

Hope this helps a little.

Last edited by btmiller; 04-15-2006 at 02:30 PM.
 
Old 04-15-2006, 02:47 PM   #3
jjlune
LQ Newbie
 
Registered: Apr 2006
Location: California
Posts: 8

Original Poster
Rep: Reputation: 0
Thank you very much, btmiller.
and... Happy Easter!

This is what I get running "fsck":
WARNING: couldn't open /etc/fstab: No such file or directory.

Getting a Linux expert in house or a consultant is great suggestion - not that I haven't thought of it before... however, it definitely is to be seriously considered at later time.

But for now...
I have a DLT tape drive on that server. How can I back up the data onto that tape and restore from it?

And, how do I get that Knoppix CD or from where?

Thanks.
 
Old 04-15-2006, 02:56 PM   #4
btmiller
Senior Member
 
Registered: May 2004
Location: In the DC 'burbs
Distribution: Arch, Scientific Linux, Debian, Ubuntu
Posts: 4,284

Rep: Reputation: 371Reputation: 371Reputation: 371Reputation: 371
You'll have to tell fsck which partition you want it to check. This requires knowing which partition your root is. You can look at the GRUB bootloader on boot to find this out. When GRUB comes up, hit e (edit) and you will see the entirety of the boot command line, which should include an argument like root=/dev/hdXX or root=/dev/sdXX (the last two characters are variable based on exactly where the root partition is, hd is an IDE drive, sd is a SCSI). If you can find that, or it has something like root="LABEL=/" which is no help then you'll have to try and determine it another way. It looks from your error like the root partition is on a SCSI, and it may be the 18th partition of the 8th SCSI drive (sd(8,17) -- the first number 8 is the drive and it starts from 0 so 8 is the 9th, and the second the partition, also numbered from 0). The reason I say may be is your SCSI controller may number drives strangely. Do you know how many drives you have? Hopefully, though you'll just see root=/dev/hdXX or root=/dev/sdXX in the boot command line.

When you have the /dev/hdXX or /dev/sdXX this is the device name as Linux sees it. You can then do fsck /dev/hdXX or fsck /dev/sdXX (using the hdXX or sdXX string you got from above).

As for backing up, I'd suggest reading man dump -- the tape drive is probably /dev/rmt0 but you can check for this in the boot messages. You'll need to know some parameters about the tape drive and you'll have to give some arguments to the dump command (again, see the man page). When you make it this far, I or someone else can help a bit more).

Last edited by btmiller; 04-15-2006 at 02:59 PM.
 
Old 04-15-2006, 03:11 PM   #5
jjlune
LQ Newbie
 
Registered: Apr 2006
Location: California
Posts: 8

Original Poster
Rep: Reputation: 0
Right, these are the actual commands I've used:
fsck /
fsck /dev/sdb1
on both instances, I had the same reply as I've described above.

FYI, my GRUB argument just says 'root=LABEL=/'
Also, the server is equipped with:
1 - 4.3 Gb for system
1 - 9.1 Gb stand alone
2 - 9.1 Gb with linear raid (for the old email data)
3 - 18 Gb which I was trying to make a replacement data storage with linear raid

This happened after I plugged in those 3 drives (HotSwap Disks) and made an enty in /etc/raidtab file and reboot the server prior to partition them as a single logical drive.

I thought it was an easy task to do since RH9 even recognized those drives and gave device ids as sde, sdf and sdg right away - now, this statement sure can tell I am a newby, even to me. I should've consulted with a Linux guru...

So... is there a cure for such a humble newby like myself?

Thanks.
 
Old 04-15-2006, 03:17 PM   #6
jjlune
LQ Newbie
 
Registered: Apr 2006
Location: California
Posts: 8

Original Poster
Rep: Reputation: 0
Well... I even stumble with the mouse now...
I clicked on that "Post Quick Reply" button a bit too early.

Here are the bottom part of the message from 'fsck /dev/sdb1':

e2fsck 1.32 (09-Nov-2002)
Group descriptors look bad... trying backing blocks...
fsck.ext2: Invalid argument while checking ext3 journal for /

Thanks.
 
Old 04-15-2006, 03:27 PM   #7
btmiller
Senior Member
 
Registered: May 2004
Location: In the DC 'burbs
Distribution: Arch, Scientific Linux, Debian, Ubuntu
Posts: 4,284

Rep: Reputation: 371Reputation: 371Reputation: 371Reputation: 371
You'll need to try one of the back-up superblocks with the -b option. I think every ext3 filesystem will create a backup superblock at block 8192, so try this first. The other locations vary based on the size of your disk. You can use the mke2fs -n /dev/sdb1 command to print a list of where the others are (the -n command makes mke2fs "pretend" to create a new filesystem, but it doesn't actually do anything -- check the man page for details). It will, however, print out where it would put the back-up superblocks). Be very careful with the mke2fs command as not using the -n option will nuke the partition.
 
Old 04-15-2006, 03:40 PM   #8
jjlune
LQ Newbie
 
Registered: Apr 2006
Location: California
Posts: 8

Original Poster
Rep: Reputation: 0
I did NOT forget that -n option, thank you.

Here is the reply:

mke2fs 1.32 (09-Nov-2002)
Filesystem label=
OS type: Linux
Block size=4096 (log=2)
Fragment size=4096 (log=2)
981120 inodes, 1959922 blocks
97996 blocks (5.00%) reserved for the super user
First data block=0
60 block groups
32768 blocks per group, 32768 fragments per group
16352 inodes per group
Superblock backups stored on blocks:
32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632

How do I "try one of the back-up superblocks with the -b option"?
I mean, what is the command?

Thanks.
 
Old 04-15-2006, 04:26 PM   #9
jjlune
LQ Newbie
 
Registered: Apr 2006
Location: California
Posts: 8

Original Poster
Rep: Reputation: 0
Disregard about the superblock checks command.
It gave me 'invalid argument' error msg with 8192.
I figured it out (fsck -b 32768 /dev/sdb1) however,
the reply was same at all blocks:

fsck 1.32 (09-Nov-2002)
WARNING: couldn't open /etc/fstab: No such file or directory.
e2fsck 1.32 (09-nOV-2002)
fsck.ext2: Invalid argument while checking ext3 journal for /

It seems to me that the prior command (fsck /dev/sdb1) took care of checking the Superblocks already.

Is there anything I can do before I shoot the server?

Thanks.
 
Old 04-15-2006, 04:43 PM   #10
btmiller
Senior Member
 
Registered: May 2004
Location: In the DC 'burbs
Distribution: Arch, Scientific Linux, Debian, Ubuntu
Posts: 4,284

Rep: Reputation: 371Reputation: 371Reputation: 371Reputation: 371
The exact syntax is:

fsck.ext3 -b 163840 /dev/sdb1

Try that instead. I am using a different superblock from the list you pointed out. Let us know if that doesn't work (but it should) because there are a couple more tricks to try. I would also suggest getting a Knoppix CD as I suggested above and checking to see if all your other partitions (like your RAID-1 and the other holding mail spools) are intact.
 
Old 04-15-2006, 07:24 PM   #11
jjlune
LQ Newbie
 
Registered: Apr 2006
Location: California
Posts: 8

Original Poster
Rep: Reputation: 0
Sorry it took so long - I was on empty stomach for 36 hrs.

Tried. Failed. I think it's time for me to get my Berry 9mil.
What do you think?

Thanks.
 
Old 04-15-2006, 10:01 PM   #12
btmiller
Senior Member
 
Registered: May 2004
Location: In the DC 'burbs
Distribution: Arch, Scientific Linux, Debian, Ubuntu
Posts: 4,284

Rep: Reputation: 371Reputation: 371Reputation: 371Reputation: 371
Did it give the same error message or a different one? If none of the backup superblocks are usable, you can try running mke2fs -S /dev/sdb1 -- what this does is a partial reformatting of the filesystem, writing only the superblocks and group descriptors, but not the inode tables or data blocks. Because this does do a partial reformat, it is extremely risky and there is no guarantee that it will actually work. It might make things completely unrecoverable. I'd suggest reading the section in the man page before proceeding and deciding if this is really the route you want to take. I'd also really suggest getting that Knoppix disk and backing up your other partitions at this point.

Before proceeding, do you have any idea what caused this failure? Were there any recent storms or power outages where you were? It's possible that the sdb disk itself has failed irrecoverably, in which case you will need to replace it. Regardless of this, the data on the other disks should still be intact. That's why I'm suggesting you get Knoppix or another CD bootable distribution to have a look at them, especially before trying something as drastic as a mke2fs -S.

At this point, if you're not completely comfortable with this and you have mission critical data on that server, it's really time to start thinking about hiring a consultant with Linux experience who can do a hands on examination of the server. The consultant may be able to see something I can't see because I'm not physically at the machine and unable to see the complete set of boot-up diagnostics.

Last edited by btmiller; 04-15-2006 at 10:03 PM.
 
Old 04-16-2006, 03:13 AM   #13
jjlune
LQ Newbie
 
Registered: Apr 2006
Location: California
Posts: 8

Original Poster
Rep: Reputation: 0
First of all, I am overwhelmed on your sincere and thoughtful responses.

And yes, you're right. It is Mission Critical issue which I just couldn't
let the time slide by.

I end up purchasing Windows 2003 based Email Management software from Plesk
and finished installing. Just got few more users to add and I am done.
Purchasing on-line and installation, total time spent 3 hours.

However, I haven't given up on this Linux issue just yet.
I will definitely attack it when the storm's gone.

Again, thank you very much for your helpful assistance.
It really was an educational experience.

Have a Happy Easter.

p.s.: I surely will be back for the 2nd round.
 
Old 04-16-2006, 05:38 PM   #14
btmiller
Senior Member
 
Registered: May 2004
Location: In the DC 'burbs
Distribution: Arch, Scientific Linux, Debian, Ubuntu
Posts: 4,284

Rep: Reputation: 371Reputation: 371Reputation: 371Reputation: 371
No problem -- I'm very glad you got the problem resolved. I know too well how much fun it is when mission critical services go offline and you've got people breathing down your next to get it back.

Have a happy Easter yourself!
 
  


Reply


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off



Similar Threads
Thread Thread Starter Forum Replies Last Post
EXT3-fs: group descriptors corrupted dadriver Linux - Hardware 2 02-04-2006 04:16 AM
ext3 group descriptors corrupt humbletech99 Linux - General 5 12-02-2005 08:20 AM
Corrupted Ext3 Filesystem pyrosim Linux - General 2 07-23-2005 12:21 PM
EXt3-fs:group descriptors corrupted. anjanesh Fedora 0 11-15-2004 01:26 AM
usb.c: couldn't get all of config descriptors (RH9+kernel 2.4.25) xgdfalcon Linux - Hardware 0 03-09-2004 10:29 AM

LinuxQuestions.org > Forums > Linux Forums > Linux - General

All times are GMT -5. The time now is 07:21 AM.

Main Menu
Advertisement
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
Facebook: linuxquestions Google+: linuxquestions
Open Source Consulting | Domain Registration