LinuxQuestions.org
Download your favorite Linux distribution at LQ ISO.
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - Hardware
User Name
Password
Linux - Hardware This forum is for Hardware issues.
Having trouble installing a piece of hardware? Want to know if that peripheral is compatible with Linux?

Notices


Reply
  Search this Thread
Old 06-03-2008, 05:32 PM   #1
mottl3y
LQ Newbie
 
Registered: Jan 2007
Posts: 7

Rep: Reputation: 0
missing inode table [SOLVED]


Hey hey,

I'm running ClarkConnect v 4.1 Community. I have set up a RAID 5 array using mdadm across 3 SATA drives. I have a separate SATA drive for the CC install.

Recently we had some power issues (bad mains switch made the power flicker on and off a whole bunch), and unfortunately my RAID array will no longer mount up. My incredible analysis of the situation is that there is a problem with the filesystem. Due to my long history with Linux, it took me only 3 days to work out this much. Please note the sarcasm about a long history with linux.

CC fails to boot when it tries to mount the array, dropping to shell with the request for either CTRL+D to continue, or type root password to enter console.

In console I ran fsck to see what would happen. After manually running through a whole lot of repeated y/n questions, I ran fsck -y.

After the initial run with lots of questions, fsck has now dropped down to only one question, but it seems to be stuck in a loop:

Code:
 
fsck 1.40.8 (13-Mar-2008)
e2fsck 1.40.8 (13-Mar-2008)
fsck.ext3: Group descriptors look bad... trying backup blocks...
Inode table for group 384 is not in group.  (block 0)
WARNING: SEVERE DATA LOSS POSSIBLE.
Relocate? yes

/dev/md0 contains a file system with errors, check forced.
Pass 1: Checking inodes, blocks, and sizes
Error allocating 512 contiguous block(s) in block group 384 for inode table: Could not allocate block in ext2 filesystem
Restarting e2fsck from the beginning...
fsck.ext3: Group descriptors look bad... trying backup blocks...
Inode table for group 384 is not in group.  (block 0)
WARNING: SEVERE DATA LOSS POSSIBLE.
Relocate? yes
Running debugfs, and looking at the stats, it seems that the majority of the data is intact, there is just one inode table missing for group 384:

Code:
 
...
 Group 384: block bitmap at 12582963, inode bitmap at 12582986, inode table at 0
           28352 free blocks, 284 free inodes, 0 used directories
 Group 385: block bitmap at 12615680, inode bitmap at 12615681, inode table at 12615682
           0 free blocks, 0 free inodes, 0 used directories
...
I haven't been through and checked every single group (there are a couple of thousand), but that is the only one I've found with a dodgey inode table address.

Would it be possible to set the inode table address by taking the number in between the inode addresses for group 383 and group 385?

As it's a RAID array, can I find which drive that inode corresponds to, replace that drive and let mdadm rebuild the data?

I'm not really well versed on how the data is stored on the drives, so if i've incorrectly assumed that each inode is stored on a single drive then please correct me.

Unfortunately I don't have a large enough drive to backup the data somewhere, which is kinda why it's a RAID array. If necessary I can buy a 750G HDD, however I will still need to solve the problem of a messed up filesystem.

Sorry about the essay, I hope I've provided enough information, and look forward to any help you can give!
m.

Last edited by mottl3y; 06-14-2008 at 09:03 AM.
 
Old 06-03-2008, 05:55 PM   #2
jailbait
LQ Guru
 
Registered: Feb 2003
Location: Mineral, Virginia
Distribution: Debian 8
Posts: 7,877

Rep: Reputation: 318Reputation: 318Reputation: 318Reputation: 318
Quote:
Originally Posted by mottl3y View Post

Unfortunately I don't have a large enough drive to backup the data somewhere, which is kinda why it's a RAID array. If necessary I can buy a 750G HDD, however I will still need to solve the problem of a messed up filesystem.
The first thing to do is set up a backup system and back up what you have. Then you can try to fix your filesystem and run the risk that your recovery efforts might cause you to lose more data.

----------------------
Steve Stites
 
Old 06-06-2008, 07:35 AM   #3
mottl3y
LQ Newbie
 
Registered: Jan 2007
Posts: 7

Original Poster
Rep: Reputation: 0
while it's not the suggestion I was hoping for, it's certainly not the worst suggestion you could've given me (reformat and start again).

I've bought myself a 1TB hdd, installed, formatted and partitioned it (ext3).

After having a look around the internets for various ways to backup the raid array, I settled on the approach in http://www.linuxquestions.org/questi...opy+filesystem
Code:
rsync -avH mount_point/* other_mount_point/
which resulted in
Code:
ubuntu@ubuntu:/$ sudo rsync -avH /dev/md0/* /mnt/sda1/
building file list ... rsync: link_stat "/dev/md0/*" failed: Not a directory (20)
done

sent 29 bytes  received 20 bytes  98.00 bytes/sec
total size is 0  speedup is 0.00
rsync error: some files could not be transferred (code 23) at main.c(977) [sender=2.6.9]
before I go crashing my way through the backup process I thought I should check if there is a right way to do it.

I assume that rsync didnt work because the filesystem on md0 is corrupt, so does that mean I have to use a different command to copy all the data across?

Looking at md0 in GParted shows up two partitions:
/dev/mdop1 (ext 3, 298G)
unnallocated (unallocated, 298G)

Your help is really appreciated!
m.

Last edited by mottl3y; 06-06-2008 at 08:05 AM.
 
Old 06-06-2008, 03:03 PM   #4
jailbait
LQ Guru
 
Registered: Feb 2003
Location: Mineral, Virginia
Distribution: Debian 8
Posts: 7,877

Rep: Reputation: 318Reputation: 318Reputation: 318Reputation: 318
Quote:
Originally Posted by mottl3y View Post

I assume that rsync didnt work because the filesystem on md0 is corrupt, so does that mean I have to use a different command to copy all the data across?

Looking at md0 in GParted shows up two partitions:
/dev/mdop1 (ext 3, 298G)
unnallocated (unallocated, 298G)
Since your file system is corrupted and since you now have plenty of backup space I suggest that you create two backup partitions and try to backup 2 different ways. Since your new backup is installed on the same machine as your corrupt filesystem I suggest that you not use rsync as it is more complicated than local commands.

I suggest that you make one backup using dd (See the dd man pages.). dd will copy every block in your filesystem including the errors. It will copy disconnected inodes etc. The resulting copy is in no better shape than your original but it can be used as a starting point if you mess up the original even further.

For the second backup I suggest that you use the cp command with the -a option to preserve permissions, ownership, and timestamps. (See the cp man pages.) Like rsync, cp will probably copy some of your files and then fail. But those files that are successfully copied have been salvaged and you do so with no risk of further degridation of your original file system. At that point take a look at what you have salvaged and what is still stuck on the bad file system and plan out your next move.

If your next move is to try to play around with the bad inodes to try to salvage some garbage files then I suggest that you experiment with the partition you created with dd rather than the original corrupt filesystem.

----------------------
Steve Stites
 
Old 06-07-2008, 12:45 AM   #5
mottl3y
LQ Newbie
 
Registered: Jan 2007
Posts: 7

Original Poster
Rep: Reputation: 0
I got halfway through making the 2 separate backups when I realised that md0 is 2x290GB = 540GB, which means I can only fit one copy of md0 on the 1TB hdd.

Currently I have the 1TB hdd in two 500GB partitions: sda1 and sda2.

sda1 currently has most of md0 on it. i used dd as follows:

Code:
ubuntu@ubuntu:/$ sudo dd if=/dev/md0 of=/dev/sda1 bs=4096 conv=notrunc,noerror
dd: writing `/dev/sda1': No space left on device
122096001+0 records in
122096000+0 records out
500105217024 bytes (500 GB) copied, 8149.58 s, 61.4 MB/s
which made me realise i wouldn't fit 2 copies of md0 on sda.

i thought i'd give cp a go anyway, just to see what would happen. after a couple of attempts i used

Code:
ubuntu@ubuntu:/$ sudo cp -afv /dev/md0 /dev/sda2
`/dev/md0' -> `/dev/sda2'
now looking in GParted, it says both sda1 and sda2 have 215.99GB used.

i cannot open either of them in debugfs:
Code:
debugfs 1.40.8 (13-Mar-2008)
debugfs:  open /dev/sda1
/dev/sda1: Can't read an inode bitmap while reading inode bitmap
debugfs:  open /dev/sda2
/dev/sda2: Can't read an block bitmap while reading block bitmap
should i reformat sda into a single partition and use dd to copy md0 across?

once I have a good backup, what is the next step in actually recovering the filesystem?

Despite the troubles I'm currently having, as I learn more about linux I get more tempted to install it on my other computers.

Thanks for your help,
m.

Last edited by mottl3y; 06-07-2008 at 08:12 PM.
 
Old 06-09-2008, 10:06 PM   #6
mottl3y
LQ Newbie
 
Registered: Jan 2007
Posts: 7

Original Poster
Rep: Reputation: 0
i'm not sure if it's necessarily a bad thing, but when I try to open /dev/md0 in debugfs, it comes up with the same error as for /sda1 and /sda2.

previously i have managed to browse md0 and have seen that the directories seem to be in tact, so I'm not sure what i'm doing differently this time.

m.
 
Old 06-11-2008, 09:14 AM   #7
mottl3y
LQ Newbie
 
Registered: Jan 2007
Posts: 7

Original Poster
Rep: Reputation: 0
*bump*

Anyone have any suggestions on how to recover the data? I've tried using testdisk, which seems to realise the problem. However the changes are seemingly not saved.

Hope you can help!
m.
 
Old 06-11-2008, 06:12 PM   #8
mottl3y
LQ Newbie
 
Registered: Jan 2007
Posts: 7

Original Poster
Rep: Reputation: 0
I've reformated the 1TB hdd as a single partition, and used dd to copy all the data onto it.

Using debugfs, the only problems I can see is that group 383 and group 384 both have their inode table address incorrect.

I whipped out the calculator, and it seems you can recalculate where the inode tables should be pointing (each group increments by 32,768), so now all I need to know is how to change the inode table address.

m.
 
Old 06-14-2008, 09:01 AM   #9
mottl3y
LQ Newbie
 
Registered: Jan 2007
Posts: 7

Original Poster
Rep: Reputation: 0
so it only took me 2 weeks, but it looks like i've recovered all (or at least a large majority) of my data. hooray!

I used the rdump command in debugfs to copy the data to another drive.

Hope someone else finds this useful!

m.
 
  


Reply


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off



Similar Threads
Thread Thread Starter Forum Replies Last Post
DMraid table missing Edgeman Linux - Hardware 0 11-03-2007 01:46 PM
inode table location(UFS2) anwerreyaz *BSD 3 11-20-2006 06:50 AM
A dialog box saying " inode missing" !!!! m.parthiban Fedora 1 04-02-2006 12:46 AM
viewing the Inode table 0perat0r Linux - Newbie 1 07-20-2004 02:20 PM
HELP!!! to recover lost inode table moden Linux - General 3 11-26-2002 05:53 AM

LinuxQuestions.org > Forums > Linux Forums > Linux - Hardware

All times are GMT -5. The time now is 03:57 AM.

Main Menu
Advertisement
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
Open Source Consulting | Domain Registration