LinuxQuestions.org
LinuxAnswers - the LQ Linux tutorial section.
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - Software
User Name
Password
Linux - Software This forum is for Software issues.
Having a problem installing a new program? Want to know which application is best for the job? Post your question in this forum.

Notices

Reply
 
Search this Thread
Old 10-16-2006, 03:29 AM   #1
Chojin
LQ Newbie
 
Registered: Dec 2005
Posts: 5

Rep: Reputation: 0
RAID1+LVM: data keeps getting corrupted


After a recent HD failure, I decided to start using (software)RAID-1 (md). But I only had one disk, but I would soon have another (smaller disk) available after I migrated my fileserver to a (software)RAID-1 using two new disks.
So I created a partition on my first disk of the size as the second disk was going to be and configured it as a degraded RAID-1. On that RAID configuration I created a LVM2 volume group on which I created 2 LVM2 volumes (home and data). That worked perfectly for a few months. A few days ago, I added that other disk to the currently degraded (software)RAID-1. It started updating and the (software)RAID became clean. Everything seemed alright. Until somewhat later, I found out my home partition was suddenly remounted read-only because of some troubles with the ext3 journaling. After an fsck it turned I had had a lot of errors on that partition, lots of inodes I had to clean or fix.. :-(
But the system came up again, with no more errors.. Until, while deleting a few big files from the data partition: read-only filesystem. In the logging again the ext3-journaling who gave up and remounted the partition read-only. Again lots of data corruption and a lot of files lost..
After another reboot, everything seemed ok again. But after a while again the home partition read-only.
Both disks never gave any problem before, and the problems started since I hot-added one disk to the initialy degraded (software)raid-1. There is no message about DriveReadySeek errors or anything alike. It's always the ext-3 journaling system that seems to find something wrong causing the drive to be remounted read-only. No other errors in the logs which could point to any hardware failure.
I decided to remove that other disk again from raid, since problems started with that disk.
But even after the removal of it, the corruption keeps going on on the LVM2 volumes.

What could have gone wrong? And how should I fix this?

Last edited by Chojin; 10-16-2006 at 03:48 AM.
 
Old 10-17-2006, 10:41 AM   #2
Chojin
LQ Newbie
 
Registered: Dec 2005
Posts: 5

Original Poster
Rep: Reputation: 0
Anyone?

Do I have to conclude that I should not put an LVM2 on a software RAID(1)?

In the meanwhile, the data corruption keeps going on..however it seems that without the second disk, fsck is now always able to repair filesystem errors on his own, so no fatal errors anymore, but every few hours my partitions get remounted read-only, and I have to fsck them to use them again...

But this isn't such a special configuration, is it? It should work without problems, not? What could have gone wrong? what do I have to mind when I start to recreate the RAID/LVM configuration? Or should I try EVMS?

Am I going to face the same problems with my fileserver, when a disk crashes -> the raid gets degraded -> I add a new disk to replace the old wone -> data corruption?

Last edited by Chojin; 10-17-2006 at 10:46 AM.
 
Old 10-17-2006, 11:00 AM   #3
lazlow
Senior Member
 
Registered: Jan 2006
Posts: 4,362

Rep: Reputation: 171Reputation: 171
The only thing that comes to mind would be a stick of memory having intermittent problems. I had one go on an older system. It would run memtest fine for 1/2 an hour but if I left it run for a few hours the problem would show up. Stuff like that is a real PITA to diagnose.

Lazlow
 
Old 10-22-2006, 07:18 AM   #4
Chojin
LQ Newbie
 
Registered: Dec 2005
Posts: 5

Original Poster
Rep: Reputation: 0
As suggested I tried the memtest86 utillity and let it run for nearly 24hours. It did 61 passes without any errors. And since the corruption always occurs within 24hours, I think my memory is not the cause .
Also strange. I noticed since I degraded the raid again so it runs on one disk only again, only my home partition seems to get 'infected' by small filesystem corruption. My data partition doesn't seem to have any problems anymore. While running the raid on 2 disks , I had severe corruption on both partitions ... still wondering what the problem can be
 
Old 10-22-2006, 09:55 AM   #5
Chojin
LQ Newbie
 
Registered: Dec 2005
Posts: 5

Original Poster
Rep: Reputation: 0
I seem to have found the source of the problem.
I tried using that second disc individually without RAID or LVM and now I get a lot of errors like this:
Code:
attempt to access beyond end of device
I only think it is strange I didn't see any kind these errors when using the disc in the raid..
I will now check this disk througly to find out what is going wrong on it...

Also strange that my home partition within the still degraded raid1 kept going corrupt.. I now reformated that partition into reiserfs hoping the problem goes away there too..
 
  


Reply

Tags
corruption, data, lvm, raid


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off


Similar Threads
Thread Thread Starter Forum Replies Last Post
Trying to rescue data from a corrupted jfs partition jchance Linux - Hardware 1 09-15-2006 12:27 PM
retrieve data from old (working) hd with lvm JohnLocke Linux - General 2 07-03-2006 09:47 PM
LVM - possible data loss zdenisl Linux - Software 0 05-11-2006 06:21 PM
Recovering data from LVM Child of Wonder Linux - Hardware 5 11-12-2005 06:29 AM
Preserving LVM data through reinstall letrout Linux - Newbie 5 08-26-2005 04:08 PM


All times are GMT -5. The time now is 05:27 PM.

Main Menu
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
identi.ca: @linuxquestions
Facebook: linuxquestions Google+: linuxquestions
Open Source Consulting | Domain Registration