LinuxQuestions.org
Visit Jeremy's Blog.
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - Software
User Name
Password
Linux - Software This forum is for Software issues.
Having a problem installing a new program? Want to know which application is best for the job? Post your question in this forum.

Notices


Reply
  Search this Thread
Old 06-11-2013, 06:40 AM   #1
rylan76
Senior Member
 
Registered: Apr 2004
Location: Potchefstroom, South Africa
Distribution: Fedora 17 - 3.3.4-5.fc17.x86_64
Posts: 1,540

Rep: Reputation: 101Reputation: 101
LVM - Centos 6 - disc became full, now many files missing - how to fix?


Hi guys

I've got a Centos 6 box that had an LVM partition mounted on /dev/sda2 which was formatted ext3.

The drive became overfull - it is an Asterisk system that logs calls, and these filled up the entire 500GB space available on the drive, mounted as LVM.

It was in this state for more than two days. When I eventually became aware of the problem, I copied about 50GB worth of files away, and rebooted the system normally.

It came back up, but now I'm finding that many directories on the LVM volume show they contiain GBs of files, but if you go into the directory ls -l just returns 0?

E. g. you see

drwx-xr-x 2 root root 335872 May 5 2013 01

but if you cd 01 and ls -la you just see 0. I also see that ls -la takes a looooong time on this "visually empty" folder.

E.g. it appears the files are "there" but you cannot see them or access them. LS omits no errors, it just returns the exact result if you ls a "really empty" directory.

How on earth can I try to recover these files? They seem to still be there - the directory size indicates that some element of LVM / ext3 still "sees" the files but they cannot be accessed anymore.

Any ideas?

Thank you
 
Old 06-11-2013, 04:54 PM   #2
smallpond
Senior Member
 
Registered: Feb 2011
Location: Massachusetts, USA
Distribution: CentOS 6 & 7
Posts: 3,450

Rep: Reputation: 924Reputation: 924Reputation: 924Reputation: 924Reputation: 924Reputation: 924Reputation: 924Reputation: 924
reboot with the filesystem full will lose any unwritten data, directory files, etc. and leave the filesystem in a bad state. After that, bring the system up in single user mode and run e2fsck on the LVM, like below before remounting.
Code:
e2fsck -y /dev/VolGroup01/LogVol00
 
1 members found this post helpful.
Old 06-12-2013, 03:33 AM   #3
rylan76
Senior Member
 
Registered: Apr 2004
Location: Potchefstroom, South Africa
Distribution: Fedora 17 - 3.3.4-5.fc17.x86_64
Posts: 1,540

Original Poster
Rep: Reputation: 101Reputation: 101
Hi Smallpond

Thanks, I'll give that command a try...

Regards,
 
Old 06-12-2013, 08:04 AM   #4
rknichols
Senior Member
 
Registered: Aug 2009
Distribution: CentOS
Posts: 4,474

Rep: Reputation: 2040Reputation: 2040Reputation: 2040Reputation: 2040Reputation: 2040Reputation: 2040Reputation: 2040Reputation: 2040Reputation: 2040Reputation: 2040Reputation: 2040
Quote:
Originally Posted by rylan76 View Post
E. g. you see

drwx-xr-x 2 root root 335872 May 5 2013 01

but if you cd 01 and ls -la you just see 0. I also see that ls -la takes a looooong time on this "visually empty" folder.
That is the size of the directory file itself, which at one point apparently contained tens of thousands of files and needed that much space to store their names. A directory file does not shrink when file names are removed; the space is just marked available. The only ways to release that space and get the directory back to a sane size are to either (a) rmdir the directory and re-create it, or (b) run fsck with the "-D" option (optimize directories) on the unmounted filesystem, which will shrink all the oversized directories.

Before doing (b), I suggest running "tune2fs -O dir_index" on the logical volume that contains the filesystem to make sure that feature is enabled. "fsck -D" will then index all the directories, which greatly speeds up searches in large directories.
 
1 members found this post helpful.
  


Reply


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off



Similar Threads
Thread Thread Starter Forum Replies Last Post
Retrieving files from damaged LVM disk on CentOS 5.5 swbac Linux - General 1 11-04-2010 09:29 PM
Kickstart Centos 5.4 x86_64 missing files. DHCP/PXE/TFTP/HTTP. tdo896 Linux - Networking 3 12-20-2009 05:51 PM
Possibly missing /usr/include/linux header files.. how to fix? brfindla Debian 2 08-10-2009 12:21 PM
Disc Memory Full fassie Linux - Newbie 6 01-31-2008 12:15 AM
Disc full error after a power failure P_Ko Linux - Newbie 4 01-31-2005 12:33 PM

LinuxQuestions.org > Forums > Linux Forums > Linux - Software

All times are GMT -5. The time now is 09:08 PM.

Main Menu
Advertisement
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
Open Source Consulting | Domain Registration