[SOLVED] 151GB of free space has gone missing

thund3rstruck · 07-07-2015, 11:45 AM

I'm really struggling to understand what's happening here. My server is Ubuntu 14.04.02 LTS and I've seemed to have misplaced 151GB of file space. The server started reporting this morning that it was out of free space, but as far as I can tell, I'm only using 76GB.

This df is shady because it says there is 212G used (of 226G), which is not 2.8G (is 226-212 somehow not 14G?)

Code:

developer@BAILEYFS01:~$ df -h
Filesystem      Size  Used Avail Use% Mounted on
/dev/sda1       226G  212G  2.8G  99% /
none            4.0K     0  4.0K   0% /sys/fs/cgroup
udev            1.7G   12K  1.7G   1% /dev
tmpfs           345M  7.3M  338M   3% /run
none            5.0M     0  5.0M   0% /run/lock
none            1.7G     0  1.7G   0% /run/shm
none            100M     0  100M   0% /run/user
/dev/md0        7.3T  4.6T  2.3T  67% /media/share/raid0
/dev/sdd1       2.7T  1.9T  780G  72% /media/share/backup1
/dev/sde1       2.8T  2.5T  299G  90% /media/seagate3

Then I listed the disk usage for every directory off / and it shows I'm only using 76GB (approx.)

Code:

developer@BAILEYFS01:~$ ls /
bin   etc             lib         mnt   run   tmp      vmlinuz.old
boot  home            lib64       opt   sbin  usr      webmin-setup.out
data  initrd.img      lost+found  proc  srv   var
dev   initrd.img.old  media       root  sys   vmlinuz

I've omitted the /media path since those are remote samba & USB mounts to separate volumes

Code:

developer@BAILEYFS01:~$ sudo du -ckh /bin | tail -n 1 | cut -f1
9.7M
developer@BAILEYFS01:~$ sudo du -ckh /boot | tail -n 1 | cut -f1
59M
developer@BAILEYFS01:~$ sudo du -ckh /data | tail -n 1 | cut -f1                
0
developer@BAILEYFS01:~$ sudo du -ckh /dev | tail -n 1 | cut -f1
12K
developer@BAILEYFS01:~$ sudo du -ckh /etc | tail -n 1 | cut -f1
61M
developer@BAILEYFS01:~$ sudo du -ckh /home | tail -n 1 | cut -f1
69G
developer@BAILEYFS01:~$ sudo du -ckh /lib | tail -n 1 | cut -f1
439M
developer@BAILEYFS01:~$ sudo du -ckh /lib64 | tail -n 1 | cut -f1
4.0K
developer@BAILEYFS01:~$ sudo du -ckh /lost+found/ | tail -n 1 | cut -f1
16K
developer@BAILEYFS01:~$ sudo du -ckh /mnt | tail -n 1 | cut -f1
4.0K
developer@BAILEYFS01:~$ sudo du -ckh /opt/ | tail -n 1 | cut -f1
84M
developer@BAILEYFS01:~$ sudo du -ckh /root | tail -n 1 | cut -f1
124K
developer@BAILEYFS01:~$ sudo du -ckh /run | tail -n 1 | cut -f1
7.2M
developer@BAILEYFS01:~$ sudo du -ckh /sbin | tail -n 1 | cut -f1
12M
developer@BAILEYFS01:~$ sudo du -ckh /srv | tail -n 1 | cut -f1
4.0K
developer@BAILEYFS01:~$ sudo du -ckh /sys | tail -n 1 | cut -f1
0
developer@BAILEYFS01:~$ sudo du -ckh /tmp | tail -n 1 | cut -f1
65M
developer@BAILEYFS01:~$ sudo du -ckh /usr | tail -n 1 | cut -f1
1.9G
developer@BAILEYFS01:~$ sudo du -ckh /var | tail -n 1 | cut -f1
3.5G
developer@BAILEYFS01:~$ sudo du -ckh /vmlinuz | tail -n 1 | cut -f1
0

I've got virtualbox vms and scripts that are now failing since the system thinks its out of free space on the root partition /dev/sda1.

What do I need to do to recover this 151GB of space that has somehow vanished into thin air...

Please help....

Keruskerfuerst · 07-07-2015, 11:52 AM

I also had this problem with hidden file.
Solution: reinstallation.

thund3rstruck · 07-07-2015, 12:08 PM

Quote:

Originally Posted by Keruskerfuerst

I also had this problem with hidden file.
Solution: reinstallation.

Please don't say that. This server is the engine that drives a small network and I don't even want to think about what a rebuild would entail at this point.

Keruskerfuerst · 07-07-2015, 12:53 PM

I had the same problem with
1. Windows 7
2. Gentoo Linux

astrogeek · 07-07-2015, 01:03 PM

You may have a sparse file or two taking up space that is not reported by du by default, but is reported by df, ls and other stat checks.

You can use the --apparent-size option to du to find where those are. They are often associated with virtualization, among other things, so that might be a first place to look.

... or see the next post by rknichols...

rknichols · 07-07-2015, 01:21 PM

In the df output, the reserved space (by default, 5%) is not included in either the "Used" or "Avail" columns. That accounts for your missing 11.3G.

There are two possibilities for the extra used space (and "sparse file" is not one of them):

There are one or more large files that have been deleted, and thus invisible to du, but are still held open by some program.
You have files hidden under an active mount point.

You can look for unlinked files by running

Code:

lsof | grep -i del | less

and scroll through the output looking for huge files. Yes, there will be a lot of cruft to ignore in that list.

You can look for files hidden under an active mount point this way:

Code:

mkdir /tmp/tmproot
mount --bind / /tmp/tmproot
du --max-depth=1 /tmp/tmproot

You can of course dig deeper by adjusting the path and/or the max-depth setting. If you find files there that should not exist, you can simply rm them, or move them elsewhere.

After you are done:

Code:

umount /tmp/tmproot
rmdir /tmp/tmproot

All of the above needs to be done as root.

michaelk · 07-07-2015, 01:35 PM

The output of the df command does not show reserved space which is why the numbers do not add up.

In addition to a sparse file, if for some reason one of the USB drives were not mounted and a scheduled backup wrote to / instead of sdd1 or sde2 then once the drive(s) were remounted the used space would appear to be missing. If you deleted large files but the running process still held them opened then used space would appear to be missing.

rknichols was faster...

lrzak · 07-07-2015, 01:36 PM

I had one that ran three years, no problem. Then it reported no free space. I rebooted, space was back. Looked for trash file build up, logs, etc. Could not find anything.

I wrote a script and put it in crontab that sends me a df report each day. When the used space gets to over 90 I login and reboot it, then we start at around 22% used again. It gains 1-3% per day. The site has one web site on it and nothing else.

Here's the script:

df -h | grep /dev/ | mailx -s "%free - When the % hits 90, we worry." myemail@myaddress.com

I spent hours looking for it. Saved the output of du then ran it again a couple days later and compared the files sizes, found nothing substantial.

This doesn't fix it, but maybe the script will help you prevent a problem.

leon ...

thund3rstruck · 07-07-2015, 05:52 PM

Quote:

Originally Posted by michaelk

if for some reason one of the USB drives were not mounted and a scheduled backup wrote to / instead of sdd1 or sde2 then once the drive(s) were remounted the used space would appear to be missing. If you deleted large files but the running process still held them opened then used space would appear to be missing.

There are nightly backups scheduled to both those USB drives and I did indeed determine that /media/seagate3 (an ntfs-3g volume) somehow lost its mount. The backup script uses rsync which I assume will not arbitrarily write to /? Who knows, that would explain how a full 3rd of the hard drive mysteriously filled up overnight.

I rebooted the server a few times and the space didn't reappear. I guess I'll try the stuff @rknichols recommended.

michaelk · 07-07-2015, 06:22 PM

The only way to reclaim the space is to delete the files on the / file system. Rsync can not tell if a directory is a mount point. You might want to add a check to your backup script to check if the drive is still mounted.

thund3rstruck · 07-07-2015, 06:44 PM

Quote:

Originally Posted by rknichols

You can look for files hidden under an active mount point this way:

Code:

mkdir /tmp/tmproot
mount --bind / /tmp/tmproot
du --max-depth=1 /tmp/tmproot

You can of course dig deeper by adjusting the path and/or the max-depth setting. If you find files there that should not exist, you can simply rm them, or move them elsewhere.

After you are done:

Code:

umount /tmp/tmproot
rmdir /tmp/tmproot

Okay... I'm not 100 sure what I'm looking at.

Code:

developer@BAILEYFS01:~$ sudo du -h --max-depth=1 /tmp/tmproot/
4.0K    /tmp/tmproot/proc
4.0K    /tmp/tmproot/sys
69G     /tmp/tmproot/home
9.7M    /tmp/tmproot/bin
4.0K    /tmp/tmproot/srv
16K     /tmp/tmproot/lost+found
123G    /tmp/tmproot/media
124K    /tmp/tmproot/root
3.5G    /tmp/tmproot/var
59M     /tmp/tmproot/boot
8.0K    /tmp/tmproot/dev
84M     /tmp/tmproot/opt
61M     /tmp/tmproot/etc
1.9G    /tmp/tmproot/usr
439M    /tmp/tmproot/lib
4.0K    /tmp/tmproot/run
4.0K    /tmp/tmproot/mnt
4.0K    /tmp/tmproot/lib64
1.6M    /tmp/tmproot/tmp
12M     /tmp/tmproot/sbin
197G    /tmp/tmproot/

From this output I'm noticing the /tmp/tmproot/media folder is using 123G. All the directories in /media are mount points to other volumes (USB and Samba).

I guess I'm asking, how do I ascertain what, if anything is safe to delete from this remounted /tmp/tmproot path?

Thanks again for all the help. There's zero chance I'd have figured this one out on my own.

--UPDATE--
It think that was it. Looks like when the destination USB drive is not mounted any more it sends the backups into some dangling hidden path...

Here's the df output after blowing out that /tmp/tmproot/media/seagate3 path

Code:

developer@BAILEYFS01:~$ df -h
Filesystem      Size  Used Avail Use% Mounted on
/dev/sda1       226G   75G  140G  35% /
none            4.0K     0  4.0K   0% /sys/fs/cgroup
udev            1.7G  4.0K  1.7G   1% /dev
tmpfs           345M  7.1M  338M   3% /run
none            5.0M     0  5.0M   0% /run/lock
none            1.7G     0  1.7G   0% /run/shm
none            100M     0  100M   0% /run/user
/dev/sde1       2.8T  2.5T  314G  89% /media/seagate3
/dev/md0        7.3T  4.6T  2.3T  67% /media/share/raid0
/dev/sdd1       2.7T  1.9T  821G  70% /media/share/backup1

rknichols · 07-07-2015, 07:37 PM

Quote:

Originally Posted by thund3rstruck

--UPDATE--
It think that was it. Looks like when the destination USB drive is not mounted any more it sends the backups into some dangling hidden path...

That's the sort of thing I expected to see. At some point you performed the backup, and since the mount point directory "/media/seagate3" existed, that's where the backup was sent. But, without the external drive mounted there, that directory was just part of the root filesystem. When the external drive is mounted there, you can't see those root filesystem files without doing the bind-mount that I suggested.

FWIW, the way I protect myself against this sort of mishap is by making a subdirectory, like "backups", on my backup drive and storing my backups there. If I try to run backups without the drive mounted, backup fails because directory "/media/seagate3/backups/" doesn't exist.

thund3rstruck · 07-08-2015, 07:57 AM

Quote:

Originally Posted by rknichols

FWIW, the way I protect myself against this sort of mishap is by making a subdirectory, like "backups", on my backup drive and storing my backups there. If I try to run backups without the drive mounted, backup fails because directory "/media/seagate3/backups/" doesn't exist.

Excellent suggestion indeed. As soon as I get a few minutes of spare time I'm going to add some logic like:

Code:

if [ ! -d "/media/seagate3/Backup" ]; then
  ...
fi

into the nightly, monthly, and annually backup scripts.

Again, I can't express my gratitude enough for your help on this!