Root directory 100% full - but disk utilization doesn't add up. Maybe solved?

gustep12 · 10-17-2016, 10:52 PM

I ran into a common problem that took me more than an hour to solve. Here is what happened - maybe someone can explain this better?

My root directory / is sda2 (~40GB capacity, normally 25% full)
My home directory /home is sda3 (~400GB capacity, normally 20% full)

I also have several network shares (Samba, SMB) that reside on a separate NAS, which I mount at boot through fstab like this, for convenient local access:

//192.168.1.245/Public/Shared_folder ==> mounted locally as /mnt/Shared_Folder (~1TB capacity, normally 10% full)

df / shows root as 100% used
du / shows a lot of data since /mnt/Shared_Folder is included

du -x /(to exclude remote folders under /mnt) shows ~10GB of data, about

The problem was that I also had an rsync script that was supposed to copy lots of data to /mnt/Shared_Folder. However, apparently the NAS went down, and somehow Linux then created an actual folder under /mnt/Shared_Folder on the local root (sda2). Later on, the NAS was back online.

The fundamental problem is that it was very difficult for me to tell apart which directories in /mnt are actually residing locally on sda2, and which ones are remote SMB shares.

Eventually, I edited my fstab and rebooted without the network shares. And see there, now I found a large sub-folder inside /mnt/Shared_Folder, which means it must have been local, even though it shouldn't have been. I deleted that, restored my fstab, rebooted again.

Now df shows root directory back at ~25% usage. All is well again.

If the same problem happens again, how can I use ls, df, du, or something else, to see which directories (or files) in /mnt are local (by accident), when they all should have been remote on the NAS? And how do I check the size of local files in /mnt but not any remote files that are mapped to /mnt ?

Thank you!

jailbait · 10-18-2016, 12:10 PM

Quote:

Originally Posted by gustep12

The problem was that I also had an rsync script that was supposed to copy lots of data to /mnt/Shared_Folder. However, apparently the NAS went down, and somehow Linux then created an actual folder under /mnt/Shared_Folder on the local root (sda2). Later on, the NAS was back online.

If the same problem happens again, how can I use ls, df, du, or something else, to see which directories (or files) in /mnt are local (by accident), when they all should have been remote on the NAS?

I suggest that you place an if statement in your rsync script to check if /mnt/Shared_Folder is mounted. If /mnt/Shared_Folder is not mounted you can either mount it or issue an error message and exit the script. Here is an example where I check to see if a USB drive is mounted before I run a rsync backup to it:

# Mount the USB hard drive
if ! (mount -t ext4 -L dog /media/usb.hard.drive);
then
echo "mount failed for USB hard drive on dog partition"
exit
fi

---------------------
Steve Stites

michaelk · 10-18-2016, 01:04 PM

There are many ways to find out mount points. You can look at the output of mount, df, /proc/mounts, /etc/mtab, via mountpoint etc.

To check that the share is mounted via the mountpoint command.

Code:

res=$( mountpoint -q "/mnt/Shared_Folder" )
if (( $res != 0 )); then
    exit
fi

# Your rsync code goes here

rknichols · 10-18-2016, 01:08 PM

You can use a bind-mount to see what might be hiding under an active mount point. Example for the root filesystem:

Code:

mkdir /tmp/tmpmnt
mount --bind / /tmp/tmpmnt

You can now look in /tmp/tmpmnt and see the root filesystem as it would appear without anything mounted. You can move or delete files that you find in places they shouldn't be.