Root disk full after rsync failure

reefoid · 05-21-2012, 09:27 AM

I woke up this morning to find my Samba shares on my Debian headless server were not accessible from my Win7 machine. After some investigation I discovered that the root disk on the Debian box had 0% free space, which confused me as I had around 5.5G free yesterday.

After some further investigation I think I've discovered the issue. I have a cron job that runs each morning that rsyncs one of the Debian disks to a shared Windows folder on my Win7 machine. However, for some reason the Win7 machine powered down last night so the shared disk was not available to rsync. I've based this on syslog, which shows the script ran at 8:00, followed by a cifs_mount failure, then immediate disk space warnings from Nagios.

I'm new to command line Linux so I've been trying to find out were all this space has been used. I'm assuming rsync couldn't find the mount so has started writing to root? However, I can't find any large directories that could explain this.

Some output:

df -h

Code:

Filesystem            Size  Used Avail Use% Mounted on
/dev/sda1             9.2G  9.1G     0 100% /
tmpfs                 501M     0  501M   0% /lib/init/rw
udev                  496M  152K  496M   1% /dev
tmpfs                 501M     0  501M   0% /dev/shm
/dev/sda6              27G  173M   25G   1% /home
/dev/sdc1             147G   92G   48G  66% /data
/dev/sdb1              37G  3.9G   31G  12% /backups
overflow              1.0M     0  1.0M   0% /tmp
//192.168.1.3/deb-backup/    271G   93G  179G  35% /mnt/backup

du -h --max-depth=1 /

Code:

92G     /data
22M     /etc
4.0K    /opt
du: cannot access `/proc/3447/task/3447/fd/4': No such file or directory
du: cannot access `/proc/3447/task/3447/fdinfo/4': No such file or directory
du: cannot access `/proc/3447/fd/4': No such file or directory
du: cannot access `/proc/3447/fdinfo/4': No such file or directory
0       /proc
94M     /lib
4.0K    /selinux
4.6M    /sbin
40K     /home
15M     /boot
0       /tmp
3.8G    /backups
4.0K    /srv
76K     /root
386M    /var
152K    /dev
16K     /lost+found
93G     /mnt
12K     /media
1.8G    /usr
5.1M    /bin
0       /sys
190G    /

I've also searched for (a) directories larger than 1GB and (b) directories created around the time of the rsync failure but neither of these are turning up anything unusual.

So, how can I find out what is using this space?

Thanks

MensaWater · 05-21-2012, 10:29 AM

When you mount a filesystem (local, NFS, Samba etc...) your putting a device on top of a "mount point" which is simply an empty directory. After the mount the directory name is the same but now shows the contents of what is in the filesystem device you mounted.

Since the mount point directory exists even when the filesystem is not mounted it means it can be written to.

So what likely filled up your space given the lack of the mount from the Win7 is the mount point that you typically have for that. This would be hidden from view if you subsequently DID mount the Win7 share.

An example from your system would be:
//192.168.1.3/deb-backup/ 271G 93G 179G 35% /mnt/backup

The mount point directory is "/mnt/backup" and the device is the network share "//192.168.1.3/deb-backup".

If you run "df -h /mnt/backup" it should show you that it is a mount from the above device. If you then unmount (using command "umount /mnt/backup" and do "df -h" you'll see /mnt/backup is now actually just a directory in root.

If you run "du -sk /mnt/backup" you'll see files in the directory. You may be surprised to see it contains lots of files which are likely the result of the rsync being done to the directory even though the device wasn't mounted. That is to say they wrote into the root filesystem because at the time of write /mnt/backup was just a subdirectory of root rather than a mounted filesystem in its own right.

It should be safe to remove the files in /mnt/backup with it unmounted then remount the device ("mount /mnt/backup").

Ideally the entry for mounting the filesystem is in /etc/fstab so the umount and mount will work as I showed above. It is barely possible you're doing something else to mount (e.g. using automount) in which case it takes a little more effort. If you see the entry in /etc/fstab though you should be good to go.

reefoid · 05-21-2012, 10:39 AM

Thanks very much, that's sorted it. I hadn't thought to unmount the shared drive. Thanks again.

MensaWater · 05-21-2012, 12:32 PM

Glad I could help. If your problem is fixed please go to thread tools and mark this as Solved. It helps others with similar problems more quickly find solutions when doing web searches in future.

jlinkels · 05-21-2012, 01:46 PM

I have run into regularly as well with mounted drives. It is not without risk, as dependent on you partition scheme, it might render your server inaccessible.

Therefor I use this script to execute the rsync job. The script is called from cron. Change the rsync parameters to your needs.

The mount point definition should be clear.

Set echo_flag='echo' to test the script and echo the rsync command to stdout, instead of executing it. Set it back to '' when you are done testing.

This script produces output. It will be mailed to you, so if you have set up your mail correctly you'll receive an email when the share is not mounted. In other cases you'll receive mail as well.

Code:

#!/bin/bash

mount_point='/mnt/sdc2'
echo_flag=''

# Find if the device is mounted

df -h | grep $mount_point > /dev/null
if [ $? -eq 0 ]
then
        $echo_flag rsync -ua --exclude='/mnt' --exclude='/proc' --exclude='/sys' --delete / $mount_point > /var/log/rsync_daily
        echo "mount point $mount_point exists, rsync started"
else
        echo "Error: mount point $mount_point does not exist, rsync operation skipped"
fi

jlinkels

linuxguy7820 · 05-21-2012, 02:34 PM

(my bad didn't see the previous post) please delete this comment.