LinuxQuestions.org
Download your favorite Linux distribution at LQ ISO.
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - Newbie
User Name
Password
Linux - Newbie This Linux forum is for members that are new to Linux.
Just starting out and have a question? If it is not in the man pages or the how-to's this is the place!

Notices


Reply
  Search this Thread
Old 05-21-2012, 10:27 AM   #1
reefoid
LQ Newbie
 
Registered: May 2012
Posts: 2

Rep: Reputation: Disabled
Root disk full after rsync failure


I woke up this morning to find my Samba shares on my Debian headless server were not accessible from my Win7 machine. After some investigation I discovered that the root disk on the Debian box had 0% free space, which confused me as I had around 5.5G free yesterday.

After some further investigation I think I've discovered the issue. I have a cron job that runs each morning that rsyncs one of the Debian disks to a shared Windows folder on my Win7 machine. However, for some reason the Win7 machine powered down last night so the shared disk was not available to rsync. I've based this on syslog, which shows the script ran at 8:00, followed by a cifs_mount failure, then immediate disk space warnings from Nagios.

I'm new to command line Linux so I've been trying to find out were all this space has been used. I'm assuming rsync couldn't find the mount so has started writing to root? However, I can't find any large directories that could explain this.

Some output:

df -h
Code:
Filesystem            Size  Used Avail Use% Mounted on
/dev/sda1             9.2G  9.1G     0 100% /
tmpfs                 501M     0  501M   0% /lib/init/rw
udev                  496M  152K  496M   1% /dev
tmpfs                 501M     0  501M   0% /dev/shm
/dev/sda6              27G  173M   25G   1% /home
/dev/sdc1             147G   92G   48G  66% /data
/dev/sdb1              37G  3.9G   31G  12% /backups
overflow              1.0M     0  1.0M   0% /tmp
//192.168.1.3/deb-backup/    271G   93G  179G  35% /mnt/backup
du -h --max-depth=1 /
Code:
92G     /data
22M     /etc
4.0K    /opt
du: cannot access `/proc/3447/task/3447/fd/4': No such file or directory
du: cannot access `/proc/3447/task/3447/fdinfo/4': No such file or directory
du: cannot access `/proc/3447/fd/4': No such file or directory
du: cannot access `/proc/3447/fdinfo/4': No such file or directory
0       /proc
94M     /lib
4.0K    /selinux
4.6M    /sbin
40K     /home
15M     /boot
0       /tmp
3.8G    /backups
4.0K    /srv
76K     /root
386M    /var
152K    /dev
16K     /lost+found
93G     /mnt
12K     /media
1.8G    /usr
5.1M    /bin
0       /sys
190G    /
I've also searched for (a) directories larger than 1GB and (b) directories created around the time of the rsync failure but neither of these are turning up anything unusual.

So, how can I find out what is using this space?

Thanks
 
Old 05-21-2012, 11:29 AM   #2
MensaWater
LQ Guru
 
Registered: May 2005
Location: Atlanta Georgia USA
Distribution: Redhat (RHEL), CentOS, Fedora, Debian, FreeBSD, HP-UX, Solaris, SCO
Posts: 6,680
Blog Entries: 14

Rep: Reputation: 1022Reputation: 1022Reputation: 1022Reputation: 1022Reputation: 1022Reputation: 1022Reputation: 1022Reputation: 1022
When you mount a filesystem (local, NFS, Samba etc...) your putting a device on top of a "mount point" which is simply an empty directory. After the mount the directory name is the same but now shows the contents of what is in the filesystem device you mounted.

Since the mount point directory exists even when the filesystem is not mounted it means it can be written to.

So what likely filled up your space given the lack of the mount from the Win7 is the mount point that you typically have for that. This would be hidden from view if you subsequently DID mount the Win7 share.

An example from your system would be:
//192.168.1.3/deb-backup/ 271G 93G 179G 35% /mnt/backup

The mount point directory is "/mnt/backup" and the device is the network share "//192.168.1.3/deb-backup".

If you run "df -h /mnt/backup" it should show you that it is a mount from the above device. If you then unmount (using command "umount /mnt/backup" and do "df -h" you'll see /mnt/backup is now actually just a directory in root.

If you run "du -sk /mnt/backup" you'll see files in the directory. You may be surprised to see it contains lots of files which are likely the result of the rsync being done to the directory even though the device wasn't mounted. That is to say they wrote into the root filesystem because at the time of write /mnt/backup was just a subdirectory of root rather than a mounted filesystem in its own right.

It should be safe to remove the files in /mnt/backup with it unmounted then remount the device ("mount /mnt/backup").

Ideally the entry for mounting the filesystem is in /etc/fstab so the umount and mount will work as I showed above. It is barely possible you're doing something else to mount (e.g. using automount) in which case it takes a little more effort. If you see the entry in /etc/fstab though you should be good to go.
 
Old 05-21-2012, 11:39 AM   #3
reefoid
LQ Newbie
 
Registered: May 2012
Posts: 2

Original Poster
Rep: Reputation: Disabled
Thanks very much, that's sorted it. I hadn't thought to unmount the shared drive. Thanks again.
 
Old 05-21-2012, 01:32 PM   #4
MensaWater
LQ Guru
 
Registered: May 2005
Location: Atlanta Georgia USA
Distribution: Redhat (RHEL), CentOS, Fedora, Debian, FreeBSD, HP-UX, Solaris, SCO
Posts: 6,680
Blog Entries: 14

Rep: Reputation: 1022Reputation: 1022Reputation: 1022Reputation: 1022Reputation: 1022Reputation: 1022Reputation: 1022Reputation: 1022
Glad I could help. If your problem is fixed please go to thread tools and mark this as Solved. It helps others with similar problems more quickly find solutions when doing web searches in future.
 
Old 05-21-2012, 02:46 PM   #5
jlinkels
Senior Member
 
Registered: Oct 2003
Location: Bonaire
Distribution: Debian Wheezy/Jessie/Sid, Linux Mint DE
Posts: 4,493

Rep: Reputation: 636Reputation: 636Reputation: 636Reputation: 636Reputation: 636Reputation: 636
I have run into regularly as well with mounted drives. It is not without risk, as dependent on you partition scheme, it might render your server inaccessible.

Therefor I use this script to execute the rsync job. The script is called from cron. Change the rsync parameters to your needs.

The mount point definition should be clear.

Set echo_flag='echo' to test the script and echo the rsync command to stdout, instead of executing it. Set it back to '' when you are done testing.

This script produces output. It will be mailed to you, so if you have set up your mail correctly you'll receive an email when the share is not mounted. In other cases you'll receive mail as well.

Code:
#!/bin/bash

mount_point='/mnt/sdc2'
echo_flag=''

# Find if the device is mounted

df -h | grep $mount_point > /dev/null
if [ $? -eq 0 ]
then
        $echo_flag rsync -ua --exclude='/mnt' --exclude='/proc' --exclude='/sys' --delete / $mount_point > /var/log/rsync_daily
        echo "mount point $mount_point exists, rsync started"
else
        echo "Error: mount point $mount_point does not exist, rsync operation skipped"
fi
jlinkels
 
Old 05-21-2012, 03:34 PM   #6
linuxguy7820
Member
 
Registered: Mar 2011
Distribution: CentOS, RHEL, Fedora
Posts: 35

Rep: Reputation: 0
(my bad didn't see the previous post) please delete this comment.

Last edited by linuxguy7820; 05-21-2012 at 03:35 PM.
 
  


Reply


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off



Similar Threads
Thread Thread Starter Forum Replies Last Post
[SOLVED] rsync execution issue with crontab - Have given full path to rsync too!! Prabagaran Linux - Server 6 04-15-2011 02:39 AM
Cannot login to CentOS system even as root...disk failure? sneakyimp Linux - Newbie 10 04-12-2008 12:34 AM
is this a disk failure msg - hdc: request senense failure walterj Linux - Hardware 3 03-26-2008 12:01 AM
root disk failure OSI-laya Slackware 0 06-05-2003 12:50 PM
I need a full root/boot disk with all utilities, can anybody help? gbolly_bolly Linux - Software 3 04-25-2002 10:00 AM


All times are GMT -5. The time now is 08:56 PM.

Main Menu
Advertisement
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
Facebook: linuxquestions Google+: linuxquestions
Open Source Consulting | Domain Registration