I woke up this morning to find my Samba shares on my Debian headless server were not accessible from my Win7 machine. After some investigation I discovered that the root disk on the Debian box had 0% free space, which confused me as I had around 5.5G free yesterday.
After some further investigation I think I've discovered the issue. I have a cron job that runs each morning that rsyncs one of the Debian disks to a shared Windows folder on my Win7 machine. However, for some reason the Win7 machine powered down last night so the shared disk was not available to rsync. I've based this on syslog, which shows the script ran at 8:00, followed by a cifs_mount failure, then immediate disk space warnings from Nagios.
I'm new to command line Linux so I've been trying to find out were all this space has been used. I'm assuming rsync couldn't find the mount so has started writing to root? However, I can't find any large directories that could explain this.
Filesystem Size Used Avail Use% Mounted on
/dev/sda1 9.2G 9.1G 0 100% /
tmpfs 501M 0 501M 0% /lib/init/rw
udev 496M 152K 496M 1% /dev
tmpfs 501M 0 501M 0% /dev/shm
/dev/sda6 27G 173M 25G 1% /home
/dev/sdc1 147G 92G 48G 66% /data
/dev/sdb1 37G 3.9G 31G 12% /backups
overflow 1.0M 0 1.0M 0% /tmp
//192.168.1.3/deb-backup/ 271G 93G 179G 35% /mnt/backup
du -h --max-depth=1 /
du: cannot access `/proc/3447/task/3447/fd/4': No such file or directory
du: cannot access `/proc/3447/task/3447/fdinfo/4': No such file or directory
du: cannot access `/proc/3447/fd/4': No such file or directory
du: cannot access `/proc/3447/fdinfo/4': No such file or directory
I've also searched for (a) directories larger than 1GB and (b) directories created around the time of the rsync failure but neither of these are turning up anything unusual.
So, how can I find out what is using this space?