[SOLVED] sudden decrease in free space

anctop · 12-05-2011, 09:11 PM

My system has two disks, each one has two partitions and they are mirrored (RAID 1), i.e.

/dev/md1 = "sda1" and "sdb1" mirrored
/dev/md2 = "sda2" and "sdb2" mirrored

A cron job is scheduled to report disks usage daily using "df -a -T".

Today I notice a significant drop in free space of md1. The last check (on 2011.12.06 02:08:09) shows :

Code:

Filesystem    Type   1K-blocks      Used Available Use% Mounted on
/dev/md1      ext2    44402252   4529824  37616912  11% /
proc          proc           0         0         0   -  /proc
sysfs        sysfs           0         0         0   -  /sys
devpts      devpts           0         0         0   -  /dev/pts
/dev/md2      ext3    88811952  17448148  66852360  21% /home
nfsd          nfsd           0         0         0   -  /proc/fs/nfsd

but now it reads :

Code:

Filesystem    Type   1K-blocks      Used Available Use% Mounted on
/dev/md1      ext2    44402252   5391660  36755076  13% /
proc          proc           0         0         0   -  /proc
sysfs        sysfs           0         0         0   -  /sys
devpts      devpts           0         0         0   -  /dev/pts
/dev/md2      ext3    88811952  17443208  66857300  21% /home
nfsd          nfsd           0         0         0   -  /proc/fs/nfsd

It appears that about 850MB of space on "md1" has been eaten in the past few hours.

I've checked all the files created today, but I can't find any exceptionally huge file, nor a very large number of files.

Also, "dmesg" shows no error.

Could someone suggest further diagnosis ?

fukawi1 · 12-05-2011, 11:21 PM

Please use code tags, as it makes reading console output much easier by maintaining spacing etc.

Try checking your /var/log, for huge log files.
Sometimes, depending on how its configured, if something goes wrong, syslog can generate a whole bunch of log entries that can eat up disk space in no time.

Just a thought.

jschiwal · 12-06-2011, 12:28 AM

From 13% to 11% doesn't necessarily sound that massive to me.

After checking for rotated log files (those ending in .gz) you can delete, look at deleted files which are still open:
lsof +L1

Some programs, especially services will create a file, delete it but keep it open for use. This prevents name collision. Also, if the service crashes, the service's working file will be deleted.

If you use the desktop, the cache files can build up to GB in size. Look in ~/.cache/ and ~/.thumbnails/.

Since /tmp is a directory on your root partition, that is a good place to check. Files here can be deleted between reboots. Your system may have a setting to do this, and another setting to delete old files.

Using a graphical file size program such as Konqueror's file size view, may help to quickly locate a large file causing the problem. Back when I used openSuSE 10.0, I had a problem with KDE causing the ~/.xsession-errors file to grow to over 1GB in size. A run away program could be creating a very large file. The FS view may be the best way to locate it.

Indexing programs such as updatedb, strigi or Picasa can build very large databases over time. One thing to check is if they index directories they shouldn't.

Linux.tar.gz · 12-06-2011, 03:59 AM

+1 for /var/log
Had systems with 8G+ logs ^^.

anctop · 12-06-2011, 06:06 PM

Thank you for the suggestions.

In fact I've examined all the files descending from "/" except for "/home" at which "/dev/md2" is mounted.
There're about 40 files created/modified during the period, but none of them is suspicious.

Then I try to kill and re-load the daemons one by one. I find that the cause of the problem should be a dead thread of "httpd". When the branch is killed, the disk usage returns to 11% !

But I'm confused : how is it possible to eat up disk space without forming visible files ?

asimba · 12-07-2011, 12:23 AM

I might have started with checking following

1. Temp Files
2. Log Files
3. Have you updated ANY software/
4. Temporary Internet files/partial downloads/cancelled downloads.
5. Any Installed database ? - this may be unlikely if - 1. its not a heavy workload environment 2.If you are not storing any any binary data in database

6. Does this system act as web server

Next We could generate list of files created/accessed/modified past 24 hours if that was going to drop any hint.