checking disk space

misterdanny · 04-20-2009, 03:47 PM

I am trying to figure out why my disk space usage is so high. I recently was having issues because the disk space usage hit 100%, so I reverted to an old backup to get things running and to investigate (the backup image was from about 24 hours before), I have a VPS server, running centos 5.x.

Anyways here are a couple ssh commands I ran..

Code:

[root@ses01 /]# cd /
[root@ses01 /]# du -sh
1.6G    .
[root@ses01 /]# df -h
Filesystem            Size  Used Avail Use% Mounted on
/dev/sda1             9.4G  8.8G  148M  99% /
none                  129M     0  129M   0% /dev/shm

I changed to the / directory, and running du -sh (which I read in another post here should get me the disk space usage of THAT directory), but using the command I am more accustomed to (df -h) it says I'm using 8.8 gb, and it is all on the /dev/sda1 filesystem.

Is there something that du can't pick up that df can? Is there a good utility to use to find the source of my disk usage problems? I am beginning to expect that my server has been compromised, but the only evidence I have to support that is how my disk usage seems to have sky rocketed out of nowhere.

thanks for your help.

ilikejam · 04-20-2009, 03:59 PM

Hi.

Yup, files which have been rm'ed (unlinked), but which are still open won't show up in 'du' totals, but will be counted in 'df'.

Try 'lsof | grep deleted' as root to see if there are any open, unlinked files.

Dave

misterdanny · 04-20-2009, 04:41 PM

Code:

~]# lsof | grep deleted                                                                                     
mysqld    4035     mysql    4u      REG                8,1          0    1229633 /tmp/ibORtCyo (deleted)                
mysqld    4035     mysql    5u      REG                8,1          0    1229634 /tmp/ibKkjYff (deleted)                
mysqld    4035     mysql    6u      REG                8,1          0    1229635 /tmp/ibIYpkX5 (deleted)                
mysqld    4035     mysql    7u      REG                8,1          0    1229636 /tmp/ib3uDCHW (deleted)                
mysqld    4035     mysql   11u      REG                8,1          0    1229637 /tmp/ibZlRLIO (deleted)

I installed lsof using yum and ran the command you wrote, and this is the output. I tried using the -s parameter to show the file sizes, but the output is the same. They look like they are mysql records that didn't get deleted for whatever reason, so I woulnd't expect for them to be large.

ilikejam · 04-20-2009, 05:32 PM

Hmmm. That's a bit disappointing - the files listed are pretty small (column 8 is the file size in bytes, if memory serves).

Do you have a lot (thousands? millions?) of small files? Depending on the filesystem, a smaller-than-block-size file will take up a full block, giving a discrepancy between df and du.

Alternatively, it could be something weird going on with the virtualisation - maybe the hypervisor is lying about the disk image size?

If nothing springs to mind, and you can afford the downtime, it might be worth running an fsck on your disk.

Dave

Robhogg · 04-20-2009, 05:59 PM

Another thing to try would be debugfs (assuming the filesystem is ext2/3). I don't believe df shows the whole story. For instance, one of my partitions is shown as:

Code:

/dev/sda6             7.9G  7.0G  476M  94% /home/rob/Music

... but 7.0 / 7.9 is more like 89%, so where's the remaining space? Running sudo debugfs /dev/sda6 and then executing the stats command, shows (among other things):

Code:

Block count:              2070369
Reserved block count:     103518
Free blocks:              225202

So the Free block count equates with 89% usage (roughly), but the Reserved block count is about 5% of the Block count (which would make up the difference). Running debugfs's ffb (find_free_blocks) command gives a free block count of 124815, which is around 6% free / 94% usage.

A little confusing, but seems that free block counts are a little like unemployment statistics - it depends how you count them.

From man mke2fs

Code:

-m reserved-blocks-percentage
              Specify the percentage of the filesystem blocks reserved for
              the super-user.  This avoids fragmentation, and allows root-
              owned daemons, such as syslogd(8), to continue  to  function
              correctly  after non-privileged processes are prevented from
              writing to the filesystem.  The default percentage is 5%.

Rob

zuzoa · 04-21-2009, 03:20 PM

You could always try a program like Filelight or Baobab to find where all your space is being eaten up on a drive. They make very easy-to-read, interactive graphical charts to display disk usage.