A couple weird things are going. First, if I do a df -h, my system shows that 19G are being used on our /var parition, with 0 free space:

# df -h
Filesystem Size Used Avail Use% Mounted on
/dev/sda3 1012M 394M 567M 42% /
/dev/sda2 99M 47M 48M 50% /boot
/dev/sda8 9.9G 1.5G 7.9G 16% /home
none 1004M 0 1004M 0% /dev/shm
/dev/sda9 347G 93G 236G 29% /srv
/dev/sda11 2.0G 81M 1.8G 5% /tmp
/dev/sda10 4.0G 1.3G 2.6G 33% /usr
/dev/sda6 4.0G 2.1G 1.8G 55% /usr/local
/dev/sda5 20G 19G 0 100% /var

However, if I do a du -sh /var it reports that only 4.6G are being used:

# du -sh /var
4.6G /var

Furthermore, free space is somehow disappearing. If I delete say a 1G file, I can watch the newly freed space rapidly dwindle away by repeatedly doing a df -h. But du -sh /var still shows only 4.6G are being used. And I can't find any files with a larger file size in /var. Really weird. Any ideas?

df and du measure two different things....

du reports the space used by files and folders--even this is more than the file size. A few quick experiments on my system show that 4K is a minimum file size in terms of disk space.

df reports the space used by the file system. This includes the overhead for journals and inode tables and such.

Thanks, a simple reboot set everything straight.

Use lsof to identify deleted files that are still open:

# lsof | grep deleted

The disk space the deleted file occupies will not be freed until the process with it open is terminated. In my case, a nohup'd tcpdump process was writing output to a deleted file in /tmp . After killing the process, the filesystem cleaned up the deleted file inodes within a minute or two. Afterward, du and df user space were nearly identical. A restart effectively does the same steps, but is not required.

Furthermore, note that df works by examining directory entries, which generally are updated only when a file is closed.

Also note that free space + used space will not be equal to total space when looking at df on most ext filesystems. Some (default is 5% for ext3) of the sectors are reserved for root, and don't appear as "available" in df.