Disk Space Used is Less Than Size Still Available Disk Space is Zero

How comes it that Used is < Size still Avail is showing 0 in the output below:

# df -hT /export
Filesystem    Type    Size  Used Avail Use% Mounted on
/dev/drbd0    ext3     29G   27G     0 100% /export


# cat /etc/issue
Red Hat Enterprise Linux Server release 5.6 (Tikanga)
Kernel \r on an \m

This is usually open but deleted files. whilst you can delete a filename entry, until nothing is referencing that part of the disk, that disk space is still unavailable.

I remember filesystems reserving some space for root exclusive access after they are declared full for non-root users.

I don't know if ext3 have this feature.

Also worth checking inodes usage

df -hi

The discrepancy between 29G and 27G is the 5% that by default is reserved for root. You can use tune2fs to reduce that percentage, but fragmentation can become a problem if the free space gets much below 5%. The filesystem needs some free space in each cylinder group or else files might end up scattered widely around.

This happens when live files are deleted.

lsof /export | grep -i deleted - will give you the processes that are still "writing" to files that were deleted.

If the processes are bounceable processes e.g application processes try bouncing ( restarting ) the process.

else

kill -9 <pid> - to kill the processes that are "writing" to the deleted files.

Works for me everytime.

If you dont mind getting into meetings, reboot the system. Its a thoughtfree / sweatfree fix.

Originally Posted by smbhandary This happens when live files are deleted.

No, it doesn't. This is different from the frequently reported discrepancy between what df and du report for a filesystem. While du won't see deleted, but in use, files, for df a used block is a used block, and it makes no difference whether the file using that block does or does not have links in the directory tree.

My response is based on my working experience.
My 'usefulness' to my employer is decided on what I can fix or get fixed.
The intention of my being a member here is that I hope my experience will help someone fix her/his problem.
No offense intended in any of the above statements, please.

The problem as I understand it :

# df -hT /export
Filesystem Type Size Used Avail Use% Mounted on
/dev/drbd0 ext3 29G 27G 0 100% /export

And du / du -<summary switch> does not show cause for it


Ive doing platform support for 11 years now.

Application support teams deleting live ( in use / being written to ) application logs is a common occurrence. Ive never spent time wondering why they do it.

The ticket comes to us saying the df says the filesystem is full and du -<summary switch> does not show cause for it.

I have fixed every single such ticket using the commands provided in my last post.

Another cause of df and du not saying the same thing could be the filesystem running out of inodes. I have never faced that condition till date.


I will accept that there is something I have not considered / am not aware of, if devUNIX states that my suggestion did not fix the problem at hand.

Its is not what we have done, the cause could be anything you told or told somewhere above, mostly the cause is deleted files that are still open to write somewhere, so @devunix confirm the inode usage and crosscheck with what @smbhandary has provided, you will have the solution and cause both.

cheers

Originally Posted by smbhandary The problem as I understand it : # df -hT /export Filesystem Type Size Used Avail Use% Mounted on /dev/drbd0 ext3 29G 27G 0 100% /export And du / du -<summary switch> does not show cause for it

If you would care to point out where the OP mentioned anything about du, I would like to see it.

This is a simple matter of the 5% default reserved space in the filesystem and df therefore reporting "100%" usage while the number of used blocks is less than the total.

rknichols . Point taken....
Now I know what Voltaire meant when he said " The tragedy of speaking nonsense is when it is spoken pompously"