Disk space

jovie · 03-13-2007, 05:39 AM

Hi

Can someone explain this to me?

I started off with

Code:

[root@zs]# df -h
Filesystem            Size  Used Avail Use% Mounted on
...
/dev/hda6              48G   45G     0 100% /var

So I did some investigating and found these three files
6.6G /var/log/httpd/error_log.1
7.3G /var/log/httpd/error_log.2
6.7G /var/log/httpd/ssl_error_log.2

I had a look in them, saw which php was causing the trouble, fixed it and deleted the log files. Then I had...

Code:

[root@zeldia tmp]# df -h
Filesystem            Size  Used Avail Use% Mounted on
...
/dev/hda6              48G   25G   21G  55% /var

But whithin about an hour the used space (according to df) had gone up to 39G, the newly used 14G were not in the log files, infact I can't find them anywhere and du is reporting only 27G used. So why the descrepancy? Where have my 14Gs gone?

Code:

[root@zs]# df -h /var
Filesystem            Size  Used Avail Use% Mounted on
/dev/hda6              48G   39G  6.0G  87% /var
[root@zs]# du -sh /var
27G     /var
[root@zs]# du -sh /var/*
12K     /var/account
22M     /var/cache
16K     /var/crash
8.0K    /var/cvs
28K     /var/db
16K     /var/empty
28K     /var/gdm
146M    /var/lib
8.0K    /var/local
124K    /var/lock
2.9G    /var/log
20K     /var/lost+found
4.0K    /var/mail
80K     /var/named
8.0K    /var/nis
8.0K    /var/opt
8.0K    /var/preserve
8.0K    /var/racoon
288K    /var/run
1.6M    /var/spool
8.0K    /var/tmp
8.0K    /var/tux
24G     /var/www
24K     /var/yp

jschiwal · 03-13-2007, 06:04 AM

This is a guess. Look at the output of lsof. If a program has a lock on one of the files, it might not be dropped until that program releases the lock.

jovie · 03-13-2007, 10:45 AM

Thanks for the suggestion but no significantly large files show up in the list.

I'm now back to ...

Code:

[root@zs]# df -h
Filesystem            Size  Used Avail Use% Mounted on
...
/dev/hda6              48G   45G     0 100% /var
[root@zs]# du -sh /var
27G     /var

So I cat'd a file into /var until I got the 'cat: write error: No space left on device' message. The file was 2.5G when it stopped writing - conclusion df is reporting the correct amount of used space but what ever is using up my free space (21G in a matter of hours) is not recognized by du. And that's the problem, if I can't find out what's using it how can I stop it?

These figures haven't changed since my first post but 6G of free space on /var have gone...

Code:

[root@zs]# du -sh /var/*
12K     /var/account
22M     /var/cache
16K     /var/crash
8.0K    /var/cvs
28K     /var/db
16K     /var/empty
28K     /var/gdm
146M    /var/lib
8.0K    /var/local
124K    /var/lock
2.9G    /var/log
20K     /var/lost+found
4.0K    /var/mail
80K     /var/named
8.0K    /var/nis
8.0K    /var/opt
8.0K    /var/preserve
8.0K    /var/racoon
288K    /var/run
1.6M    /var/spool
8.0K    /var/tmp
8.0K    /var/tux
24G     /var/www
24K     /var/yp

win32sux · 03-14-2007, 04:23 AM

hi, just wanted to post some new info, posted by jovie in the Linux - Security forum, related to this (the thread was closed for being essentially the same issue):

Quote:

Originally Posted by jovie

Could someone tell me if this behavior is like to be as a result of a hack and if so is there a better way of dealing with it than reformatting the hard drives and starting again with better security?

Symptoms
Disapearing disk space from the /var partition only. First noticed about 36hrs ago

Code:

[root@zs]# df -h
Filesystem            Size  Used Avail Use% Mounted on
...
/dev/hda6              48G   45G     0 100% /var
[root@zs]# du -sh /var
27G     /var

[...]

while /var/log/http/access_log was still being up dated /var/log/http/error_log last changed ~36 hrs ago and I would have expected a few php warnings to appear in there since then.
ssh connections refused, first noticed about 12hrs ago.
No unrecognized processes were seen running, no malicious damage to web pages and nothing unusual on any other partition, just /var.

I disconnect it from the network about 12 hrs ago so now we have no web server running. There are the usual failed logins in /var/log/secure but the successful ones (that remain in the log) are accounted for.

I read the first thing to do is make sure I'm in control of the machine. How can I do that if I don't know where the disk space went and what its being used for?

The data are backed up so a rebuild is not out of the question but I'd rather not do it if this has some other cause that I can fix.

Thanks for your opinions.

jtmoon · 04-06-2007, 04:46 AM

Hi,
I'm having similar problems.
According to df I have 77 Gigabytes of space.
However, all attempts to create a file fail with the error 'no space left on device'.
These examples are done as user root.

$ df -Tah
Filesystem Type Size Used Avail Use% Mounted on
...
/dev/hdh ext3 466G 367G 77G 83% /media/.Backup

and here's the error (it's consistent no matter which user tries to create a file).
$ touch /media/.Backup/foo
touch: cannot touch `/media/.Backup/foo': No space left on device

Permissions are set to allow rwx by everyone (for the sake of this example).
If I remove some files from /media/.Backup then the call to touch /media/.Backup/foo succeeds.

Here's some more information about the disk and filesystem.
$ parted -s /dev/hdh print
Model: Maxtor 6H500R0 (ide)
Disk /dev/hdh: 500GB
Sector size (logical/physical): 512B/512B
Partition Table: loop
Number Start End Size File system Flags
1 0.00B 500GB 500GB ext3

$ tune2fs -l /dev/hdh
tune2fs 1.38 (30-Jun-2005)
Filesystem volume name: MAXTOR500GB
...
Filesystem magic number: 0xEF53
Filesystem revision #: 1 (dynamic)
Filesystem features: has_journal ext_attr resize_inode dir_index filetype needs_recovery sparse_super large_file
Default mount options: (none)
Filesystem state: clean
Errors behavior: Continue
Filesystem OS type: Linux
Inode count: 477056
Block count: 122096646
Reserved block count: 6104832
Free blocks: 26151205
Free inodes: 4
First block: 0
Block size: 4096
Fragment size: 4096
Reserved GDT blocks: 1024
Blocks per group: 32768
Fragments per group: 32768
Inodes per group: 128
Inode blocks per group: 4
...
Mount count: 20
Maximum mount count: 10
Last checked: Thu Nov 16 06:11:43 2006
Check interval: 1728000 (2 weeks, 6 days)
Next check after: Wed Dec 6 06:11:43 2006
Reserved blocks uid: 0 (user root)
Reserved blocks gid: 0 (group root)
First inode: 11
Inode size: 128
Journal inode: 8
Default directory hash: tea
...
Journal backup: inode blocks

I also tried lsof to list all open files. There are no files open on /media/.Backup
My best idea is there is 77GB of filesystem overhead (superblocks,journal). But I would guess any smart space reporting tool would only report the amount of possible space since users aren't interested in disk space they can never use (most of the time)...

Any ideas on why there is plenty of free space reported yet 'no space' errors?

-J_Tom_Moon_79

jovie · 04-10-2007, 02:46 AM

Sorry I should have posted the solution when I found it. jschiwal was on the right lines I just didn't understand the implications.

I'd deleted the log file used by apache but didn't restart apache. Apache had a lock on the file and kept using it. Thus 'du' considered the file deleted and didn't count the space but 'df' considered the file in use and did count the space. Every time I made free space on the partition apache used it to write to the deleted log file which 'du' wasn't reading so it appeared that the space was mysteriously disappearing.

If you know which service you've been messing with restart it or else you could reboot the machine.