Commands fail with disk full messages, but df says I've got plenty of space

rsgrimes · 02-09-2010, 12:35 PM

I've been getting error messages indicating my disk might be full, but I don't really think it is. It is so bad that I could not log on via Gnome, but fortunately I was able SSH into my box. Once in, however, I can't even create a directory!

So I did the df command:

Code:

vree:~/tmp$ df -h
Filesystem            Size  Used Avail Use% Mounted on
/dev/sdb1             132G   57G   69G  46% /
tmpfs                 505M     0  505M   0% /lib/init/rw
varrun                505M  124K  505M   1% /var/run
varlock               505M     0  505M   0% /var/lock
udev                  505M  2.7M  502M   1% /dev
tmpfs                 505M   24K  505M   1% /dev/shm
lrm                   505M  2.0M  503M   1% /lib/modules/2.6.27-11-generic/volatile

Hmm... Doesn't look full to me, as df reports 69G available! Or am I confused?

Any ideas?

TIA,
-Bob

amani · 02-09-2010, 12:38 PM

fsck the partitions

Get full info on the relevant partitions incl inodes

ext4?

SharpyWarpy · 02-09-2010, 06:02 PM

Exactly what are the error messages saying? Also, just for kicks and giggles, post the results of:
fdisk -l
Is this a new install? What distro are you using?

anomie · 02-09-2010, 06:06 PM

Also, post the output of mount and df -i.

You may have a read-only filesystem, which the fsck could probably evaluate and repair. (Likely cause.) Or... you could be out of inodes. (Unlikely.)

chrism01 · 02-09-2010, 06:08 PM

Definitely post the exact error msgs. You can also check the inodes

df -i

worm5252 · 02-09-2010, 10:21 PM

your df command shows / mounted on /dev/sdb1. What is on /dev/sda?

eckirchn · 02-09-2010, 10:31 PM

Quote:

Originally Posted by rsgrimes

I've been getting error messages indicating my disk might be full

What is the error you are getting?

rsgrimes · 02-10-2010, 09:11 AM

Wow! Thanks for all the responses! Here are my replies...

1. Sorry I wasn't clear on ther actual error messages. Here is one example; basically, I get the same "No space left on device" regardless of what I try:

Code:

mkdir: cannot create directory `testdir': No space left on device

2. It is an old installation (Ubuntu 8.10), about a year, pretty much just used as a Subversion and Trac server...

3. After rebooting, I was able to log on in a fail-safe terminal, and by deleting a few files, I was able to do the following, as suggested by several posts I've found:

Code:

sudo touch /forcefsck

Rebooting again, I still have the problem.

4. fdisk -l and mount results

Code:

vree:~$ fdisk -l
Cannot open /dev/sda
Cannot open /dev/sdb

rsg@#vree
vree:~$ mount
/dev/sdb1 on / type ext3 (rw,relatime,errors=remount-ro)
tmpfs on /lib/init/rw type tmpfs (rw,nosuid,mode=0755)
/proc on /proc type proc (rw,noexec,nosuid,nodev)
sysfs on /sys type sysfs (rw,noexec,nosuid,nodev)
varrun on /var/run type tmpfs (rw,nosuid,mode=0755)
varlock on /var/lock type tmpfs (rw,noexec,nosuid,nodev,mode=1777)
udev on /dev type tmpfs (rw,mode=0755)
tmpfs on /dev/shm type tmpfs (rw,nosuid,nodev)
devpts on /dev/pts type devpts (rw,noexec,nosuid,gid=5,mode=620)
fusectl on /sys/fs/fuse/connections type fusectl (rw)
lrm on /lib/modules/2.6.27-11-generic/volatile type tmpfs (rw,mode=755)
securityfs on /sys/kernel/security type securityfs (rw)
rpc_pipefs on /var/lib/nfs/rpc_pipefs type rpc_pipefs (rw)
nfsd on /proc/fs/nfsd type nfsd (rw)
binfmt_misc on /proc/sys/fs/binfmt_misc type binfmt_misc (rw,noexec,nosuid,nodev)

5. df -i results

Code:

vree:~$ df -i
Filesystem            Inodes   IUsed   IFree IUse% Mounted on
/dev/sdb1            8773632 8773616      16  100% /
tmpfs                 129117       4  129113    1% /lib/init/rw
varrun                129117      59  129058    1% /var/run
varlock               129117       3  129114    1% /var/lock
udev                  129117    4941  124176    4% /dev
tmpfs                 129117       1  129116    1% /dev/shm
lrm                   129117      17  129100    1% /lib/modules/2.6.27-11-generic/volatile

So it's an INode problem! Okay, I know what INodes are, but I don't know how to manage them. FWIW, I wouldn't think I have that many files, so I would welcome suggestions on the usual suspects, you know, where to look, what generally causes lots of extraneous inodes, etc.

Thanks!

rsgrimes · 02-10-2010, 10:14 AM

I found my problem! I have been using BackInTime (a poor man's Time Machine) on my workstation, and it was taking snapshots on the server; I haven't constrained it a bit, and so it has generated quite a few files - I'm currently counting them, and it's up to almost 4 million - and counting!

That's what you get with "set and forget" tools! (No, that's not a criticism of BackInTime!).

Thanks,
-Bob

michaelk · 02-10-2010, 10:21 AM

An inode is a data structure that is associated with each file. With ext3 the max number of inodes is set when the filesystem is created and the only way to increase the number is to reformat. You can not manage them i.e if you run out of inodes you can not create more files. There are other filesystems that can dynamically allocate inodes like jfs, xfs or ext4.

rsgrimes · 02-10-2010, 12:11 PM

Quote:

Originally Posted by michaelk

An inode is a data structure that is associated with each file.

Yeah, this I know, but...

Quote:

Originally Posted by michaelk

With ext3 the max number of inodes is set when the filesystem is created and the only way to increase the number is to reformat. You can not manage them i.e if you run out of inodes you can not create more files. There are other filesystems that can dynamically allocate inodes like jfs, xfs or ext4.

...this I didn't - thanks for the info!

Cheers,
-Bob

chrism01 · 02-10-2010, 06:24 PM

Also, there's not much point in backing up to the same disk; what happens if the disk goes bad?

rsgrimes · 02-11-2010, 06:57 AM

Quote:

Originally Posted by chrism01

Also, there's not much point in backing up to the same disk; what happens if the disk goes bad?

No, it was another computer backingup to this computer - of course, what you say makes sense!

rnturn · 02-11-2010, 03:46 PM

Quote:

Originally Posted by rsgrimes

I haven't constrained it a bit, and so it has generated quite a few files - I'm currently counting them, and it's up to almost 4 million - and counting!

That's what you get with "set and forget" tools! (No, that's not a criticism of BackInTime!).

Not being familiar with that tool, I'm wonder whether it is possible to configure it so that you can be emailed a summary of what was backed up/saved? If so, maybe you could put the number of files acted on right into the "Subject:" line you'd get that "In Your Face" reminder to keep track of things.

Later...

--
Rick

chrism01 · 02-11-2010, 06:03 PM

You can add a cron job to check the disk space, both filespace & inode cnt and email you if you exceed specified values.
You can even use quotas, eg soft limit at warning level, hard limit at full disk.
Quotas have both diskspace and inode settings.