Here's a puzzle for you. I'm stumped.
--------------------------------------------------------------
[root@gandalf root]# cd /var/run/console
[root@gandalf console]# echo "hello" > dum.tmp
-bash: dum.tmp: No space left on device
[root@gandalf run]# df /var
Filesystem 1K-blocks Used Available Use% Mounted on
/dev/hda5 381139 173446 188015 48% /var


--------------------------
What's going on? Ooodles of space on /var, can't write to it because "No space left on device".

Any sysadmin gurus out there have any ideas of an explanation?

Thanks in advance...

Charlie Peppler

i dont have a /var/run/console so im not sure but i would say its not a normal directory but a mount point

I don't think it's a mount point, I haven't created any. Where would I check?

just type mount at the console and it will list all mounts. if its not a mount point then i havnt got a clue what it is.

Here is the result of the mount command:
-------------------------------------------
[root@gandalf var]# mount
/dev/hda3 on / type ext3 (rw)
none on /proc type proc (rw)
usbdevfs on /proc/bus/usb type usbdevfs (rw)
/dev/hda1 on /boot type ext3 (rw)
none on /dev/pts type devpts (rw,gid=5,mode=620)
none on /dev/shm type tmpfs (rw)
/dev/hda6 on /usr type ext3 (rw)
/dev/hda5 on /var type ext3 (rw)
----------------------------------------
You can see /var mounted directly on /dev/hda5

ok here are two silly questions

1) is /var/run/console a symlink to somewhere?
2) does /var/run/console/dum.tmp exist? is it a device node?

-------------Question #1
[root@gandalf run]# ls -al
total 23
drwxr-xr-x 8 root root 1024 Oct 6 11:16 .
drwxr-xr-x 23 root root 1024 Mar 9 2003 ..
-rw-r--r-- 1 root root 4 Oct 6 11:16 atd.pid
drwxr-xr-x 2 root root 1024 Oct 1 17:35 console
-rw-r--r-- 1 root root 4 Oct 6 11:16 crond.pid
-rw------- 1 root root 4 Oct 6 11:16 gpm.pid
-rw-r--r-- 1 root root 4 Oct 6 11:16 httpd.pid
-rw------- 1 root root 4 Oct 6 11:15 klogd.pid
-rw-r--r-- 1 lp lp 4 Oct 6 11:16 lpd.515
srwxrwxrwx 1 root lp 0 Oct 6 11:16 lprng
drwxr-xr-x 2 named named 1024 Aug 7 2002 named
drwxrwxr-x 2 root root 1024 Sep 4 2002 netreport
drwxr-xr-x 2 root root 1024 Oct 6 11:16 samba
drwxr-xr-x 2 root root 1024 Dec 9 2002 saslauthd
-rw-r--r-- 1 root root 4 Oct 6 11:15 sshd.pid
drwx------ 5 root root 1024 Mar 11 2003 sudo
-rw------- 1 root root 4 Oct 6 11:15 syslogd.pid
-rw-rw-r-- 1 root utmp 4608 Oct 6 11:50 utmp
-rw-r--r-- 1 root root 6 Oct 6 11:16 xfs.pid
-rw-r--r-- 1 root root 4 Oct 6 11:15 xinetd.pid
[root@gandalf run]#
------------------Question #2
[root@gandalf run]# pwd
/var/run
[root@gandalf run]# cd console
[root@gandalf console]# ls -al
total 2
drwxr-xr-x 2 root root 1024 Oct 1 17:35 .
drwxr-xr-x 8 root root 1024 Oct 6 11:16 ..
------------------End

Hope this helps.

BTW. I can't create a file on other directories in the same device:
------------------
------------------

googling reveals this error can be caused by a lack of inodes try df -i

[root@gandalf run]# df -i /var
Filesystem Inodes IUsed IFree IUse% Mounted on
/dev/hda5 98392 98392 0 100% /var


Bingo!

How do I get inodes back?

each file takes an inode so you could delete some files, they only way i know to add more inodes is to reformat the partition and specify the -N option to mke2fs. perhaps you should try a different filesystem like reiserfs, ive heard thats really good with lots of little files.

theres probably a big discussion on this in the linux administrators guide(tldp.org) or the rute users guide(in my bookmarks)

OK, now I can put this together with what happened earlier with this machine. I started out here:

http://www.linuxquestions.org/questi...446#post507446

and ended up here:

http://www.linuxquestions.org/questi...threadid=98312

Will this e2fsck command work?

I'ld love to be able to get the inodes back again, and I'm assuming that I'ld need e2fsck to be able to do that.

I would hate to have to nuke this machine, because I put a lot of work into getting this machine setup with all the software I need.

If the inodes are just gonzo, I guess I have no other option than to backup and reinstall from scratch

Charlie

I found the directory /var/lost+found, from the documentation you gave me. In there were 96824 files, all with the file format of #(some number).

I have now written a script to remove all these deadhead files, and my inodes are returning.

Thanks for your help, I'm hoping this will return things to normal.

Charlie Peppler

PostNote:

After running this script, I now have my system back

Sweet! THANK YOU kev82!

<edit> ignore the rest of this post because your deleting files and regaining inodes so theres no problem

to me it sounds as though the /var filesystem is really buggered, the simplest thing would be to format it but because its a seperate partition you wont need to lose any data or reinstall from scratch. shouldnt even need to reboot.

all you need to do is switch to single user mode, backup all the things in /var you want to keep like mail spools and system logs then unmount and run mke2fs on hda5, remount copy eveything back, re-create the directory heirarchy and go back to multi-user mode. you might have to regenerate a few files but it shouldnt be a problem.

before you do this check with a red hat person but i dont think it should be a problem.