[SOLVED] No space left on device Fedora 10

pwave70 · 09-16-2010, 03:35 AM

In message log I've got this informations:

Quote:

Sep 16 10:28:25 pomocnik kernel: lockd: cannot monitor rt135
Sep 16 10:28:25 pomocnik rpc.statd[1902]: creat(/var/lib/nfs/statd/sm/rt135.jelcz.com.pl) failed: No space left on device
Sep 16 10:28:25 pomocnik rpc.statd[1902]: STAT_FAIL to pomocnik.jelcz.com.pl for SM_MON of 10.100.2.135
Sep 16 10:28:25 pomocnik kernel: lockd: cannot monitor rt135
Sep 16 10:28:28 pomocnik nmbd[2531]: [2010/09/16 10:28:28, 0] nmbd/nmbd_serverlistdb.c:write_browse_list(350)
Sep 16 10:28:28 pomocnik nmbd[2531]: write_browse_list: Can't open file /var/lib/samba/browse.dat.. Error was Brak miejsca na urządzeniu
Sep 16 10:28:28 pomocnik nmbd[2531]: [2010/09/16 10:28:28, 0] nmbd/nmbd_serverlistdb.c:write_browse_list(350)
Sep 16 10:28:28 pomocnik nmbd[2531]: write_browse_list: Can't open file /var/lib/samba/browse.dat.. Error was Brak miejsca na urządzeniu
Sep 16 10:28:28 pomocnik rpc.statd[1902]: creat(/var/lib/nfs/statd/sm/rt135.jelcz.com.pl) failed: No space left on device
Sep 16 10:28:28 pomocnik rpc.statd[1902]: STAT_FAIL to pomocnik.jelcz.com.pl for SM_MON of 10.100.2.135
Sep 16 10:28:28 pomocnik kernel: lockd: cannot monitor rt135
Sep 16 10:28:37 pomocnik nmbd[2531]: [2010/09/16 10:28:37, 0] nmbd/nmbd_serverlistdb.c:write_browse_list(350)
Sep 16 10:28:37 pomocnik nmbd[2531]: write_browse_list: Can't open file /var/lib/samba/browse.dat.. Error was Brak miejsca na urządzeniu

df -h:

Quote:

[root@pomocnik ~]# df -h
System plików rozm. użyte dost. %uż. zamont. na
/dev/sda2 2,8G 853M 1,8G 32% /
/dev/sdb2 459G 224G 212G 52% /home
/dev/sda5 9,2G 4,1G 4,7G 47% /usr
/dev/sda6 4,6G 142M 4,3G 4% /usr/local
/dev/sda8 49G 14G 36G 28% /mnt/instalacje
/dev/sda7 1,9G 155M 1,6G 9% /tmp
/dev/sda9 23G 1,3G 20G 7% /home/www
/dev/sda10 20G 11G 8,3G 57% /var
/dev/sda1 183M 20M 154M 12% /boot
tmpfs 981M 168K 981M 1% /dev/shm
/dev/sdc1 147G 21G 127G 14% /mnt/mybook01
/dev/sdd1 7,5G 375M 7,1G 5% /media/KINGSTON
/dev/sr0 192M 192M 0 100% /media/SLAX
/dev/sda11 20G 1,4G 17G 8% /media/disk
/dev/sdc2 315G 146G 154G 49% /media/mybook02
/dev/sdc2 315G 146G 154G 49% /mnt/mybook02

Seems that nn '/var - sda10' is 8GB free, i've checked fsck sda1, everything is ok

Mount:

Quote:

[root@pomocnik ~]# mount
/dev/sda2 on / type ext3 (rw)
proc on /proc type proc (rw)
sysfs on /sys type sysfs (rw)
devpts on /dev/pts type devpts (rw,gid=5,mode=620)
/dev/sdb2 on /home type ext3 (rw)
/dev/sda5 on /usr type ext3 (rw)
/dev/sda6 on /usr/local type ext3 (rw)
/dev/sda8 on /mnt/instalacje type ext3 (rw)
/dev/sda7 on /tmp type ext3 (rw)
/dev/sda9 on /home/www type ext3 (rw)
/dev/sda10 on /var type ext3 (rw)
/dev/sda1 on /boot type ext3 (rw)
tmpfs on /dev/shm type tmpfs (rw)
none on /proc/sys/fs/binfmt_misc type binfmt_misc (rw)
sunrpc on /var/lib/nfs/rpc_pipefs type rpc_pipefs (rw)
nfsd on /proc/fs/nfsd type nfsd (rw)
/dev/sdc1 on /mnt/mybook01 type vfat (rw,noexec,nosuid,nodev,utf8,umask=0000)
gvfs-fuse-daemon on /home/piotrek/.gvfs type fuse.gvfs-fuse-daemon (rw,nosuid,nodev,user=piotrek)
gvfs-fuse-daemon on /root/.gvfs type fuse.gvfs-fuse-daemon (rw,nosuid,nodev)
/dev/sdd1 on /media/KINGSTON type vfat (rw,nosuid,nodev,uhelper=hal,shortname=lower,uid=0)
/dev/sr0 on /media/SLAX type iso9660 (ro,nosuid,nodev,uhelper=hal,uid=0)
/dev/sda11 on /media/disk type ext3 (rw,nosuid,nodev,uhelper=hal)
/dev/sdc2 on /media/mybook02 type ext3 (rw,nosuid,nodev,uhelper=hal)
/dev/sdc2 on /mnt/mybook02 type ext3 (rw,usrquota)

I can't print in netwotk because of those messages...

Computers can connect via samba and nfs.

Any suggestions?

EricTRA · 09-16-2010, 03:38 AM

Hi,

Check if you haven't used up all of your inodes for that partition with

Code:

df -i

If you save lots of small files to a partition it might be that your inodes are all used up and although you'd still have space left on your device you'll be unable to save any more files.

Kind regards,

Eric

pwave70 · 09-16-2010, 03:49 AM

You're right

df -i

Quote:

System plików iwęzły użyteI wolneI %uż.I zamont. na
...
/dev/sda10 1279632 1279632 0 100% /var
...

No space left, so what I must do?

Enlarge the partition?

EricTRA · 09-16-2010, 04:00 AM

Hi,

If you set it up with LVM for example you can enlarge the partition which would solve your problem (temporarily). Better practice is to clean up on a regular basis, check the size of your inodes and if you need to set it smaller, then save / copy what you need from the partition and rebuild the filesystem setting smaller inodes.

Kind regards,

Eric

pwave70 · 09-16-2010, 04:17 AM

Eric thanks for advices,

I try make your second way, but I don't know how.

I've checked inodes size on the partition:

Quote:

# tune2fs -l /dev/sda10 | grep Inode
Inode count: 1279632
Inodes per group: 8048
Inode blocks per group: 503
Inode size: 256

Can you tell me how to rebuild the filesystem with smaller inodes?

regards

EricTRA · 09-16-2010, 04:25 AM

Hello,

When you have backup'ed your partition, delete it, recreate the partition with the new size should you choose to make it bigger and set the inode size using mkfs.ext3 -i or -I.
From the man page of mkfs.ext3:

Quote:

-i bytes-per-inode
Specify the bytes/inode ratio. mke2fs creates an inode for every bytes-per-inode bytes of space on the disk. The larger the bytes-per-
inode ratio, the fewer inodes will be created. This value generally shouldnât be smaller than the blocksize of the filesystem, since in
that case more inodes would be made than can ever be used. Be warned that it is not possible to expand the number of inodes on a
filesystem after it is created, so be careful deciding the correct value for this parameter.

-I inode-size
Specify the size of each inode in bytes. mke2fs creates 256-byte inodes by default. In kernels after 2.6.10 and some earlier vendor
kernels it is possible to utilize inodes larger than 128 bytes to store extended attributes for improved performance. The inode-size
value must be a power of 2 larger or equal to 128. The larger the inode-size the more space the inode table will consume, and this
reduces the usable space in the filesystem and can also negatively impact performance. Extended attributes stored in large inodes are
not visible with older kernels, and such filesystems will not be mountable with 2.4 kernels at all. It is not possible to change this
value after the filesystem is created.

Needless to say that this cannot be done while users are connected, you'll have to be in single user mode. Also keep in mind that if you set a smaller inode size in combination with a newer kernel it might influence performance in a negative way.

Kind regards,

Eric

EricTRA · 09-16-2010, 04:28 AM

Hello,

Before taking this 'drastic' step, have you looked what's occupying the space? If it's not a logfile growing insane out of proportion? Did you cleanup the /var? Those are at least the things I'd look at before rebuilding the file-system. First look for the reason why it's getting filled, then try to solve that problem. Maybe logging is configured poorly, maybe something else. I've got 18 Linux servers at this moment that I administer and since I started working here never had to rebuild a file-system because of lack of inodes available.

Kind regards,

Eric

pwave70 · 09-16-2010, 04:40 AM

Hello,

You're right, it's first time when a I have this problem.

I have some Linux server with slackware, and I never have this problem.

I'll check system log.

On this server I have runing: samba, nfs, mysql, apache, cups.
Maybe one of them generating unnecessary logs.
I'll try watch the /var/log

Thanks a lot for Your help

Best regards

Piotrek

EricTRA · 09-16-2010, 04:48 AM

Hi Piotrek,

You're welcome. That's what LQ is all about, helping each other. You can also have a look at how logrotate is configured, if the logs are rotated correctly and no needless logs are saved. I once had an issue with a log that was saved to a 'strange' location because it was configured like that by my predecessor and he didn't document it. By the time I found it out it was +5 Gb in size. Just to show that you'll not only need to look at the logs, but also if the are growing beyond proportions and why.

If you can avoid having to rebuild your filesystem this way then that's all for the better.

Please keep us posted on your progress.

Kind regards,

Eric

pwave70 · 09-16-2010, 05:01 AM

Ok Eric,

I've just removed some logs from samba, cups, an old logs.

But inodes 'very slow go to be small'

Below some info:

Quote:

# df -i
System plików iwęzły użyteI wolneI %uż.I zamont. na
/dev/sda10 1279632 1278243 1389 100% /var

logrotate.conf:

Quote:

# cat log
login.defs logrotate.conf logrotate.d/ logwatch/
[root@pomocnik etc]# cat logrotate.conf
# see "man logrotate" for details
# rotate log files weekly
weekly

# keep 4 weeks worth of backlogs
rotate 4

# create new (empty) log files after rotating old ones
create

# use date as a suffix of the rotated file
dateext

# uncomment this if you want your log files compressed
#compress

# RPM packages drop log rotation information into this directory
include /etc/logrotate.d

# no packages own wtmp and btmp -- we'll rotate them here
/var/log/wtmp {
monthly
create 0664 root utmp
minsize 1M
rotate 1
}

/var/log/btmp {
missingok
monthly
create 0600 root utmp
rotate 1
}

# system-specific logs may be also be configured here.

smb.conf:

Quote:

[GLOBAL]
...
log level = 3
log file = /var/log/samba/log.%m
max log size = 50
...

Hmm, how to check wich files are growing in /var location?

regards,

Piotrek

EricTRA · 09-16-2010, 05:22 AM

Hi,

You could also delete the .gz files in /var/log without problems if they are occupying to much space.

Just check on regular times which log is growing with ls -al or using the du command.

Kind regards,

Eric

pwave70 · 09-16-2010, 06:01 AM

Hello,

I don't have .gz files in log, but I found the reason! (supouse) -> SARG's reports.

In right now I delete them and inodes go dwon very fast

Eric, du command helps me very.

Thanks a lot

I'm very greatful for Your advices

Best regards

Piotrek

EricTRA · 09-16-2010, 06:23 AM

Quote:

Originally Posted by pwave70

Hello,

I don't have .gz files in log, but I found the reason! (supouse) -> SARG's reports.

In right now I delete them and inodes go dwon very fast

Eric, du command helps me very.

Thanks a lot

I'm very greatful for Your advices

Best regards

Piotrek

Hi,

Looks like you don't have compression set in logrotate so you should have

Code:

logname.1
logname.2

and so on if you're logfiles are growing in size and/or age.

I'm glad you found the reason since cleaning up these things saves you the trouble of recreating your filesystem.

You're welcome for the advice and I'm glad you got it solved.

Have fun with Linux.

Kind regards,

Eric