running out of inodes

drManhattan · 05-10-2011, 04:43 PM

Hi

I have the following issue related to inode quantity on ext3 file system.
Structure of /home file system looks like there is a lot of small files.
I have free space on my file system but don't have free inodes and I can't write to it.

Can I expand inodes on this file system (/home) ?

Output from 'df -h' cmd.

Code:

[root@redhat5 ~]# df -h
Filesystem            Size  Used Avail Use% Mounted on
/dev/mapper/rootvg-rootlv
                      2.3G  260M  1.9G  13% /
/dev/mapper/rootvg-tmplv
                      496M   19M  452M   4% /tmp
/dev/mapper/rootvg-varlv
                      496M   60M  411M  13% /var
/dev/mapper/rootvg-homelv
                      2.0G  101M  1.8G   6% /home
/dev/mapper/rootvg-usrlv
                      2.0G  1.7G  242M  88% /usr
/dev/sda1             190M   15M  166M   8% /boot
tmpfs                 188M     0  188M   0% /dev/shm

Output from 'df -i' cmd.

Code:

[root@redhat5 ~]# df -i
Filesystem            Inodes   IUsed   IFree IUse% Mounted on
/dev/mapper/rootvg-rootlv
                      598272    4762  593510    1% /
/dev/mapper/rootvg-tmplv
                      131072      17  131055    1% /tmp
/dev/mapper/rootvg-varlv
                      131072     591  130481    1% /var
/dev/mapper/rootvg-homelv
                      524288  524288       0  100% /home
/dev/mapper/rootvg-usrlv
                      524288   79237  445051   16% /usr
/dev/sda1              50200      34   50166    1% /boot
tmpfs                  48053       1   48052    1% /dev/shm

Output from 'mount' cmd. File system /home is ext3 type.

Code:

[root@redhat5 ~]# mount
/dev/mapper/rootvg-rootlv on / type ext3 (rw)
proc on /proc type proc (rw)
sysfs on /sys type sysfs (rw)
devpts on /dev/pts type devpts (rw,gid=5,mode=620)
/dev/mapper/rootvg-tmplv on /tmp type ext3 (rw)
/dev/mapper/rootvg-varlv on /var type ext3 (rw)
/dev/mapper/rootvg-homelv on /home type ext3 (rw)
/dev/mapper/rootvg-usrlv on /usr type ext3 (rw)
/dev/sda1 on /boot type ext3 (rw)
tmpfs on /dev/shm type tmpfs (rw)
none on /proc/sys/fs/binfmt_misc type binfmt_misc (rw)
sunrpc on /var/lib/nfs/rpc_pipefs type rpc_pipefs (rw)

I get the following error when try to create a new file.

Code:

[root@redhat5 jakubn]# touch file2.txt
touch: cannot touch `file2.txt': No space left on device

thx for help.

vikas027 · 05-10-2011, 07:24 PM

Hi,

I think there are many files (probably few bytes, may be of zero byte) in any of the home directories of users which has fully used the inodes of the /home file system.

In order to save yourself always remember to have quota enabled.

Anyways, try to use the below commands to find any files which you feel would be junk.

Code:

find . -size 0  # looks for zero size files
OR
find . -size -1k # looks for files less than 1 KB

TobiSGD · 05-10-2011, 07:52 PM

As far as I know it is not possible the change the amount of inodes. You have to backup your files and reformat with a higher amount of inodes if you want to keep a big number of small files.

chrism01 · 05-10-2011, 08:44 PM

I think he's right; there's no 'amend inode cnt' option in http://linux.die.net/man/8/tune2fs, so you'll have to backup your stuff & use http://linux.die.net/man/8/mkfs.ext3

drManhattan · 05-11-2011, 06:58 AM

I understand now and made some testing to make sure.
I can expand ext3 filesystem online by using 'resize2fs' cmd.

Tests relate /share file system.

Code:

[root@centos11 /]# df -h
Filesystem            Size  Used Avail Use% Mounted on
/dev/hda1             7.3G  4.1G  2.9G  60% /
tmpfs                 252M     0  252M   0% /dev/shm
/dev/mapper/rootvg-storage
                      3.9G   72M  3.8G   2% /share

Content of /share fs

Code:

[root@centos11 share]# ls -la
total 32
drwxr-xr-x  3 root root  4096 May 11 16:31 .
drwxr-xr-x 26 root root  4096 May 11 15:35 ..
drwx------  2 root root 16384 May 11 16:27 lost+found
-rw-r--r--  1 root root    18 May 11 16:31 test_file.txt

Expand of LVM responsible for /share fs.

Code:

[root@centos11 /]# lvextend -L+1G /dev/rootvg/share
  Logical volume share not found in volume group rootvg

Still /share fs not resized only lvm resized.

Code:

[root@centos11 /]# df -h
Filesystem            Size  Used Avail Use% Mounted on
/dev/hda1             7.3G  4.1G  2.9G  60% /
tmpfs                 252M     0  252M   0% /dev/shm
/dev/mapper/rootvg-storage
                      3.9G   72M  3.8G   2% /share

Now resizing fs /share online without unmounting.

Code:

[root@centos11 /]# resize2fs -p /dev/mapper/rootvg-storage
resize2fs 1.39 (29-May-2006)
Filesystem at /dev/mapper/rootvg-storage is mounted on /share; on-line resizing required
Performing an on-line resize of /dev/mapper/rootvg-storage to 1286144 (4k) blocks.
The filesystem on /dev/mapper/rootvg-storage is now 1286144 blocks long.

Content of /share fs after expanding storage, the old structure remains the same, only space is bigger on this fs.

Code:

[root@centos11 /]# ls -la /share
total 32
drwxr-xr-x  3 root root  4096 May 11 16:31 .
drwxr-xr-x 26 root root  4096 May 11 15:35 ..
drwx------  2 root root 16384 May 11 16:27 lost+found
-rw-r--r--  1 root root    18 May 11 16:31 test_file.txt

Display inode quantity of /share fs.

Code:

[root@centos11 /]# df -i
Filesystem            Inodes   IUsed   IFree IUse% Mounted on
/dev/hda1            1966080   40752 1925328    3% /
tmpfs                  64364       1   64363    1% /dev/shm
/dev/mapper/rootvg-storage
                      750080      12  750068    1% /share

If I wanted to exand inode quantity I had to backup files from /share fs and recreacte fs with additional parameter -N <inode_quantity>, which tells mkfs cmd to create desired by me inode quantity.

Code:

[root@centos11 /]# mkfs -t ext3 -m 1 -v -N 800000  /dev/rootvg/storage
mke2fs 1.39 (29-May-2006)
Filesystem label=
OS type: Linux
Block size=4096 (log=2)
Fragment size=4096 (log=2)
800000 inodes, 1286144 blocks
12861 blocks (1.00%) reserved for the super user
First data block=0
Maximum filesystem blocks=1317011456
40 block groups
32768 blocks per group, 32768 fragments per group
20000 inodes per group
Superblock backups stored on blocks:
        32768, 98304, 163840, 229376, 294912, 819200, 884736

Writing inode tables: done
Creating journal (32768 blocks): done
Writing superblocks and filesystem accounting information: done

This filesystem will be automatically checked every 36 mounts or
180 days, whichever comes first.  Use tune2fs -c or -i to override.

After reformatiing with desired inode quantity entire fs content is lost.

Code:

[root@centos11 share]# ls -la /share
total 28
drwxr-xr-x  3 root root  4096 May 11 16:46 .
drwxr-xr-x 26 root root  4096 May 11 15:35 ..
drwx------  2 root root 16384 May 11 16:46 lost+found

But inode quantity is bigger from 750080 to 800000 nr.

Code:

[root@centos11 share]# df -i
Filesystem            Inodes   IUsed   IFree IUse% Mounted on
/dev/hda1            1966080   40752 1925328    3% /
tmpfs                  64364       1   64363    1% /dev/shm
/dev/mapper/rootvg-storage
                      800000      11  799989    1% /share

I was wonderingt is there other file system type that allows expanding inode quantity on the fly and without reformatting entire fs?

thx for help.

drManhattan · 05-11-2011, 01:19 PM

I don't understand how to calc inodes to bytes storage.

Knowledge :
------------
When ext3 file system is created, specific number of inodes is created.

Questions :
-----------
- what decides how many inodes will be created on file system if not specified ?

thx for help.

vikas027 · 05-11-2011, 02:32 PM

Quote:

Originally Posted by drManhattan

I don't understand how to calc inodes to bytes storage.

Knowledge :
------------
When ext3 file system is created, specific number of inodes is created.

Questions :
-----------
- what decides how many inodes will be created on file system if not specified ?

thx for help.

Generally, we do not need to define inodes at the time of creation of filesystem.

But, if in case you feel like you can try something like the below mke2fs command.

Code:

mke2fs -N 10000 /dev/sda1

Here, 10000 are the number of Inodes.

But, I still doubt this would solve you issue, as if there is some script/process which is continuously creating files (and thus consuming lnodes) will again haunt you.

By the way, did you tried my suggestion posted here.

chrism01 · 05-11-2011, 06:54 PM

Have a more in depth read of http://linux.die.net/man/8/mkfs.ext3, with particular ref to -b, -i, -O, -T options. See also the refs to the relevant conf file.

drManhattan · 05-12-2011, 04:18 AM

'chrism01' I understand mkfs command but have two question.
From quote below I don't understand the difference between -d vs -I.

Quote:

-b block-size
Specify the size of blocks in bytes. Valid block size vales are 1024, 2048 and 4096 bytes per block. If omitted, mke2fs block-size is heuristically determined by the file system size and the expected usage of the filesystem (see the -T option). If block-size is negative, then mke2fs will use heuristics to determine the appropriate block size, with the constraint that the block size will be at least block-size bytes. This is useful for certain hardware devices which require that the blocksize be a multiple of 2k.

-i bytes-per-inode
Specify the bytes/inode ratio. mke2fs creates an inode for every bytes-per-inode bytes of space on the disk. The larger the bytes-per-inode ratio, the fewer inodes will be created. This value generally shouldn't be smaller than the blocksize of the filesystem, since then too many inodes will be made. Be warned that is not possible to expand the number of inodes on a filesystem after it is created, so be careful deciding the correct value for this parameter.

Questions :
-----------
1. If I create fs with this cmd `mkfs -t ext3 -N 700000 -d 2048 /dev/mapper/rootvg-homelv`, it will create 800000 inodes and block size of 2048 B ?

2. If I create fs with `mkfs -t ext3 -d 2048 -i 65536 /dev/mapper/rootvg-homelv`, does it mean that 65536 B will be block size of this fs ?

thx for help.

sundialsvcs · 05-12-2011, 09:04 AM

If you have "a tremendous number of small files," consider porting the applications in question to use a relational database, instead.

chrism01 · 05-12-2011, 09:12 PM

Quote:

-d vs -I

I guess you mean

Quote:

-b -vs -i

Please try to use the exact switch you mean (& Linux is case sensitive).

Anyway, I'd start by discovering your current block size http://www.cyberciti.biz/tips/determ...isk-quota.html. For a modern distro, I'd expect a default of 4096.
Compare this to avg size of files in qn. I'd prob not mess with this.
Then, I'd ignore -i switch and just use -N

Quote:

-N number-of-inodes
overrides the default calculation of the number of inodes that should be reserved for the filesystem (which is based on the number of blocks and the bytes-per-inode ratio). This allows the user to specify the number of desired inodes directly.

Remember, an inode is just an index entry; 1 for each file & dir you create (dir is a file in *nix)

HTH

drManhattan · 05-13-2011, 04:02 AM

Yes I meant

Quote:

-b -vs -i

savan · 05-13-2011, 09:54 AM

"man proc" on my RHEL 5.3 x32bit VM shows:

/proc/sys/fs/inode-max
This file contains the maximum number of in-memory inodes. On
some (2.4) systems, it may not be present. This value should be
3-4 times larger than the value in file-max, since stdin, stdout
and network sockets also need an inode to handle them. When you
regularly run out of inodes, you need to increase this value.

and:

/proc/sys/fs/inode-state
This file contains seven numbers: nr_inodes, nr_free_inodes,
preshrink and four dummy values. nr_inodes is the number of
inodes the system has allocated. This can be slightly more than
inode-max because Linux allocates them one page full at a time.
nr_free_inodes represents the number of free inodes. preshrink
is non-zero when the nr_inodes > inode-max and the system needs
to prune the inode list instead of allocating more.

So you could use adjust this using sysctl for persistence or just change it on the fly to get you by until you can address the underlying issue as suggested above? Haven't tried it to know if you can do it on the fly and whatnot...Just a thought...

The /proc/fs/inode-max file doesn't exist on mine though, but it is a 2.6 kernel, maybe it's outdated or not applicable...?