LinuxQuestions.org
Visit Jeremy's Blog.
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - Newbie
User Name
Password
Linux - Newbie This Linux forum is for members that are new to Linux.
Just starting out and have a question? If it is not in the man pages or the how-to's this is the place!

Notices


Reply
  Search this Thread
Old 05-10-2011, 04:43 PM   #1
drManhattan
Member
 
Registered: Aug 2010
Location: Warsaw, Poland
Distribution: Ubuntu 10.04 LTS on IBM Lenovo R61e, RHEL5-6,SLES10-11
Posts: 262

Rep: Reputation: 1
running out of inodes


Hi

I have the following issue related to inode quantity on ext3 file system.
Structure of /home file system looks like there is a lot of small files.
I have free space on my file system but don't have free inodes and I can't write to it.

Can I expand inodes on this file system (/home) ?

Output from 'df -h' cmd.
Code:
[root@redhat5 ~]# df -h
Filesystem            Size  Used Avail Use% Mounted on
/dev/mapper/rootvg-rootlv
                      2.3G  260M  1.9G  13% /
/dev/mapper/rootvg-tmplv
                      496M   19M  452M   4% /tmp
/dev/mapper/rootvg-varlv
                      496M   60M  411M  13% /var
/dev/mapper/rootvg-homelv
                      2.0G  101M  1.8G   6% /home
/dev/mapper/rootvg-usrlv
                      2.0G  1.7G  242M  88% /usr
/dev/sda1             190M   15M  166M   8% /boot
tmpfs                 188M     0  188M   0% /dev/shm
Output from 'df -i' cmd.
Code:
[root@redhat5 ~]# df -i
Filesystem            Inodes   IUsed   IFree IUse% Mounted on
/dev/mapper/rootvg-rootlv
                      598272    4762  593510    1% /
/dev/mapper/rootvg-tmplv
                      131072      17  131055    1% /tmp
/dev/mapper/rootvg-varlv
                      131072     591  130481    1% /var
/dev/mapper/rootvg-homelv
                      524288  524288       0  100% /home
/dev/mapper/rootvg-usrlv
                      524288   79237  445051   16% /usr
/dev/sda1              50200      34   50166    1% /boot
tmpfs                  48053       1   48052    1% /dev/shm
Output from 'mount' cmd. File system /home is ext3 type.
Code:
[root@redhat5 ~]# mount
/dev/mapper/rootvg-rootlv on / type ext3 (rw)
proc on /proc type proc (rw)
sysfs on /sys type sysfs (rw)
devpts on /dev/pts type devpts (rw,gid=5,mode=620)
/dev/mapper/rootvg-tmplv on /tmp type ext3 (rw)
/dev/mapper/rootvg-varlv on /var type ext3 (rw)
/dev/mapper/rootvg-homelv on /home type ext3 (rw)
/dev/mapper/rootvg-usrlv on /usr type ext3 (rw)
/dev/sda1 on /boot type ext3 (rw)
tmpfs on /dev/shm type tmpfs (rw)
none on /proc/sys/fs/binfmt_misc type binfmt_misc (rw)
sunrpc on /var/lib/nfs/rpc_pipefs type rpc_pipefs (rw)
I get the following error when try to create a new file.
Code:
[root@redhat5 jakubn]# touch file2.txt
touch: cannot touch `file2.txt': No space left on device
thx for help.

Last edited by drManhattan; 05-10-2011 at 04:51 PM.
 
Old 05-10-2011, 07:24 PM   #2
vikas027
Senior Member
 
Registered: May 2007
Location: Sydney
Distribution: RHEL, CentOS, Ubuntu, Debian, OS X
Posts: 1,305

Rep: Reputation: 107Reputation: 107
Hi,

I think there are many files (probably few bytes, may be of zero byte) in any of the home directories of users which has fully used the inodes of the /home file system.

In order to save yourself always remember to have quota enabled.

Anyways, try to use the below commands to find any files which you feel would be junk.
Code:
find . -size 0  # looks for zero size files
OR
find . -size -1k # looks for files less than 1 KB

Last edited by vikas027; 05-10-2011 at 07:24 PM. Reason: Corrected typo error
 
Old 05-10-2011, 07:52 PM   #3
TobiSGD
Moderator
 
Registered: Dec 2009
Location: Germany
Distribution: Whatever fits the task best
Posts: 17,148
Blog Entries: 2

Rep: Reputation: 4886Reputation: 4886Reputation: 4886Reputation: 4886Reputation: 4886Reputation: 4886Reputation: 4886Reputation: 4886Reputation: 4886Reputation: 4886Reputation: 4886
As far as I know it is not possible the change the amount of inodes. You have to backup your files and reformat with a higher amount of inodes if you want to keep a big number of small files.
 
1 members found this post helpful.
Old 05-10-2011, 08:44 PM   #4
chrism01
LQ Guru
 
Registered: Aug 2004
Location: Sydney
Distribution: Rocky 9.2
Posts: 18,344

Rep: Reputation: 2746Reputation: 2746Reputation: 2746Reputation: 2746Reputation: 2746Reputation: 2746Reputation: 2746Reputation: 2746Reputation: 2746Reputation: 2746Reputation: 2746
I think he's right; there's no 'amend inode cnt' option in http://linux.die.net/man/8/tune2fs, so you'll have to backup your stuff & use http://linux.die.net/man/8/mkfs.ext3
 
1 members found this post helpful.
Old 05-11-2011, 06:58 AM   #5
drManhattan
Member
 
Registered: Aug 2010
Location: Warsaw, Poland
Distribution: Ubuntu 10.04 LTS on IBM Lenovo R61e, RHEL5-6,SLES10-11
Posts: 262

Original Poster
Rep: Reputation: 1
I understand now and made some testing to make sure.
I can expand ext3 filesystem online by using 'resize2fs' cmd.

Tests relate /share file system.
Code:
[root@centos11 /]# df -h
Filesystem            Size  Used Avail Use% Mounted on
/dev/hda1             7.3G  4.1G  2.9G  60% /
tmpfs                 252M     0  252M   0% /dev/shm
/dev/mapper/rootvg-storage
                      3.9G   72M  3.8G   2% /share
Content of /share fs
Code:
[root@centos11 share]# ls -la
total 32
drwxr-xr-x  3 root root  4096 May 11 16:31 .
drwxr-xr-x 26 root root  4096 May 11 15:35 ..
drwx------  2 root root 16384 May 11 16:27 lost+found
-rw-r--r--  1 root root    18 May 11 16:31 test_file.txt
Expand of LVM responsible for /share fs.
Code:
[root@centos11 /]# lvextend -L+1G /dev/rootvg/share
  Logical volume share not found in volume group rootvg
Still /share fs not resized only lvm resized.
Code:
[root@centos11 /]# df -h
Filesystem            Size  Used Avail Use% Mounted on
/dev/hda1             7.3G  4.1G  2.9G  60% /
tmpfs                 252M     0  252M   0% /dev/shm
/dev/mapper/rootvg-storage
                      3.9G   72M  3.8G   2% /share
Now resizing fs /share online without unmounting.
Code:
[root@centos11 /]# resize2fs -p /dev/mapper/rootvg-storage
resize2fs 1.39 (29-May-2006)
Filesystem at /dev/mapper/rootvg-storage is mounted on /share; on-line resizing required
Performing an on-line resize of /dev/mapper/rootvg-storage to 1286144 (4k) blocks.
The filesystem on /dev/mapper/rootvg-storage is now 1286144 blocks long.
Content of /share fs after expanding storage, the old structure remains the same, only space is bigger on this fs.
Code:
[root@centos11 /]# ls -la /share
total 32
drwxr-xr-x  3 root root  4096 May 11 16:31 .
drwxr-xr-x 26 root root  4096 May 11 15:35 ..
drwx------  2 root root 16384 May 11 16:27 lost+found
-rw-r--r--  1 root root    18 May 11 16:31 test_file.txt
Display inode quantity of /share fs.
Code:
[root@centos11 /]# df -i
Filesystem            Inodes   IUsed   IFree IUse% Mounted on
/dev/hda1            1966080   40752 1925328    3% /
tmpfs                  64364       1   64363    1% /dev/shm
/dev/mapper/rootvg-storage
                      750080      12  750068    1% /share
If I wanted to exand inode quantity I had to backup files from /share fs and recreacte fs with additional parameter -N <inode_quantity>, which tells mkfs cmd to create desired by me inode quantity.
Code:
[root@centos11 /]# mkfs -t ext3 -m 1 -v -N 800000  /dev/rootvg/storage
mke2fs 1.39 (29-May-2006)
Filesystem label=
OS type: Linux
Block size=4096 (log=2)
Fragment size=4096 (log=2)
800000 inodes, 1286144 blocks
12861 blocks (1.00%) reserved for the super user
First data block=0
Maximum filesystem blocks=1317011456
40 block groups
32768 blocks per group, 32768 fragments per group
20000 inodes per group
Superblock backups stored on blocks:
        32768, 98304, 163840, 229376, 294912, 819200, 884736

Writing inode tables: done
Creating journal (32768 blocks): done
Writing superblocks and filesystem accounting information: done

This filesystem will be automatically checked every 36 mounts or
180 days, whichever comes first.  Use tune2fs -c or -i to override.
After reformatiing with desired inode quantity entire fs content is lost.
Code:
[root@centos11 share]# ls -la /share
total 28
drwxr-xr-x  3 root root  4096 May 11 16:46 .
drwxr-xr-x 26 root root  4096 May 11 15:35 ..
drwx------  2 root root 16384 May 11 16:46 lost+found
But inode quantity is bigger from 750080 to 800000 nr.
Code:
[root@centos11 share]# df -i
Filesystem            Inodes   IUsed   IFree IUse% Mounted on
/dev/hda1            1966080   40752 1925328    3% /
tmpfs                  64364       1   64363    1% /dev/shm
/dev/mapper/rootvg-storage
                      800000      11  799989    1% /share


I was wonderingt is there other file system type that allows expanding inode quantity on the fly and without reformatting entire fs?


thx for help.

Last edited by drManhattan; 05-11-2011 at 09:50 AM.
 
Old 05-11-2011, 01:19 PM   #6
drManhattan
Member
 
Registered: Aug 2010
Location: Warsaw, Poland
Distribution: Ubuntu 10.04 LTS on IBM Lenovo R61e, RHEL5-6,SLES10-11
Posts: 262

Original Poster
Rep: Reputation: 1
I don't understand how to calc inodes to bytes storage.

Knowledge :
------------
When ext3 file system is created, specific number of inodes is created.

Questions :
-----------
- what decides how many inodes will be created on file system if not specified ?

thx for help.

Last edited by drManhattan; 05-11-2011 at 01:43 PM.
 
Old 05-11-2011, 02:32 PM   #7
vikas027
Senior Member
 
Registered: May 2007
Location: Sydney
Distribution: RHEL, CentOS, Ubuntu, Debian, OS X
Posts: 1,305

Rep: Reputation: 107Reputation: 107
Quote:
Originally Posted by drManhattan View Post
I don't understand how to calc inodes to bytes storage.

Knowledge :
------------
When ext3 file system is created, specific number of inodes is created.

Questions :
-----------
- what decides how many inodes will be created on file system if not specified ?

thx for help.
Generally, we do not need to define inodes at the time of creation of filesystem.

But, if in case you feel like you can try something like the below mke2fs command.
Code:
mke2fs -N 10000 /dev/sda1
Here, 10000 are the number of Inodes.

But, I still doubt this would solve you issue, as if there is some script/process which is continuously creating files (and thus consuming lnodes) will again haunt you.


By the way, did you tried my suggestion posted here.

Last edited by vikas027; 05-11-2011 at 02:33 PM. Reason: Hyperlinked mke2fs
 
Old 05-11-2011, 06:54 PM   #8
chrism01
LQ Guru
 
Registered: Aug 2004
Location: Sydney
Distribution: Rocky 9.2
Posts: 18,344

Rep: Reputation: 2746Reputation: 2746Reputation: 2746Reputation: 2746Reputation: 2746Reputation: 2746Reputation: 2746Reputation: 2746Reputation: 2746Reputation: 2746Reputation: 2746
Have a more in depth read of http://linux.die.net/man/8/mkfs.ext3, with particular ref to -b, -i, -O, -T options. See also the refs to the relevant conf file.
 
Old 05-12-2011, 04:18 AM   #9
drManhattan
Member
 
Registered: Aug 2010
Location: Warsaw, Poland
Distribution: Ubuntu 10.04 LTS on IBM Lenovo R61e, RHEL5-6,SLES10-11
Posts: 262

Original Poster
Rep: Reputation: 1
'chrism01' I understand mkfs command but have two question.
From quote below I don't understand the difference between -d vs -I.

Quote:
-b block-size
Specify the size of blocks in bytes. Valid block size vales are 1024, 2048 and 4096 bytes per block. If omitted, mke2fs block-size is heuristically determined by the file system size and the expected usage of the filesystem (see the -T option). If block-size is negative, then mke2fs will use heuristics to determine the appropriate block size, with the constraint that the block size will be at least block-size bytes. This is useful for certain hardware devices which require that the blocksize be a multiple of 2k.

-i bytes-per-inode
Specify the bytes/inode ratio. mke2fs creates an inode for every bytes-per-inode bytes of space on the disk. The larger the bytes-per-inode ratio, the fewer inodes will be created. This value generally shouldn't be smaller than the blocksize of the filesystem, since then too many inodes will be made. Be warned that is not possible to expand the number of inodes on a filesystem after it is created, so be careful deciding the correct value for this parameter.
Questions :
-----------
1. If I create fs with this cmd `mkfs -t ext3 -N 700000 -d 2048 /dev/mapper/rootvg-homelv`, it will create 800000 inodes and block size of 2048 B ?

2. If I create fs with `mkfs -t ext3 -d 2048 -i 65536 /dev/mapper/rootvg-homelv`, does it mean that 65536 B will be block size of this fs ?

thx for help.
 
Old 05-12-2011, 09:04 AM   #10
sundialsvcs
LQ Guru
 
Registered: Feb 2004
Location: SE Tennessee, USA
Distribution: Gentoo, LFS
Posts: 10,599
Blog Entries: 4

Rep: Reputation: 3905Reputation: 3905Reputation: 3905Reputation: 3905Reputation: 3905Reputation: 3905Reputation: 3905Reputation: 3905Reputation: 3905Reputation: 3905Reputation: 3905
If you have "a tremendous number of small files," consider porting the applications in question to use a relational database, instead.
 
Old 05-12-2011, 09:12 PM   #11
chrism01
LQ Guru
 
Registered: Aug 2004
Location: Sydney
Distribution: Rocky 9.2
Posts: 18,344

Rep: Reputation: 2746Reputation: 2746Reputation: 2746Reputation: 2746Reputation: 2746Reputation: 2746Reputation: 2746Reputation: 2746Reputation: 2746Reputation: 2746Reputation: 2746
Quote:
-d vs -I
I guess you mean
Quote:
-b -vs -i
Please try to use the exact switch you mean (& Linux is case sensitive).

Anyway, I'd start by discovering your current block size http://www.cyberciti.biz/tips/determ...isk-quota.html. For a modern distro, I'd expect a default of 4096.
Compare this to avg size of files in qn. I'd prob not mess with this.
Then, I'd ignore -i switch and just use -N
Quote:
-N number-of-inodes
overrides the default calculation of the number of inodes that should be reserved for the filesystem (which is based on the number of blocks and the bytes-per-inode ratio). This allows the user to specify the number of desired inodes directly.
Remember, an inode is just an index entry; 1 for each file & dir you create (dir is a file in *nix)

HTH
 
Old 05-13-2011, 04:02 AM   #12
drManhattan
Member
 
Registered: Aug 2010
Location: Warsaw, Poland
Distribution: Ubuntu 10.04 LTS on IBM Lenovo R61e, RHEL5-6,SLES10-11
Posts: 262

Original Poster
Rep: Reputation: 1
Yes I meant
Quote:
-b -vs -i
 
Old 05-13-2011, 09:54 AM   #13
savan
LQ Newbie
 
Registered: May 2011
Posts: 7

Rep: Reputation: 1
Temp. solution?

"man proc" on my RHEL 5.3 x32bit VM shows:

/proc/sys/fs/inode-max
This file contains the maximum number of in-memory inodes. On
some (2.4) systems, it may not be present. This value should be
3-4 times larger than the value in file-max, since stdin, stdout
and network sockets also need an inode to handle them. When you
regularly run out of inodes, you need to increase this value.

and:

/proc/sys/fs/inode-state
This file contains seven numbers: nr_inodes, nr_free_inodes,
preshrink and four dummy values. nr_inodes is the number of
inodes the system has allocated. This can be slightly more than
inode-max because Linux allocates them one page full at a time.
nr_free_inodes represents the number of free inodes. preshrink
is non-zero when the nr_inodes > inode-max and the system needs
to prune the inode list instead of allocating more.

So you could use adjust this using sysctl for persistence or just change it on the fly to get you by until you can address the underlying issue as suggested above? Haven't tried it to know if you can do it on the fly and whatnot...Just a thought...

The /proc/fs/inode-max file doesn't exist on mine though, but it is a 2.6 kernel, maybe it's outdated or not applicable...?
 
  


Reply


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off



Similar Threads
Thread Thread Starter Forum Replies Last Post
Inodes? ddzc Linux - Software 1 05-24-2008 09:18 PM
Had a crash and need help and info, running a fsck and clearing millions of inodes edkhosting Linux - Software 3 12-13-2005 10:12 AM
Inodes bambolin Linux - Newbie 3 04-24-2005 02:17 PM
Out of Inodes tmiles Linux - Newbie 4 02-06-2005 08:35 PM
What is inodes? ionian2k Linux - Newbie 1 12-15-2002 10:29 AM

LinuxQuestions.org > Forums > Linux Forums > Linux - Newbie

All times are GMT -5. The time now is 03:29 AM.

Main Menu
Advertisement
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
Open Source Consulting | Domain Registration