LinuxQuestions.org
Share your knowledge at the LQ Wiki.
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - Software
User Name
Password
Linux - Software This forum is for Software issues.
Having a problem installing a new program? Want to know which application is best for the job? Post your question in this forum.

Notices


Reply
  Search this Thread
Old 05-16-2016, 11:42 AM   #1
AliceXES
LQ Newbie
 
Registered: May 2016
Location: Romania
Posts: 2

Rep: Reputation: 0
Too many inodes trying to resize partition


Hi,

I have an ext4 partition of about 55TB, with 3,690,987,520 inodes.
Now I want to resize it to about 85TB (and probably more in the future).

There are about 62,669,735 inodes used, most of them are files, in a directory structure with about 1000 files in each directory.
Also there are somewhere between 1 and 25 hard links for each file, in other directories.

If I try to resize the partition, I get over 2^32 inodes.
I've tried to copy the files to another volume but the speed is awful, about 10-15MB/s - I've tried cp/rsync/tar/mc/etc.

The source disk is 100% used in iostat and the destination is 10-15% used.
Source:
dd if=/dev/data-vol/data of=/dev/null bs=16M count=10000 iflag=direct
10000+0 records in
10000+0 records out
167772160000 bytes (168 GB) copied, 223.263 s, 751 MB/s

Any ideeas?

Thank you!
 
Old 05-16-2016, 11:48 AM   #2
Wells
Member
 
Registered: Nov 2004
Location: Florida, USA
Distribution: Debian, Redhat
Posts: 417

Rep: Reputation: 53
Unfortunately you are probably going to have to live with that slow transfer speed if you use any copy utilities. You are running into the problem of a huge number of small files (average size for those files I would guess to be about 15kb... very small files, these days)

You are not alone in this problem. A lot of people dealing with large file systems are having to deal with the problem of lots of small files clogging up the filesystem and making it very inefficient. (If you think 55tb of data is bad, I have to deal with a filesystem that is currently in the 900tb used range, and we are going through a transfer as well...it is NOT good.)
 
Old 05-16-2016, 06:04 PM   #3
AliceXES
LQ Newbie
 
Registered: May 2016
Location: Romania
Posts: 2

Original Poster
Rep: Reputation: 0
Actually, the files are not that small, they are about 700-800KB.

I don't need to copy the files, I need to grow the partition without increasing the number of inodes... Or a way to decrease the number of inodes without affecting the existing data.
 
Old 05-16-2016, 08:48 PM   #4
jpollard
Senior Member
 
Registered: Dec 2012
Location: Washington DC area
Distribution: Fedora, CentOS, Slackware
Posts: 4,912

Rep: Reputation: 1513Reputation: 1513Reputation: 1513Reputation: 1513Reputation: 1513Reputation: 1513Reputation: 1513Reputation: 1513Reputation: 1513Reputation: 1513Reputation: 1513
Quote:
Originally Posted by AliceXES View Post
Actually, the files are not that small, they are about 700-800KB.

I don't need to copy the files, I need to grow the partition without increasing the number of inodes... Or a way to decrease the number of inodes without affecting the existing data.
Those ARE small files... They start getting "large" when you pass 3-5 MB in size.

And as long as you require one inode per file you can't reduce the number of inodes.

another problem you have (and why the transfer is slow), is that all the metadata allocations for each file (plus the directories) eats into your transfer time.

How about a different approach?

Using the overlayfs may work for you (I used UnionFS as that was available at the time and overlayfs was not). What is done is to mount the original filesystem read only, then using the stackable feature of overlayfs you add an empty read write filesystem on top.

It is possible to cause any file updates to copy a referenced file to the upper level filesystem before the update is applied. Normally this is only done when writing a file, but I believe the option exists to copy the file when inode access dates are updated - thus automatically copying files while online, and it is done in the background. The reason it is not usually done on read is to maintain a slightly higher throughput (there is a short delay while the file gets copied).

New files only go to the read/write filesystem mounted on top - thus all new data is already moved.

After it has been in use for a while, the files most recently used will have been copied, and the rate of copies will decrease. It is then possible to use an rsync to copy files not being referenced (or a crafted find that touches the files which would copy the files).

Some introductory documentation on overlayfs is at
https://www.kernel.org/doc/Documenta.../overlayfs.txt

To complete the copy does require a pass to access all files (if using the overlay) but accessing the two filesystems independantly (copy only when a file doesn't exist) can be done. The advantage this has is that you can keep from saturating the I/O by throttling the rate of files being copied (use a find to create a list of candidates, then copy batches of them when things are idle).

Even though the total copy still takes a long time, it is possible to use the combined filesystems while on-line, and making the copies a more background load on the system, and having a lower impact on users...

One last advantage this has is that the original filesystem (being read only) can be treated as a backup.

Last edited by jpollard; 05-16-2016 at 09:03 PM.
 
Old 05-16-2016, 09:48 PM   #5
rknichols
Senior Member
 
Registered: Aug 2009
Distribution: CentOS
Posts: 4,592

Rep: Reputation: 2114Reputation: 2114Reputation: 2114Reputation: 2114Reputation: 2114Reputation: 2114Reputation: 2114Reputation: 2114Reputation: 2114Reputation: 2114Reputation: 2114
Quote:
Originally Posted by AliceXES View Post
I don't need to copy the files, I need to grow the partition without increasing the number of inodes... Or a way to decrease the number of inodes without affecting the existing data.
You can't. The ratio of bytes of filesystem space per inode is fixed at the time the filesystem is created and cannot be changed. If, as it appears, you created the filesystem with a ratio of 16KiB per inode, that sets an upper limit of about 70TB for the filesystem size with 32-bit inode numbers (the only option for ext4 -- xfs defaults to 64-bit inode numbers).

And it doesn't make any difference how many inodes are in use. You get them whether you use them or not.

Last edited by rknichols; 05-16-2016 at 09:50 PM.
 
  


Reply


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off



Similar Threads
Thread Thread Starter Forum Replies Last Post
[SOLVED] showing no space available in partition even 2gb and free inodes are avilable gautam145 Linux - Newbie 1 12-20-2014 07:51 AM
[SOLVED] Increasing number of inodes without re-creating the partition Turbocapitalist Linux - General 3 07-14-2011 02:46 PM
add inodes to a partition or disk. mariogarcia Linux - Hardware 2 02-22-2007 11:34 PM
How to disable resize inodes FC5 ? taylorkh Fedora 2 11-18-2006 05:11 PM

LinuxQuestions.org > Forums > Linux Forums > Linux - Software

All times are GMT -5. The time now is 04:23 AM.

Main Menu
Advertisement
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
Open Source Consulting | Domain Registration