LinuxQuestions.org
Download your favorite Linux distribution at LQ ISO.
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - Newbie
User Name
Password
Linux - Newbie This Linux forum is for members that are new to Linux.
Just starting out and have a question? If it is not in the man pages or the how-to's this is the place!

Notices


Reply
  Search this Thread
Old 05-06-2011, 08:41 AM   #1
kienlarsen
Member
 
Registered: Apr 2011
Posts: 76

Rep: Reputation: 0
Linux file structure question about i-nodes and allocation


I'm on files and file systems with UNIX Academy training DVDs and I'm preparing for my first exam. I have a question to which I can't find an answer. I learned from the training and accompanying books that file name and a pointer to its actual i-node are stores in directory record. I have no problem with that, as I can clearly verify all of it.
My question is, though, most files are much larger that one i-node can accommodate, so I logically conclude that it takes many more i-nodes to store a file.
Here's the question: how file knows where rest of its parts allocated? It is not in directory records, I checked. Apparently it is not in superblock. So where is the magic place that stores all file' pieces?
 
Click here to see the post LQ members have rated as the most helpful post in this thread.
Old 05-06-2011, 09:00 AM   #2
syg00
LQ Veteran
 
Registered: Aug 2003
Location: Australia
Distribution: Lots ...
Posts: 14,834

Rep: Reputation: 1820Reputation: 1820Reputation: 1820Reputation: 1820Reputation: 1820Reputation: 1820Reputation: 1820Reputation: 1820Reputation: 1820Reputation: 1820Reputation: 1820
Try here.
 
3 members found this post helpful.
Old 05-06-2011, 03:36 PM   #3
kienlarsen
Member
 
Registered: Apr 2011
Posts: 76

Original Poster
Rep: Reputation: 0
syg00, thanks! I don't know how I missed this article on Wikipedia.
 
Old 05-07-2011, 11:26 AM   #4
kienlarsen
Member
 
Registered: Apr 2011
Posts: 76

Original Poster
Rep: Reputation: 0
Would the design like that, as I see it, create very fragmented files?

Last edited by kienlarsen; 05-07-2011 at 11:37 PM.
 
Old 05-08-2011, 09:03 AM   #5
kienlarsen
Member
 
Registered: Apr 2011
Posts: 76

Original Poster
Rep: Reputation: 0
Can somebody to advise/comment about file system fragmentation in Linux and to deal with it?
 
Old 05-08-2011, 09:29 AM   #6
carltm
Member
 
Registered: Jan 2007
Location: Canton, MI
Distribution: CentOS, SuSE, Red Hat, Debian, etc.
Posts: 703

Rep: Reputation: 97
Quote:
Originally Posted by kienlarsen View Post
My question is, though, most files are much larger that one i-node can accommodate, so I logically conclude that it takes many more i-nodes to store a file.
Here's the question: how file knows where rest of its parts allocated? It is not in directory records, I checked. Apparently it is not in superblock. So where is the magic place that stores all file' pieces?
This is not really a Linux (kernel) question, it's more a filesystem
question. Most distributions use ext2, ext3 or ext4 by default, and
these filesystems use inodes. Most other filesystems don't use them.

An inode is a reference to the first sector on the harddrive where
the file is stored. At the end of each sector is the address of the
next sector or an indicator that there are no more sectors. This means
that each file has only one inode, regardless of how many sectors are
used to store the file's contents.
 
Old 05-08-2011, 09:33 AM   #7
carltm
Member
 
Registered: Jan 2007
Location: Canton, MI
Distribution: CentOS, SuSE, Red Hat, Debian, etc.
Posts: 703

Rep: Reputation: 97
Quote:
Originally Posted by kienlarsen View Post
Can somebody to advise/comment about file system fragmentation in Linux and to deal with it?
Generally you don't need to be concerned about fragmentation on Linux
filesystems. When the kernel writes a file it automatically finds a
place that requires minimal fragmentation.
 
Old 05-09-2011, 10:08 AM   #8
kienlarsen
Member
 
Registered: Apr 2011
Posts: 76

Original Poster
Rep: Reputation: 0
I see it this way, the fragmentation must occur as different file' pieces aren't written consequently, one after another. And as far as I understand from my training, the fragmentation is inevitable as files are many, and changing in size, updating etc. The way Linux handles the writing probably improves the situation, but it is incapable of preventing the fragmentation. The fragmentation occurs.
 
Old 05-09-2011, 11:26 AM   #9
Soadyheid
Senior Member
 
Registered: Aug 2010
Location: Near Edinburgh, Scotland
Distribution: Cinnamon Mint 17.0 at present.
Posts: 1,167

Rep: Reputation: 201Reputation: 201Reputation: 201
More from Wikipedia...

See "ext3"

Quote:
DefragmentationThere is no online ext3 defragmentation tool that works on the filesystem level. An offline ext2 defragmenter, e2defrag, exists but requires that the ext3 filesystem be converted back to ext2 first. But depending on the feature bits turned on in the filesystem, e2defrag may destroy data; it does not know how to treat many of the newer ext3 features.[17]

There are userspace defragmentation tools like Shake[18] and defrag.[19][20] Shake works by allocating space for the whole file as one operation, which will generally cause the allocator to find contiguous disk space. It also tries to write files used at the same time next to each other. Defrag works by copying each file over itself. However they only work if the filesystem is reasonably empty. A true defragmentation tool does not exist for ext3.[21]

That being said, as the Linux System Administrator Guide states, "Modern Linux filesystem(s) keep fragmentation at a minimum by keeping all blocks in a file close together, even if they can't be stored in consecutive sectors. Some filesystems, like ext3, effectively allocate the free block that is nearest to other blocks in a file. Therefore it is not necessary to worry about fragmentation in a Linux system."[22]

While ext3 is more resistant to file fragmentation than the FAT filesystem, nonetheless ext3 filesystems can get fragmented over time or on specific usage patterns, like slowly-writing large files.[23][24] Consequently the successor to the ext3 filesystem, ext4, is planned to eventually include an online filesystem defragmentation utility,[25] and currently supports extents (contiguous file regions).
Play Bonny!

Last edited by Soadyheid; 05-09-2011 at 11:27 AM. Reason: Additional item
 
Old 05-09-2011, 06:09 PM   #10
kienlarsen
Member
 
Registered: Apr 2011
Posts: 76

Original Poster
Rep: Reputation: 0
Does it mean there's no standard de-fragmentation tool in Linux? So how people deal with the fragmentation, or it is just ignored? In windows it is very efficient and useful tool.
 
Old 05-09-2011, 09:00 PM   #11
frankbell
LQ Guru
 
Registered: Jan 2006
Location: Virginia, USA
Distribution: Slackware, Debian, Mint, OpenBSD
Posts: 11,340
Blog Entries: 12

Rep: Reputation: 2730Reputation: 2730Reputation: 2730Reputation: 2730Reputation: 2730Reputation: 2730Reputation: 2730Reputation: 2730Reputation: 2730Reputation: 2730Reputation: 2730
Fragmentation does not become significant enough to affect performance in native Linux file systems.

The inode system allows the kernel to find all the pieces of the file almost instantaneously. I have read (can't find the source) that, even with defragmentation, Linux file systems tend to maintain a steady, not an increasing rate of fragmentation because the file systems police themselves.

Frankly, fragmentation is not that much of an issue with NTFS, either. In FAT and FAT32, the system had to load the first block of a file to find the pointer to the second block; the second block to find the pointer to the third block; and so on. This is why fragmentation would slow performance of a machine as fragmentation increased over time.

In NTFS, my Windows 2000 instructor said that the pointers to all blocks for a file are contained in the MFT. A fragmentation map that might bring a FAT or FAT32 system to its knees will have only a marginal effect on an NTFS system--even though the map may seem scary to someone who cut his teeth on FAT, as I did--because the MFT works in the same general way as the inode system.
 
Old 05-10-2011, 02:32 PM   #12
kienlarsen
Member
 
Registered: Apr 2011
Posts: 76

Original Poster
Rep: Reputation: 0
frankbell, as I understand it from my Unix Academy DVD training, problem of fragmentation isn't that kernel has a difficulty finding i-nodes or data blocks, but that data block allocated all over the drive and takes many drive spins to read them (istead of one), so it's becoming inefficient and slow.
 
Old 05-10-2011, 08:08 PM   #13
frankbell
LQ Guru
 
Registered: Jan 2006
Location: Virginia, USA
Distribution: Slackware, Debian, Mint, OpenBSD
Posts: 11,340
Blog Entries: 12

Rep: Reputation: 2730Reputation: 2730Reputation: 2730Reputation: 2730Reputation: 2730Reputation: 2730Reputation: 2730Reputation: 2730Reputation: 2730Reputation: 2730Reputation: 2730
Quote:
Originally Posted by kienlarsen View Post
frankbell, as I understand it from my Unix Academy DVD training, problem of fragmentation isn't that kernel has a difficulty finding i-nodes or data blocks, but that data block allocated all over the drive and takes many drive spins to read them (istead of one), so it's becoming inefficient and slow.
Actually, it can be both. The longer seek times mean more drive spins during the seeking.

Fabrication is not an issue with the major Linux filesystems.

The kernel and the filesystems together police the system so that the percent of fragmentation remains well within limits and does not grow. Third party tools would be redundant.

Edit: Another way of looking at it would be to consider that fragmentation control (not elimination, but control) is built-in.

Last edited by frankbell; 05-10-2011 at 08:40 PM.
 
Old 05-11-2011, 08:38 AM   #14
kienlarsen
Member
 
Registered: Apr 2011
Posts: 76

Original Poster
Rep: Reputation: 0
How can it remain within limits? Imagine the system that has many data files (aka database) and the files often rewritten and their size is changing both ways. It is inevitable that fragmentation will occur and it will be growing. And it would be happening not because something is wrong with Linux file system, I think it is just the way it is. As I see it (may be I'm wrong), kernel has limited ability to control how and where it puts new data. Otherwise the data writing alone would become huge issue and entire disk map to be kept in memory and constantly calculate location/relocations. May be on some expensive disk controllers it would be possible (my wild guess). As I researched there are third party programs that admit and deal with fragmentation issue.

Last edited by kienlarsen; 05-11-2011 at 08:40 AM.
 
Old 05-11-2011, 09:34 PM   #15
chrism01
LQ Guru
 
Registered: Aug 2004
Location: Sydney
Distribution: Centos 6.8, Centos 5.10
Posts: 17,240

Rep: Reputation: 2324Reputation: 2324Reputation: 2324Reputation: 2324Reputation: 2324Reputation: 2324Reputation: 2324Reputation: 2324Reputation: 2324Reputation: 2324Reputation: 2324
Basically, the filesystem service called by the kernel is fairly clever and is designed to resist fragmentation in general.
Historically it's been so successful that no defrag tool has been supplied by the core code writers.
Don't forget that by default when you create a new fs on a disk it reserves 5% space (tunable) for exclusive use by root.
http://linux.die.net/man/8/mkfs.ext3

In extreme cases eg continually at 95% full and large files being created/deleted/updated, you will get some fragmentation, but by then you should allocate a bigger disk anyway.
 
1 members found this post helpful.
  


Reply


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off



Similar Threads
Thread Thread Starter Forum Replies Last Post
Convert directory structure from long file names in Linux to DOS 8.3 structure? manorina Linux - Software 5 09-12-2009 10:18 AM
Question, Apples Contribution to Open Source + MacOs file structure question Higgy3k Other *NIX 5 07-25-2005 05:23 AM
File system/structure question ashley75 Linux - General 1 05-17-2005 10:02 PM
Virtual Host file structure question cdrobsonjr Linux - Software 2 09-22-2004 09:11 PM
File structure of Linux TDEE Linux - Software 6 02-27-2003 12:10 AM


All times are GMT -5. The time now is 04:58 PM.

Main Menu
Advertisement
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
Facebook: linuxquestions Google+: linuxquestions
Open Source Consulting | Domain Registration