LinuxQuestions.org
Welcome to the most active Linux Forum on the web.
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - General
User Name
Password
Linux - General This Linux forum is for general Linux questions and discussion.
If it is Linux Related and doesn't seem to fit in any other forum then this is the place.

Notices

Reply
 
Search this Thread
Old 05-29-2008, 03:41 PM   #1
terfy
Member
 
Registered: Jan 2005
Location: Denmark
Distribution: openSUSE
Posts: 165

Rep: Reputation: 30
ext3 defrag


Hello guys..

I'm wondering about this..
Everyone says that it is not necessary to defrag ext3 filesystem.. but look at this.

"420 extents found, perfection would be 37 extents"
its a 4gig file..

why is it not necessary to defrag this file ? or any other files..

can I get a script somewhere which can automatically defrag my files when needed ??

Please don't say I don't need defragment tools.

If you still choose to do. then please explain me WHY it is not necessary.

Point: What if I want my files to be as less fragmented as possible ??

Someone very smart.. answer me please.
And noooo !!. please don't tell me to search Google or wiki.. I KNOW how fragments work on hard drives.

I just don't get why there is no defrag tool for Linux.. I don't get it.
 
Old 05-29-2008, 04:24 PM   #2
acid_kewpie
Moderator
 
Registered: Jun 2001
Location: UK
Distribution: Gentoo, RHEL, Fedora, Centos
Posts: 43,414

Rep: Reputation: 1967Reputation: 1967Reputation: 1967Reputation: 1967Reputation: 1967Reputation: 1967Reputation: 1967Reputation: 1967Reputation: 1967Reputation: 1967Reputation: 1967
There are defrag tools for ext2 / 3 (see e2defrag) but I guess it's just so tempting to say it doesn't need it. In fact ext4 has a lot of *online* defrag capabilities, so clearly it's missing in ext2 / 3 To my knowledge the point is that data is always written as well as possible to the disk in the first place, compared to fat32 and the likes which find the first hole. so lots of small / medium sized files will never fragment, but moving big files around will probably get written to suboptimal areas at some point, and continuing on can get trickier.

Last edited by acid_kewpie; 05-29-2008 at 04:25 PM.
 
Old 05-29-2008, 04:35 PM   #3
jay73
Guru
 
Registered: Nov 2006
Location: Belgium
Distribution: Ubuntu 11.04, Debian testing
Posts: 5,019

Rep: Reputation: 130Reputation: 130
That is one of the reasons that I use XFS instead of ext2/3: defragmentation.
 
Old 05-29-2008, 04:35 PM   #4
jailbait
Guru
 
Registered: Feb 2003
Location: Blue Ridge Mountain
Distribution: Debian Wheezy, Debian Jessie
Posts: 7,574

Rep: Reputation: 184Reputation: 184
Quote:
Originally Posted by terfy View Post
Hello guys..

I just don't get why there is no defrag tool for Linux.. I don't get it.
The method that ext3 uses to decide where to place a new file in a file system is designed to produce the least amount of fragmentation possible, both for the new file in question and for the file system as a whole over a period of time. Over time an ext3 file system will not fragment significantly as long as it is not so full that the file system software does not have a reasonable amount of free space to work with. Linux users just don't have fragmentation problems.

Because there is no demand for ext3 defragmentation programs the developers either don't produce a defragmention program for their file systems or do so only when they have time on their hands. If you want to defrag a partition then copy all of your files to another partition using the cp command, format the original ext3 partition, and cp the files back to the original partition.

That will reduce fragmentation to the lowest possible level. A defragmentation program which works by copying in place will generally produce a less clean final result than the cp method. The best that a defrag in place can do is the same as double copy. That is true for any file system including Windows file systems.

The method that Windows uses to decide where to place a new file in a file system ignores the problem of fragmentation entirely and produces a high degree of fragmentation as a matter of course. So Windows has a defragmentation in place program which a Windows user must use to keep his file system from becoming so fragmented that the computer becomes unusable. I used Windows for two years and found that the method of copying to another partition and back was faster and gave superior results to Windows' defragmentation in place program.


Quote:
Originally Posted by terfy View Post
Hello guys..

"420 extents found, perfection would be 37 extents"
What program produced that message?

I could see that if you created a 4G file in a partition which already contained numerous files that ext3 would have problems allocating that space in a single block. If it worries you use the copy method to clean up the fragmentation. You might want to make the second copy in two stages, the big file first and then all of the other files in a second copy.

---------------------
Steve Stites

Last edited by jailbait; 05-29-2008 at 04:39 PM.
 
Old 05-29-2008, 04:37 PM   #5
acid_kewpie
Moderator
 
Registered: Jun 2001
Location: UK
Distribution: Gentoo, RHEL, Fedora, Centos
Posts: 43,414

Rep: Reputation: 1967Reputation: 1967Reputation: 1967Reputation: 1967Reputation: 1967Reputation: 1967Reputation: 1967Reputation: 1967Reputation: 1967Reputation: 1967Reputation: 1967
xfs fragments too though. But then at an even slower rate / frequency
 
Old 05-29-2008, 07:34 PM   #6
jay73
Guru
 
Registered: Nov 2006
Location: Belgium
Distribution: Ubuntu 11.04, Debian testing
Posts: 5,019

Rep: Reputation: 130Reputation: 130
Quote:
xfs fragments too though. But then at an even slower rate / frequency
You are right. But the point I was making is that it can be de-fragmented when that happens, unlike ext2/ext3.
 
Old 05-29-2008, 09:21 PM   #7
sundialsvcs
Guru
 
Registered: Feb 2004
Location: SE Tennessee, USA
Distribution: Gentoo, LFS
Posts: 5,423

Rep: Reputation: 1157Reputation: 1157Reputation: 1157Reputation: 1157Reputation: 1157Reputation: 1157Reputation: 1157Reputation: 1157Reputation: 1157
One of the characteristics of "a real problem" is that "the patient is screaming," and "with damn-good reason."

If the patient is not screaming, you do not have a problem.

If the patient, even if screaming, does not have a damn good reason to be screaming, then you also do not have a problem.

Anytime you have 50,000 files on a single hard drive (and have you counted the number of files on your drive lately?) it's axiomatic that some of those files are going to be split into far more pieces than might be considered "perfection."

The computer industry does not award Brownie-points for "perfection."

If among those 50,000 files we find that we have (say) 15,000 files that are sliced-and-diced, and we find that actual system performance is being dumped into the toilet as a result ... then we have a genuine problem.

A modern filesystem is carefully designed to be able to churn away for (say) three or four years with no maintenance whatsoever, and to deliver fairly-acceptable (at least fairly-consistent) performance during that entire time ... no matter what happens. At any point during that time you will be able to "point an accusing finger at" this-file or that-file. But if you do, you're missing the point. There are 50,000 (or many more) files out there, and (quoth Spock...) "the needs of the many outweigh the needs of the few."
 
Old 05-29-2008, 11:42 PM   #8
i92guboj
Gentoo support team
 
Registered: May 2008
Location: Lucena, Córdoba (Spain)
Distribution: Gentoo
Posts: 4,043

Rep: Reputation: 375Reputation: 375Reputation: 375Reputation: 375
Quote:
Originally Posted by terfy View Post
Hello guys..

I'm wondering about this..
Everyone says that it is not necessary to defrag ext3 filesystem.. but look at this.

"420 extents found, perfection would be 37 extents"
its a 4gig file..

why is it not necessary to defrag this file ? or any other files..
It is not necessary because linux is smart enough to avoid the penalty of fragmentation (and also, the fragmentation is usually kept really low unless the drive is completely full).

If you are really interested, google (yes, google) for linux i/o schedulers. The i/o schedulers are pieces of the kernel that re-order all the read/write operations done in physical devices, so they are done in a smart way, and the heads of your hard drive don't have to be bouncing from one place to another all the time. That's why having a defragmented filesystem will give you virtually no extra performance, unless it's on very extreme conditions.

Quote:
can I get a script somewhere which can automatically defrag my files when needed ??
Copy the contents elsewhere, format the drive, copy it back. That's the easiest you can get. Make sure your use cp -a or tar -p to preserve permissions or you'll screw the backup. There are tools around, but, believe me, if after sooooooooo much years no one has cared enough to continue any defragmenter project (and there have been quite a few of them), maybe it's because they realized that it wasn't worth the trouble.

Quote:
Please don't say I don't need defragment tools.
Please, convince me that I need heroin. :P

Last edited by i92guboj; 05-29-2008 at 11:43 PM.
 
Old 05-30-2008, 03:29 AM   #9
terfy
Member
 
Registered: Jan 2005
Location: Denmark
Distribution: openSUSE
Posts: 165

Original Poster
Rep: Reputation: 30
Quote:
Me: Please don't say I don't need defragment tools.
You: Please, convince me that I need heroin. :P
hah. really nice point there. !!

thank you everyone for your useful answers.

I decided to program some kind of a smart-reschedule-fragmented-files-into-defragged scheduler.

Because.: If fragmentation is possible (and ofcause it is), SOME time IT WILL be larger and larger and even become a bigger problem.

I keep you informed about my results. in both performance terms and usability.

terfy
 
Old 05-30-2008, 04:48 AM   #10
acid_kewpie
Moderator
 
Registered: Jun 2001
Location: UK
Distribution: Gentoo, RHEL, Fedora, Centos
Posts: 43,414

Rep: Reputation: 1967Reputation: 1967Reputation: 1967Reputation: 1967Reputation: 1967Reputation: 1967Reputation: 1967Reputation: 1967Reputation: 1967Reputation: 1967Reputation: 1967
When I was in secondary school I was taught loads of stuff in science class, mainly about electrons and atoms and orbits. Then I get to university and am told "yeah, that was kinda nonsense, just a teaching aid and not at all true, but you couldn't handle the truth" by the lecturers there. At the risk of aligning myself to a university lecturer, I'd say this is a similar thing...
 
Old 05-30-2008, 08:04 AM   #11
sundialsvcs
Guru
 
Registered: Feb 2004
Location: SE Tennessee, USA
Distribution: Gentoo, LFS
Posts: 5,423

Rep: Reputation: 1157Reputation: 1157Reputation: 1157Reputation: 1157Reputation: 1157Reputation: 1157Reputation: 1157Reputation: 1157Reputation: 1157
Quote:
Originally Posted by terfy View Post
Because.: If fragmentation is possible (and ofcause it is), SOME time IT WILL be larger and larger and even become a bigger problem.
Enjoy the exercise. But the reality is that fragmentation is always possible and always occurs. Conditions change. Files grow and shrink. Free space comes and goes. If there was a pressing business-need for an ext3 defragmenter, it would already be out there.

Defrag tools became popular for a while because good ol' FAT was built for floppy disks. And then, when expanded to the hard-drive world, for many years those drives were small... hence, "mostly full." The requirement got too big for FAT's britches, and Microsoft rolled-out NTFS to replace it.
 
Old 05-30-2008, 12:01 PM   #12
i92guboj
Gentoo support team
 
Registered: May 2008
Location: Lucena, Córdoba (Spain)
Distribution: Gentoo
Posts: 4,043

Rep: Reputation: 375Reputation: 375Reputation: 375Reputation: 375
You guys are still missing the point.

No one is telling you that the fragmentation doesn't occur (well, some people do, but those would be comparable to those who teach pre-university physics ).

Fragmentation occurs (to a lesser degree than in a FAT filesystem, but it happens). The matter as I said above is not if it happens or not (because, again, yes, it happens). The real matter is if your system really cares about it.

Of course, the filesystem also matters, for example, reiserfs (3.x) is (in)famous for it's well known problems with fragmentation. That's true. Reiserfs is not the softest in cpu usage anyway, even if the fragmentation level is 0% and you are not using tail packing.

But, if you are not a reiserfs fan, read about I/O schedulers and why they make fragmentation pointless unless you have a full filesystem which is very fragmented.

Besides that, if you are really concerned about this kind of things, here you have some reference material:

http://forums.gentoo.org/viewtopic-t...efragment.html
http://e2fsprogs.sourceforge.net/ext2.html

Note that the very nature of the linux fs's make is very hard to operate on them while they are mounted. I mean, to operate at low level you should first make sure that the fs is not mounted because bad things can happen (that's why you never fsck a fs that's mounted with write permissions). So, if you want to operate on live mounted filesystems you need to operate on the filesystem land mainly, and not at lower levels.

If you don't mind to umount the filesystem first, as I already told you, then there's no need to research. Just tar -cfzpf backup.tar.gz your filesystem, mkfs it again, and restore the backup. It's probably the fastest and more efficient way (though you need intermediate storage, which is the downside).
 
Old 06-08-2008, 03:19 PM   #13
bgm_2008
LQ Newbie
 
Registered: Jun 2008
Posts: 1

Rep: Reputation: 0
Quote:
Originally Posted by jailbait View Post
Over time an ext3 file system will not fragment significantly as long as it is not so full that the file system software does not have a reasonable amount of free space to work with.
Which means it will fragment significantly over time, over time partitions usually tends to fill not to empty.

Quote:
Originally Posted by jailbait View Post
Linux users just don't have fragmentation problems.
Because there is no demand for ext3 defragmentation programs *snip*
I am a linux user, I have fragmentation related problems with ext3 and have been demanding for ext defragmentation tool. The only answer I had so far ranged from dumb to useless: ext3 don't fragment, you don't need to defrag ext3, wait for ext4 to be released, and so on.


Quote:
Originally Posted by jailbait View Post
What program produced that message?
Seems to me, it's filefrag (provided by e2fsprogs)

Quote:
Originally Posted by sundialsvcs View Post
If there was a pressing business-need for an ext3 defragmenter, it would already be out there.
This king of logic is flawed, as in "if there was a vital need to fix the climate thing problem and lack of energy resources, it would be fixed by now (since we've been knowing about the problem for decades)".

People who need a defrag tool just don't use ext3, or use old the "move your data elsewhere and back" trick. (which can be faster than defragging under windows).

But in this specific case I can't use this trick, I got a 2 TB ext3 RAID 5 array which is 6 months old, quite full, and already pretty much fragmented. No more room to add a single drive in this file server and anyways there's no way I can get the money to buy the disk space needed to move all the data and back.
 
Old 06-09-2008, 08:04 AM   #14
i92guboj
Gentoo support team
 
Registered: May 2008
Location: Lucena, Córdoba (Spain)
Distribution: Gentoo
Posts: 4,043

Rep: Reputation: 375Reputation: 375Reputation: 375Reputation: 375
Quote:
Originally Posted by bgm_2008 View Post
I am a linux user, I have fragmentation related problems with ext3 and have been demanding for ext defragmentation tool. The only answer I had so far ranged from dumb to useless: ext3 don't fragment, you don't need to defrag ext3, wait for ext4 to be released, and so on.
Yep, those responses are quite useless, that's true.

Quote:

This king of logic is flawed, as in "if there was a vital need to fix the climate thing problem and lack of energy resources, it would be fixed by now (since we've been knowing about the problem for decades)".
But your reasoning there is flawed as well. The two problems are not comparable, and the solution to the ext3 fragmentation one would be trivial, while the other is not as trivial plus big enterprises want money, which affects the climate, but not the file systems fragmentation (in general). To compare both problems is like to compare the grown of pineapples with the factoring of nuke bombs.

Quote:
People who need a defrag tool just don't use ext3, or use old the "move your data elsewhere and back" trick. (which can be faster than defragging under windows).
It's not only faster, but more secure. In term of data integrity, a shorter operation is always better. And an operation that can be done while the system is not operating and concurrently writing into that volume is even better.

You copy the data somewhere, so the original can be mounted read only. Then, after you are sure nothing went wrong you copy it back.

People using ext3 usually care about security, and a live defragmenter usually doesn't match too well with the whole linux/ext3 binomial.

Quote:
But in this specific case I can't use this trick, I got a 2 TB ext3 RAID 5 array which is 6 months old, quite full, and already pretty much fragmented. No more room to add a single drive in this file server and anyways there's no way I can get the money to buy the disk space needed to move all the data and back.
This is a problem. Maybe a tape drive? You will gain a very valuable resource to keep your backups.
 
Old 03-29-2009, 09:51 AM   #15
diverse-one
LQ Newbie
 
Registered: Mar 2009
Posts: 1

Rep: Reputation: 0
Unhappy Defrag for ext3

Why is there no tool for this? I need to defrag several 24 GB vm disks to improve performance on a CentOS 5.2 installation. How do I do that? Any ideas? And, please, do NOT tell me that it is not necessary!!!

Thank you in advance!
 
  


Reply


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off


Similar Threads
Thread Thread Starter Forum Replies Last Post
ext3 super block get modified after mkfs.ext3 amitpardesi Programming 1 04-25-2008 11:42 AM
USB Hardrive with FAT32 and ext3 partitions in need of repair/defrag MiniDev Linux - Hardware 6 02-15-2008 04:05 PM
Defrag (NOT for ext2/ext3, for FAT32 partitions) Red Barchetta Linux - Software 2 03-31-2007 08:01 AM
Ext3 & tune2fs tweaking question for ext3 experts wrc1944 Linux - General 8 12-11-2005 08:45 AM


All times are GMT -5. The time now is 02:10 PM.

Main Menu
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
identi.ca: @linuxquestions
Facebook: linuxquestions Google+: linuxquestions
Open Source Consulting | Domain Registration