Hello guys..

I'm wondering about this..
Everyone says that it is not necessary to defrag ext3 filesystem.. but look at this.

"420 extents found, perfection would be 37 extents"
its a 4gig file..

why is it not necessary to defrag this file ? or any other files..

can I get a script somewhere which can automatically defrag my files when needed ??

Please don't say I don't need defragment tools.

If you still choose to do. then please explain me WHY it is not necessary.

Point: What if I want my files to be as less fragmented as possible ??

Someone very smart.. answer me please.
And noooo !!. please don't tell me to search Google or wiki.. I KNOW how fragments work on hard drives.

I just don't get why there is no defrag tool for Linux.. I don't get it.

There are defrag tools for ext2 / 3 (see e2defrag) but I guess it's just so tempting to say it doesn't need it. In fact ext4 has a lot of *online* defrag capabilities, so clearly it's missing in ext2 / 3 To my knowledge the point is that data is always written as well as possible to the disk in the first place, compared to fat32 and the likes which find the first hole. so lots of small / medium sized files will never fragment, but moving big files around will probably get written to suboptimal areas at some point, and continuing on can get trickier.

That is one of the reasons that I use XFS instead of ext2/3: defragmentation.

The method that ext3 uses to decide where to place a new file in a file system is designed to produce the least amount of fragmentation possible, both for the new file in question and for the file system as a whole over a period of time. Over time an ext3 file system will not fragment significantly as long as it is not so full that the file system software does not have a reasonable amount of free space to work with. Linux users just don't have fragmentation problems.

Because there is no demand for ext3 defragmentation programs the developers either don't produce a defragmention program for their file systems or do so only when they have time on their hands. If you want to defrag a partition then copy all of your files to another partition using the cp command, format the original ext3 partition, and cp the files back to the original partition.

That will reduce fragmentation to the lowest possible level. A defragmentation program which works by copying in place will generally produce a less clean final result than the cp method. The best that a defrag in place can do is the same as double copy. That is true for any file system including Windows file systems.

The method that Windows uses to decide where to place a new file in a file system ignores the problem of fragmentation entirely and produces a high degree of fragmentation as a matter of course. So Windows has a defragmentation in place program which a Windows user must use to keep his file system from becoming so fragmented that the computer becomes unusable. I used Windows for two years and found that the method of copying to another partition and back was faster and gave superior results to Windows' defragmentation in place program.


What program produced that message?

I could see that if you created a 4G file in a partition which already contained numerous files that ext3 would have problems allocating that space in a single block. If it worries you use the copy method to clean up the fragmentation. You might want to make the second copy in two stages, the big file first and then all of the other files in a second copy.

---------------------
Steve Stites

xfs fragments too though. But then at an even slower rate / frequency

You are right. But the point I was making is that it can be de-fragmented when that happens, unlike ext2/ext3.

One of the characteristics of "a real problem" is that "the patient is screaming," and "with damn-good reason."

If the patient is not screaming, you do not have a problem.

If the patient, even if screaming, does not have a damn good reason to be screaming, then you also do not have a problem.

Anytime you have 50,000 files on a single hard drive (and have you counted the number of files on your drive lately?) it's axiomatic that some of those files are going to be split into far more pieces than might be considered "perfection."

The computer industry does not award Brownie-points for "perfection."

If among those 50,000 files we find that we have (say) 15,000 files that are sliced-and-diced, and we find that actual system performance is being dumped into the toilet as a result ... then we have a genuine problem.

A modern filesystem is carefully designed to be able to churn away for (say) three or four years with no maintenance whatsoever, and to deliver fairly-acceptable (at least fairly-consistent) performance during that entire time ... no matter what happens. At any point during that time you will be able to "point an accusing finger at" this-file or that-file. But if you do, you're missing the point. There are 50,000 (or many more) files out there, and (quoth Spock...) "the needs of the many outweigh the needs of the few."

It is not necessary because linux is smart enough to avoid the penalty of fragmentation (and also, the fragmentation is usually kept really low unless the drive is completely full).

If you are really interested, google (yes, google) for linux i/o schedulers. The i/o schedulers are pieces of the kernel that re-order all the read/write operations done in physical devices, so they are done in a smart way, and the heads of your hard drive don't have to be bouncing from one place to another all the time. That's why having a defragmented filesystem will give you virtually no extra performance, unless it's on very extreme conditions.

Copy the contents elsewhere, format the drive, copy it back. That's the easiest you can get. Make sure your use cp -a or tar -p to preserve permissions or you'll screw the backup. There are tools around, but, believe me, if after sooooooooo much years no one has cared enough to continue any defragmenter project (and there have been quite a few of them), maybe it's because they realized that it wasn't worth the trouble.

Please, convince me that I need heroin. :P

hah. really nice point there. !!

thank you everyone for your useful answers.

I decided to program some kind of a smart-reschedule-fragmented-files-into-defragged scheduler.

Because.: If fragmentation is possible (and ofcause it is), SOME time IT WILL be larger and larger and even become a bigger problem.

I keep you informed about my results. in both performance terms and usability.

terfy

When I was in secondary school I was taught loads of stuff in science class, mainly about electrons and atoms and orbits. Then I get to university and am told "yeah, that was kinda nonsense, just a teaching aid and not at all true, but you couldn't handle the truth" by the lecturers there. At the risk of aligning myself to a university lecturer, I'd say this is a similar thing...

Enjoy the exercise. But the reality is that fragmentation is always possible and always occurs. Conditions change. Files grow and shrink. Free space comes and goes. If there was a pressing business-need for an ext3 defragmenter, it would already be out there.

Defrag tools became popular for a while because good ol' FAT was built for floppy disks. And then, when expanded to the hard-drive world, for many years those drives were small... hence, "mostly full." The requirement got too big for FAT's britches, and Microsoft rolled-out NTFS to replace it.

You guys are still missing the point.

No one is telling you that the fragmentation doesn't occur (well, some people do, but those would be comparable to those who teach pre-university physics ).

Fragmentation occurs (to a lesser degree than in a FAT filesystem, but it happens). The matter as I said above is not if it happens or not (because, again, yes, it happens). The real matter is if your system really cares about it.

Of course, the filesystem also matters, for example, reiserfs (3.x) is (in)famous for it's well known problems with fragmentation. That's true. Reiserfs is not the softest in cpu usage anyway, even if the fragmentation level is 0% and you are not using tail packing.

But, if you are not a reiserfs fan, read about I/O schedulers and why they make fragmentation pointless unless you have a full filesystem which is very fragmented.

Besides that, if you are really concerned about this kind of things, here you have some reference material:

http://forums.gentoo.org/viewtopic-t...efragment.html
http://e2fsprogs.sourceforge.net/ext2.html

Note that the very nature of the linux fs's make is very hard to operate on them while they are mounted. I mean, to operate at low level you should first make sure that the fs is not mounted because bad things can happen (that's why you never fsck a fs that's mounted with write permissions). So, if you want to operate on live mounted filesystems you need to operate on the filesystem land mainly, and not at lower levels.

If you don't mind to umount the filesystem first, as I already told you, then there's no need to research. Just tar -cfzpf backup.tar.gz your filesystem, mkfs it again, and restore the backup. It's probably the fastest and more efficient way (though you need intermediate storage, which is the downside).

Which means it will fragment significantly over time, over time partitions usually tends to fill not to empty.

I am a linux user, I have fragmentation related problems with ext3 and have been demanding for ext defragmentation tool. The only answer I had so far ranged from dumb to useless: ext3 don't fragment, you don't need to defrag ext3, wait for ext4 to be released, and so on.


Seems to me, it's filefrag (provided by e2fsprogs)

This king of logic is flawed, as in "if there was a vital need to fix the climate thing problem and lack of energy resources, it would be fixed by now (since we've been knowing about the problem for decades)".

People who need a defrag tool just don't use ext3, or use old the "move your data elsewhere and back" trick. (which can be faster than defragging under windows).

But in this specific case I can't use this trick, I got a 2 TB ext3 RAID 5 array which is 6 months old, quite full, and already pretty much fragmented. No more room to add a single drive in this file server and anyways there's no way I can get the money to buy the disk space needed to move all the data and back.

Yep, those responses are quite useless, that's true.

But your reasoning there is flawed as well. The two problems are not comparable, and the solution to the ext3 fragmentation one would be trivial, while the other is not as trivial plus big enterprises want money, which affects the climate, but not the file systems fragmentation (in general). To compare both problems is like to compare the grown of pineapples with the factoring of nuke bombs.

It's not only faster, but more secure. In term of data integrity, a shorter operation is always better. And an operation that can be done while the system is not operating and concurrently writing into that volume is even better.

You copy the data somewhere, so the original can be mounted read only. Then, after you are sure nothing went wrong you copy it back.

People using ext3 usually care about security, and a live defragmenter usually doesn't match too well with the whole linux/ext3 binomial.

This is a problem. Maybe a tape drive? You will gain a very valuable resource to keep your backups.

Why is there no tool for this? I need to defrag several 24 GB vm disks to improve performance on a CentOS 5.2 installation. How do I do that? Any ideas? And, please, do NOT tell me that it is not necessary!!!

Thank you in advance!