A file-system defragmentation tool on Linux

Quakeboy02 · 04-21-2007, 10:38 PM

Quote:

1, My views are based on "Numbers/Charts/Scripts" which anybody could verify/reproduce.

Your numbers are based on voodoo and shaking a stick in a particular fashion. The bottom line is this: if we have to *look* for the problem, or manipulate filesystems in a specific way to *see* a problem, then it Simply Doesn't Exist in the real world. Those motorcycles aren't blowing up from using Quaker State/Valvoline/Havoline instead of Mobil One, and Linux filesystems aren't slowing to a crawl due to fragmentation. To quote a famous commercial "Where's The Beef?".

The quickest way to get my attention would be to cite real problems posted by real admins looking for help with fragmented filesystems.

tmcco · 04-21-2007, 11:18 PM

Quote:

Originally Posted by lazlow

tmcco

Does fragmentation exist in linux files systems? Yes

Is that fragmentation significant enough to be a real problem? Since very few people are having problems in this direction, I would say no.

Would you be more convincing if you would address posts #30 and #42? Yes

Would you be more believable if you did not self reference ( a very big no-no in the scientific world)? Yes

Keep trying.

Lazlow

1, I've answered #30 and #42 in #44, you missed it.

2, All the testing scripts and samples are opened at: http://defragfs.sourceforge.net/theory.html , anybody could verify it.

3, There are many papers talking about "disk-layout / performance", please search in "google scholar".

lazlow · 04-21-2007, 11:57 PM

tmcc0

As to #1: I was typing while you posted(note time stamp). I think you mostly danced around the issues.

#2. Then why hasn't anybody verified it?

Again, self referencing is BAD.

You certainly dance well.

tmcco · 04-22-2007, 12:26 AM

Quote:

Originally Posted by lazlow

tmcc0

As to #1: I was typing while you posted(note time stamp). I think you mostly danced around the issues.

#2. Then why hasn't anybody verified it?

Again, self referencing is BAD.

You certainly dance well.

1, Pardon? Danced around what issues?

2, Well, after carefully experiment, I believe "Linux file-system does fragment and it degrades performance greatly", so I published my views/numbers/charts/scripts/samples. While many other people(maybe including you, and I used to) believe "Linux file-system does not fragment and the performance degrade could be ignored", they just ignore any disadvantages, I hope you'd not.

3, Links to some paper is added to http://defragfs.sourceforge.net/theory.html , although Unix/BSD/FFS seems something outdated, my pages and some of my posts on this thread pointed to Linux file-systems.

rkelsen · 04-22-2007, 04:34 AM

Quote:

Originally Posted by tmcco

Pardon? Danced around what issues?

This one for starters:

Quote:

Originally Posted by studioj

real world disk access is inherently fragmented and not sequential so file fragmentation is a non issue.

And you still haven't stopped self-referencing. Provide links to some real world problems caused by file fragmentation under Linux and you might start persuading people. You're not doing too well at the moment.

Here is a link to an excellent post which you should read: Linux file system defrag.

It explains some of the reasons why file fragmentation under Linux doesn't matter. This part is particularly relevant:

"Now, this is not to say that 'file fragmentation' is a good thing. It's just that 'file fragmentation' doesn't have the *impact* here that it would have in MSDOS-based systems. The performance difference between a 'file fragmented' Linux file system and a 'file unfragmented' Linux file system is minimal to none, where the same performance difference under MSDOS would be huge.

Under the right circumstances, fragmentation is a neutral thing, neither bad nor good. As to defraging a Linux filesystem (ext2fs), there are tools available, but (because of the design of the system) these tools are rarely (if ever) needed or used. That's the impact of designing up front the multi-processing/multi-tasking multi-user capacity of the OS into it's facilities, rather than tacking multi-processing/multi-tasking multi-user support on to an inherently single-processing/single-tasking single-user system."

Emphasis added.

Can you rebut this argument with solid proof and not just another link to your own website?

tmcco · 04-22-2007, 07:14 AM

Quote:

Originally Posted by rkelsen

This one for starters:

And you still haven't stopped self-referencing. Provide links to some real world problems caused by file fragmentation under Linux and you might start persuading people. You're not doing too well at the moment.

Here is a link to an excellent post which you should read: Linux file system defrag.

It explains some of the reasons why file fragmentation under Linux doesn't matter. This part is particularly relevant:

"Now, this is not to say that 'file fragmentation' is a good thing. It's just that 'file fragmentation' doesn't have the *impact* here that it would have in MSDOS-based systems. The performance difference between a 'file fragmented' Linux file system and a 'file unfragmented' Linux file system is minimal to none, where the same performance difference under MSDOS would be huge.

Under the right circumstances, fragmentation is a neutral thing, neither bad nor good. As to defraging a Linux filesystem (ext2fs), there are tools available, but (because of the design of the system) these tools are rarely (if ever) needed or used. That's the impact of designing up front the multi-processing/multi-tasking multi-user capacity of the OS into it's facilities, rather than tacking multi-processing/multi-tasking multi-user support on to an inherently single-processing/single-tasking single-user system."

Emphasis added.

Can you rebut this argument with solid proof and not just another link to your own website?

1, You missed the post which I said those references are on: http://defragfs.sourceforge.net/theory.html now

2, You missed that I post "research_reports/numbers/charts/scripts", most of you just post "I_think/I_believe/In_my_experience/I_feel/I_heard/".

3, You missed my earlier posts: I do agree most disk access patterns are random-read, while which is also heavily affected by fragmentation: in my tests, the partition is only 40%-50% full, after 20 times of aging, the random-read performance degraded to:
ext2/3: 60%
JFS: 40%
XFS: 40%
Reiser3: 50%
Reiser4: 70%
http://defragfs.sourceforge.net/theory2.html
???WHY???
You could not understand it because you missed my post #43, but here is some "hint" from post #43:

Original: 1112233334566..........
After: 22...111..4...66...5...3333

Please think hard.

4, PLEASE! I found many of you just asking the same question I've answered before, so I hope everyone read my original posts carefully before making any subjective judgment.

tmcco · 04-22-2007, 07:46 AM

Quote:

Originally Posted by studioj

i resisted this thread at first because it felt like a troll but since it's still alive here i go.

the problem with your tests is as i see it

OK yea sequential reads get slower DUHH ! When you artificially cause fragmentation on a nearly full disk partition like your test does. but why do we care ? (its an atrifical test designed do create an artificial effect im afraid to the end of promoting the defrag program. when in real world do we do sequential reads like that ?(never). its a totally false benchmark. The giveaway is the way the page theory.html starts out talking about "lies". trying to appeal to emotional responses like advertising does. actual theory would never begin by tanking about "lies, misunderstandings around the world" thats not a theory thats an add slogan and the two things are as far apart as any two thing can be.

real world disk access is inherently fragmented and not sequential so file fragmentation is a non issue.

This is a perfect example for the person who missed everything on the page: http://defragfs.sourceforge.net/theory.html

1, The test measures random-read(rread), remove, write, sequential-read(oread) performance not "only sequential".

2, The test measures performance at three different disk usage: 90%/70%/50%, not "nearly-full".

3, The test scripts/sample/results are all on the page, fully opened for anybody. I'm glad to hear if you got quite different numbers.

rocket357 · 04-22-2007, 08:58 AM

tmcco:

You've missed one important facet of this arguement...NO ONE has noted significant performance loss on their Linux systems after 6 months or more...

Now then, I'm sure if everyone ran your defrag script on their machines, a performance hit would be seen in numbers, but until that performance hit starts interfering with day-to-day usage, I doubt anyone will take this seriously.

Sounds to me like you figured out a way to artificially fragment a system and now you want us all to use your script to fix it? How about this: run the same test on a system that's aged through actual use...you up for that?

That's the kind of "proof" I think people here would be more "open minded" to.

studioj · 04-22-2007, 12:33 PM

Quote:

Originally Posted by rocket357

tmcco:

run the same test on a system that's aged through actual use...you up for that?

That's the kind of "proof" I think people here would be more "open minded" to.

YES i think alot of what you are showing us is cached vs un-cached memory.
so no artificially manipulating and swapping the files so disk cache needs to be repalced and then
show us some performance problems.

and yes i read the strange add page as best i could but i frankly can't understand it because it seems to look like nonsense. looks like you are saying all read and writes randome and
sequential for all filesystem start out 100% equal at what you lable 100%. were these newly formatted disks ? and i just don't believe throughput was equal for both reads and writes and then sudenly began to change.
we need to see raw benchmarking data not charts.

rkelsen · 04-22-2007, 06:20 PM

Quote:

Originally Posted by tmcco

PLEASE! I found many of you just asking the same question I've nswered before

You aren't answering them. You're saying, "these were MY findings"... If you can provide ANY evidence from ANY SOURCE OUTSIDE your own website which shows findings even VAGUELY SIMILAR to yours, your argument will be more credible.

Quote:

Originally Posted by tmcco

after 20 times of aging, the random-read performance degraded to:
...
XFS: 40%
Reiser3: 50%

You make these ridiculous claims, without considering the fact that even a novice computer user would notice a 60% performance hit.

Going by your numbers, my computer should have slowed to a crawl by now. It hasn't. Explain that.

Quote:

Originally Posted by rocket357

How about this: run the same test on a system that's aged through actual use...

I agree. Make it a file server which has lots of throughput.

tmcco · 04-23-2007, 12:13 AM

Quote:

Originally Posted by rkelsen

You aren't answering them. You're saying, "these were MY findings"... If you can provide ANY evidence from ANY SOURCE OUTSIDE your own website which shows findings even VAGUELY SIMILAR to yours, your argument will be more credible.

You make these ridiculous claims, without considering the fact that even a novice computer user would notice a 60% performance hit.

Going by your numbers, my computer should have slowed to a crawl by now. It hasn't. Explain that.

I agree. Make it a file server which has lots of throughput.

Yes, this is another perfect example who missed everything of my posts.

1, You missed post #51 which I said those references by other people.

2, You missed that I mentioned the performance degraded to xx% "after 20 times of aging", ridiculous you were.

lazlow · 04-23-2007, 12:34 AM

Guys

Lets assume that it is tmcco speaking english as a second language and just let it go. He apparently cannot comprehend what we have said. IF he is actually saying something that we have missed (I do not think this is the case) then he is unable to express himself in a manner that we can comprehend.

Lazlow

Quakeboy02 · 04-23-2007, 12:54 AM

Quote:

tmcco bespake: 1, You missed the post which I said those references are on: http://defragfs.sourceforge.net/theory.html now

I looked. There are no references there. Just YOUR words and YOUR charts. We have told you time and time again that you cannot be your own authority. This is a basic principle of philosophy. You must give us some credible examples of well-known admins who are complaining about filesystem fragmentation if you expect to have any hope of being taken seriously. At this point, you aren't even bothering to make an effort. I think you should google "internet crackpot".

nirmaltom · 04-23-2007, 01:00 AM

hi,
as i and many guys said earlier test suite is the first problem.bcaz when u want people to accept ur concept ,first in test suite must be capable to find how much fragmentation is there in our aged file system.
bcaz, we need to know our current status first.i think, u will accept.
Then the test suite must show the advantage of using the program.

Thats not a perfect test suite bcaz it is not ready accept our test cases.

Regarding benchmarks,

Quote:

Originally Posted by studioj
we need to see raw benchmarking data not charts.

regards,
Nirmal Tom.

Note:Arguments leave to better product.

tmcco · 04-23-2007, 02:03 AM

Quote:

Originally Posted by Quakeboy02

I looked. There are no references there. Just YOUR words and YOUR charts. We have told you time and time again that you cannot be your own authority. This is a basic principle of philosophy. You must give us some credible examples of well-known admins who are complaining about filesystem fragmentation if you expect to have any hope of being taken seriously. At this point, you aren't even bothering to make an effort. I think you should google "internet crackpot".

Try to refresh your browser, the "Partial references" is on the top of: http://defragfs.sourceforge.net/theory.html