Originally Posted by tytso

Hi, it's been pointed out to me that there's been this thread about fragmentation, and it's apparently degenerated a fair amount, and furthermore that my name has been dragged into this. So let me try to clear up some points.

First of all, if someone wants to pull out Dean Anderson's opinion of me, some explanation of the background is in order. Since I have previously served as chair of the IP Security Working Group and as a member of the security area directorate of the Internet Engineering Task Force (IETF, the standards body of the Internet), I was asked to serve as one of the two IETF mailing list "Sergeant at Arms", which is responsible for enforcing the mailing list code of conduct. In that role, I asked Mr. Dean Anderson to refrain from violating those guidelines, and he created said web page as a result. Some time later, as a result of his continued violation of these guidelines, Mr. Anderson was subsequently banned from all IETF mailing lists, not by my decision (I, in my role an IETF Sergeant at Arms, report to the IETF chair, and can be overruled by him, although to date this has never happened), but by the Internet Engineering Steering Group. Their decision to ban Mr. Anderson was taken after a request from the IETF community, following a process defined by RFC 3683. The reasons for this ban can be found here and here . Mr Dean Anderson subsequently appealed this decision to the Internet Architecture Board, who upheld the decision by the IESG. Folks should feel free to look at the decisions reached by the leadership of the IETF, and decide whether they come down on the side of them (and me), or that of Mr. Dean Anderson.

Secondly, the note referenced by tmcco was mine. However, it should be noted that it was in reply to his benchmarks about fragmentation. It is certainly true that Linux filesystems can suffer from fragmentation, and his benchmark fairly compared multiple filesystems given a particular workload --- which happened to be to involve randomly deleting some number of files, and writing some other files of different random lengths, with all files in the same directory. Whether or not the workload as measured by his benchmark is a fair one is a different story. Indeed, it's pretty obvious that it doesn't represent an accurate model of real life usage for most users. My note was trying to encourage him to develop a better set of benchmarks.

As far as his specific program is concerned, I'm not particularly enthusiastic about pure userspace progams that attempt to defragment a filesystem. There are significant locking problems that you have to worry about --- what if the file is being accessed and modified while a userspace defragger is copying and rewriting the file? Secondly, these programs don't completely eliminate fragmentation, and indeed, to the extent they work, it is because Linux (and most modern Unix filesystems) are fragmentation resistant. (Note that I didn't say fragmentation-proof!)

So my preference is to improve existing filesystems to make them more fragmentation resistant at least for common, real-life workloads, instead of writing fragmentation programs. We should be able to do better than Microsoft Windows, and not require people run slow defragmentation programs. Of course, the ethos of open source software is that people can work on whatever they want, and if want to work on defragmentation programs, they are perfectly free to --- but it's simply not something I'm interested in myself. However, for people who do want to work on defragmenters, my recommendation is to either create off-line defraggers (such as fixing and updating e2defrag, which is an ext2-specific defragger that only works on 1k blocksizes, doesn't support some of the newer ext3/4 filesystems, and has the sad problem that if it gets interrupted mid-defrag, your filesystem is completely toasted), or to create on-line defraggers that have kernel support (such as the ext4 defragger being worked on by Takashi Sato-san from NEC). My personal interest in the kernel support for on-line defragging is to be able to do on-line shrinking of filesystems, which means that the kernel support for on-line defrag can support multiple problems, which is always a sign of a good design.

So the net of this is that my comments were not meant as an unlimited endorsement of tmcco's ideas, nor (in particular) of his selected defragmentation approach. Do I hope that he might do some more work in this area, however? Of course! In order to do that, though, it means we need to encourage each other in areas where we are on the right track, and gently correct people when they are wrong. It is for that reason that I tried to get him to focus on better benchmarks, and in fact I spent no time criticizing his defragmentation program. This is first of all because he didn't ask me to comment on it, and secondly, because it's better to encourage people in areas where you feel they might be able to contribute useful work, as opposed to tearing them down in other areas.

So let me end this by quoting from Rodney King, "Can't we all just get along?"

Regards,

-- Ted

Originally Posted by tytso

For the record, no, tmcco did not give me a link to this thread, and I never sent him any statements or comments about this thread and the comments made by him or others on this discussion for the board for the simple reason that I didn't know about it. I did not become aware of this discussion until lazlow sent me e-mail about it on Friday morning. My apologies for not replying until now; I'm pretty busy these days.

Regards,

-- Ted

Originally Posted by lazlow

If an author intentionally tries to mislead us about facts in one venue (say a forum), how can we trust the software or the data that was produced by that author?

Originally Posted by tytso

It is certainly true that Linux filesystems can suffer from fragmentation, and his benchmark fairly compared multiple filesystems given a particular workload ... Whether or not the workload as measured by his benchmark is a fair one is a different story. Indeed, it's pretty obvious that it doesn't represent an accurate model of real life usage for most users.

Originally Posted by tytso

As far as his specific program is concerned, I'm not particularly enthusiastic about pure userspace progams that attempt to defragment a filesystem.

Originally Posted by tytso

these programs don't completely eliminate fragmentation, and indeed, to the extent they work, it is because Linux (and most modern Unix filesystems) are fragmentation resistant.

Originally Posted by lazlow

In post #71 point #7. it is stated the Mr. Ts'o does not care for some of our posts, yet in post #89 Mr. Ts'o states that at that time(post #71) he was not even aware of this thread.

Originally Posted by lazlow

In Tmcco's profile, which was originated in May 2003. it states a home page of www.tmcco.com. If one looks at http://web.archive.org/web/*/http://tmcco.com (a site that archive old web pages) you can clearly see that the site was Mr. Young's (the current owner of the site) at least by April 2001. Further, in a phone conversation (email confirmation pending) Mr. Young indicates that he has owned the domain since 1997 and has no affiliation with Xu CanHao. This begs the question: why would someone claim a home page that was clearly not theirs for at least two years prior(per archive.org) and probably not since 1997 (per Mr. Young)? If it had merely been a typo, most of us would have corrected that when it was pointed out earlier in this thread(post #85). I am not saying we do not make mistakes. Clearly I made one with my post #76. When my mistake was pointed out to me, I did not delete the post (attempt to hide it). I posted an addition to the post to point to a post that gave more information and apologized to Mr. Ts'o.

Originally Posted by lazlow

When I install a piece of software on my computer, I rely on the credibility of the author to insure that the code does what he says it does. With the size and complexity of todays software I would venture a guess that 99% of us do. If an author intentionally tries to mislead us about facts in one venue (say a forum), how can we trust the software or the data that was produced by that author?

Originally Posted by rkelsen

What amuses me Lazlow is that Mr. Ts'o's post in this thread re-iterates what many people here have said, for example:

There may be some flaming and personal attacks in this thread, but if one was to read through all of that, one would see that all of the above points made by Mr. Ts'o had already been made. It certainly appears that Mr. CanHao has chosen to ignore all of the advice given, and has failed to properly understand the position of Mr. Ts'o on this issue.

Originally Posted by tmcco

Mr. Ts'o: Linux filesystems can suffer from fragmentation, but depends.

Originally Posted by tmcco

Mr. Ts'o: I'm not particularly enthusiastic about pure userspace progams that attempt to defragment a filesystem.

Originally Posted by lazlow

How does:



Come anywhere close to the meaning of:




"Teds" quote (above) is quite clear. The use of the word posting is certainly meant to indicate a forum of some sort. It is also quite different from what you are now attempting to say it meant.




This is at least very clear. You admit intentionally and knowingly presenting FALSE information. The relivance of this goes to your credibility.



What does GPLing software have to do with credibility? One could GPL a virus if one wished. If you were referring to the warranty section, that is there to protect people of good intentions from being sued for an honest mistake. As I have said before, I have no problem with people making an HONEST mistake.



Who is this "we'd"? You seem to be implying that it is you and Mr. Ts'o. If this is what you meant, I think that Mr. Ts'o knowledge of this thread would allow him to speak for himself.


One good thing is that you now seem to understand that calling a cited reference by his/her first name is bad.


Edit: This was being typed as Mr. Ts'o post (above) was being posted.

Originally Posted by tytso

Actually, I'd say that Linux filesystems can suffer from fragmentation, but for many (most?) user workloads the fragmentation problem is not a severe performance issue.

Originally Posted by tytso

OK, maybe I was being too polite. Let me be more explicit. I wouldn't use a userspace defragmentation program on my own systems, and wouldn't recommend that most users waste a lot of time on this issue.

Keep in mind, a defragmentation program takes a lot of time to run, and compared to the amount of time that you would actually save, it's not clear it really is a win in the long run. Even on Windows, where fragmentation is a larger issue, I wonder how many Windows users have wasted vast amounts of time staring at Norton Disk Optimizer while it defrags their disk, only to have their Excel spreadsheet load 2 seconds faster. Which is great, except that they spent 30-60 minutes waiting for their disk to defragment. So was it really worth it in the end? Not to mention how many hours people have wasted debating the merits of fragmentation on this and other forum discussion? ;-)

Also keep in mind that Linux is much more aggressive about caching than Windows, so for common files, once they are in your page cache, the on-disk fragmentation doesn't matter nearly as much. So not surprisingly, the best way to improve performance in the vast majority of cases is to spend a small amount of money and increase the amount of memory in the system in question.

As far as a userspace defragmentation problem, as I've said already, (1) they suffer from locking problems, which are fundamentally insoluble, since there's no way to tell if another program us using the file, possibly in read/write mode, while you are trying to defrag it, (2) they don't completely remove the fragmentation, and (3) you can replicate what they do yourself by using a simple shell command: "tar cfj /var/tmp/defrag-save.tar.bz dir ; rm -rf dir ; tar xfj /var/tmp/defrag-save.tar.bz". And again, to the extent that this works, it's because Unix filesystems are naturaly fragmentation resistant, and you are simply forcing the system to reallocate the blocks for the files in question. Is that a "fake" program? It depends on your definition of pejoratives, I suppose. It certainly isn't anything complicated, and it doesn't always work, and if there are any other programs accessing the directory when you run the procedure, you will probably cause those programs to malfunction. Understanding these issues is probably far more important than settling on whether or not a userspace defrag program is "fake" or "useless" or "thick-headed", at least in my humble opinion.

Originally Posted by lazlow

I really have no idea what you mean by:




I do not expect you to use your real name here, I doubt very many do (I do not). I have not asked you to post your private data here. Posting false data is very different from not posting your private data(I do not post a home page here.).

Most of the rules of the real world apply to the internet as well. If one lies and is caught one will be exposed. If one claims something is his and it is not, he will be exposed. Twisting someone else's opinions/statements to one's own agenda is generally not tolerated. When people try to "edit history" by repeatedly changing or changing the mean of what was said, one will be exposed. In short, just like in the real world, one will be held accountable for ones actions.