Hi guys,

I've got a school assignment where I've got to create a defragmentation tool on a Linux system.
There are a few things required :
-The defragmentation program is able to detect whether files are fragmented
-The defragmentation program is able to gather file parts together
-After completion, the defragmentation tool leaves the filesystem nearly 0% fragmented
-The defragmentation is able to work online too (mounted filesystem)

The documentation all around the web about that kind of thing is near zero...

Do you guys know anything about it ?

I really appreciate any help you can provide.

Tibo

Linux native filesystems don't suffer defragmentation problems. The usual level for 500GB disk/partition is 3%. And for online defragging, you won't get it any lower as you have to avoid relocating active files.

btrfs has its own internal defrag.

The only filesystems that NEED defragging are from Microsoft - exFAT/VFAT/NTFS. Good luck with those.

Thank you for your answer !

Yes, I've also read that it doesn't need to be defragmented.
That's why there is nothing anywhere on the web and that's why I came here to help because I have really no idea how to do that.
Unfortunately, we have to do this tool....

Thanks anyway !

Actually, there is a defragmentation tool for XFS, it works on mounted filesystems and is open source ...

NTFS/vfat/fat all exist on Linux.

Your requirement was "defragmentation tool on a Linux system"... but it did NOT specify which filesystem such a defrag tool was to support.

Depending on the level of difficulty, any of the three would serve as the target of such a defrag tool.

All right, I see what you mean jpollard.

First, we got to create a fuse filesystem and then, we got to create this tool.
But, like you said, it didn't specify any specific filesystem.

It just says that the tool has to work on "an offline (unmount) filesystem to gather files together".

So, I'll have to try to do this tool on a Microsoft filesystem I guess.

To Emerson, you're talking about xfs_fsr, right ?

if you REALLY MUST

then tar up the files
delete the originals
then untar the files

that will remove the fragments that MIGHT be on a ext4 or lvm partition
but i have never seen more than about 1% on my drives and i work with a TON of image data

Well, this was about assignment to create such a tool. That's why I recommended looking at xfs_fsr code, to get ideas how it is done.

If you're trying to be filesystem agnostic, take a look at the manpage for hdparm and scroll down to the "--fibmap" option. That option should give you the information you need to determine whether a file is fragmented.

Without getting very filesystem specific and getting down-and-dirty with the internal structures of the unmounted filesystem, about all you can do to defragment a file is to copy it somewhere else and hope that the copy is less fragmented than the original. Linux doesn't provide any means to control where a file's blocks are allocated on the disk.

I'd tar the files and test each one then back.

Opps been suggested

If NTFS is preferred, then the fuse filesystem is already done.
For the FAT/vFAT, you might not need one. In the unmounted case - it is easy, you directly access the blocks. Examples for this are also easily available (look for the mtools list for manipulating the fat/vfat disk/partition/file). For the mounted case it gets tricky because you have to stop the filesystem from functioning - then move the files that are NOT in use (if it is possible). The problem in the online case is that you WILL have to modify the kernel filesystem code so that the mounted (and in use) buffers DON'T get moved around. As I recall, the FAT/VFAT mounted filesystems don't keep buffers in memory except for the open files. But you would have to check that.

Defragmentation on UNIX and UNIX-like systems is not recommended. Because many modern systems use B+ or B- trees when building a file system they create an index or where every part of a file is and loads it accordingly. In the past, ReiserFS aka Reiser3 offered defragmentation of an offline drive, but it proved to be dangerous as it could corrupt the data tree.

Even modern file systems like JFS, Ext4, ZFS, HAMMER, UFS2, BtrFS, and others all use these tree style file systems with lots of indexing and things like copy-on-write abilities to avoid the need of defragmentation using these indexes.

This is why so little information exists. There are some volume managers like BtrFS and ZFS that offer integrity checks, but those are to reduce data corruption only.

Ntfs-3g-progs to assist NTFS on UNIX-like systems, as well as tools to assist vfat/fat12/fat16/exfat/fat32 offers some defragmentation abilities, but these are rudimentary and still, are not recommended.

Didn't know that about NTFS... though I did know it was tricky.

That pretty much leaves only FAT/vFat filesystems that could be used (they are a MUCH simpler filesystem).

Unless, of course, you wanted to write your own filesystem (which is what fuse is for).

Thank you all for your answers !
I'm going to try and get on with it.

Just to add some info into the mix, ext4, the most commonly used filesystem on Linux, *does* suffer from fragmentation, contrary to some comments above. However it is very limited due to the way the kernel filesystem driver runs things and thus is not normally problematic.

Multiple extents for the same inode, for example, are not necessarily contiguous and these can be defragmented. What I would recommend is that you download the e2fsprogs source code and have a look at the code for the following programs:

e4defrag - reduces fragmentation of an extent-based file, giving it more contiguous blocks and improving the file access speed.

filefrag - reports on how badly fragmented a file might be.

e2freefrag - scans the block bitmap to check how many free blocks are present as contiguous and aligned free space.

That should give you a start...

1 members found this post helpful.