LinuxQuestions.org
Latest LQ Deal: Latest LQ Deals
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - Software
User Name
Password
Linux - Software This forum is for Software issues.
Having a problem installing a new program? Want to know which application is best for the job? Post your question in this forum.

Notices


Reply
  Search this Thread
Old 06-01-2016, 02:31 PM   #1
TIBO
LQ Newbie
 
Registered: May 2015
Posts: 12

Rep: Reputation: Disabled
Defragmentation tool


Hi guys,

I've got a school assignment where I've got to create a defragmentation tool on a Linux system.
There are a few things required :
-The defragmentation program is able to detect whether files are fragmented
-The defragmentation program is able to gather file parts together
-After completion, the defragmentation tool leaves the filesystem nearly 0% fragmented
-The defragmentation is able to work online too (mounted filesystem)

The documentation all around the web about that kind of thing is near zero...

Do you guys know anything about it ?

I really appreciate any help you can provide.

Tibo
 
Old 06-01-2016, 02:42 PM   #2
jpollard
Senior Member
 
Registered: Dec 2012
Location: Washington DC area
Distribution: Fedora, CentOS, Slackware
Posts: 4,912

Rep: Reputation: 1513Reputation: 1513Reputation: 1513Reputation: 1513Reputation: 1513Reputation: 1513Reputation: 1513Reputation: 1513Reputation: 1513Reputation: 1513Reputation: 1513
Linux native filesystems don't suffer defragmentation problems. The usual level for 500GB disk/partition is 3%. And for online defragging, you won't get it any lower as you have to avoid relocating active files.

btrfs has its own internal defrag.

The only filesystems that NEED defragging are from Microsoft - exFAT/VFAT/NTFS. Good luck with those.
 
Old 06-01-2016, 03:06 PM   #3
TIBO
LQ Newbie
 
Registered: May 2015
Posts: 12

Original Poster
Rep: Reputation: Disabled
Thank you for your answer !

Yes, I've also read that it doesn't need to be defragmented.
That's why there is nothing anywhere on the web and that's why I came here to help because I have really no idea how to do that.
Unfortunately, we have to do this tool....

Thanks anyway !
 
Old 06-01-2016, 03:07 PM   #4
Emerson
LQ Sage
 
Registered: Nov 2004
Location: Saint Amant, Acadiana
Distribution: Gentoo ~arch
Posts: 7,231

Rep: Reputation: Disabled
Actually, there is a defragmentation tool for XFS, it works on mounted filesystems and is open source ...
 
Old 06-01-2016, 03:09 PM   #5
jpollard
Senior Member
 
Registered: Dec 2012
Location: Washington DC area
Distribution: Fedora, CentOS, Slackware
Posts: 4,912

Rep: Reputation: 1513Reputation: 1513Reputation: 1513Reputation: 1513Reputation: 1513Reputation: 1513Reputation: 1513Reputation: 1513Reputation: 1513Reputation: 1513Reputation: 1513
NTFS/vfat/fat all exist on Linux.

Your requirement was "defragmentation tool on a Linux system"... but it did NOT specify which filesystem such a defrag tool was to support.

Depending on the level of difficulty, any of the three would serve as the target of such a defrag tool.
 
Old 06-01-2016, 03:48 PM   #6
TIBO
LQ Newbie
 
Registered: May 2015
Posts: 12

Original Poster
Rep: Reputation: Disabled
All right, I see what you mean jpollard.

First, we got to create a fuse filesystem and then, we got to create this tool.
But, like you said, it didn't specify any specific filesystem.

It just says that the tool has to work on "an offline (unmount) filesystem to gather files together".

So, I'll have to try to do this tool on a Microsoft filesystem I guess.

To Emerson, you're talking about xfs_fsr, right ?
 
Old 06-01-2016, 04:52 PM   #7
John VV
LQ Muse
 
Registered: Aug 2005
Location: A2 area Mi.
Posts: 17,534

Rep: Reputation: 2622Reputation: 2622Reputation: 2622Reputation: 2622Reputation: 2622Reputation: 2622Reputation: 2622Reputation: 2622Reputation: 2622Reputation: 2622Reputation: 2622
if you REALLY MUST

then tar up the files
delete the originals
then untar the files

that will remove the fragments that MIGHT be on a ext4 or lvm partition
but i have never seen more than about 1% on my drives and i work with a TON of image data

Last edited by John VV; 06-01-2016 at 04:54 PM.
 
Old 06-01-2016, 04:54 PM   #8
Emerson
LQ Sage
 
Registered: Nov 2004
Location: Saint Amant, Acadiana
Distribution: Gentoo ~arch
Posts: 7,231

Rep: Reputation: Disabled
Well, this was about assignment to create such a tool. That's why I recommended looking at xfs_fsr code, to get ideas how it is done.
 
Old 06-01-2016, 05:39 PM   #9
rknichols
Senior Member
 
Registered: Aug 2009
Distribution: CentOS
Posts: 4,592

Rep: Reputation: 2114Reputation: 2114Reputation: 2114Reputation: 2114Reputation: 2114Reputation: 2114Reputation: 2114Reputation: 2114Reputation: 2114Reputation: 2114Reputation: 2114
If you're trying to be filesystem agnostic, take a look at the manpage for hdparm and scroll down to the "--fibmap" option. That option should give you the information you need to determine whether a file is fragmented.

Without getting very filesystem specific and getting down-and-dirty with the internal structures of the unmounted filesystem, about all you can do to defragment a file is to copy it somewhere else and hope that the copy is less fragmented than the original. Linux doesn't provide any means to control where a file's blocks are allocated on the disk.
 
Old 06-01-2016, 06:59 PM   #10
jefro
Moderator
 
Registered: Mar 2008
Posts: 21,270

Rep: Reputation: 3458Reputation: 3458Reputation: 3458Reputation: 3458Reputation: 3458Reputation: 3458Reputation: 3458Reputation: 3458Reputation: 3458Reputation: 3458Reputation: 3458
I'd tar the files and test each one then back.

Opps been suggested

Last edited by jefro; 06-01-2016 at 10:31 PM.
 
Old 06-01-2016, 07:14 PM   #11
jpollard
Senior Member
 
Registered: Dec 2012
Location: Washington DC area
Distribution: Fedora, CentOS, Slackware
Posts: 4,912

Rep: Reputation: 1513Reputation: 1513Reputation: 1513Reputation: 1513Reputation: 1513Reputation: 1513Reputation: 1513Reputation: 1513Reputation: 1513Reputation: 1513Reputation: 1513
Quote:
Originally Posted by TIBO View Post
All right, I see what you mean jpollard.

First, we got to create a fuse filesystem and then, we got to create this tool.
If NTFS is preferred, then the fuse filesystem is already done.
For the FAT/vFAT, you might not need one. In the unmounted case - it is easy, you directly access the blocks. Examples for this are also easily available (look for the mtools list for manipulating the fat/vfat disk/partition/file). For the mounted case it gets tricky because you have to stop the filesystem from functioning - then move the files that are NOT in use (if it is possible). The problem in the online case is that you WILL have to modify the kernel filesystem code so that the mounted (and in use) buffers DON'T get moved around. As I recall, the FAT/VFAT mounted filesystems don't keep buffers in memory except for the open files. But you would have to check that.
Quote:
But, like you said, it didn't specify any specific filesystem.

It just says that the tool has to work on "an offline (unmount) filesystem to gather files together".

So, I'll have to try to do this tool on a Microsoft filesystem I guess.

To Emerson, you're talking about xfs_fsr, right ?
 
Old 06-01-2016, 07:26 PM   #12
ReaperX7
LQ Guru
 
Registered: Jul 2011
Location: California
Distribution: Slackware64-Current
Posts: 6,460
Blog Entries: 15

Rep: Reputation: 2040Reputation: 2040Reputation: 2040Reputation: 2040Reputation: 2040Reputation: 2040Reputation: 2040Reputation: 2040Reputation: 2040Reputation: 2040Reputation: 2040
Defragmentation on UNIX and UNIX-like systems is not recommended. Because many modern systems use B+ or B- trees when building a file system they create an index or where every part of a file is and loads it accordingly. In the past, ReiserFS aka Reiser3 offered defragmentation of an offline drive, but it proved to be dangerous as it could corrupt the data tree.

Even modern file systems like JFS, Ext4, ZFS, HAMMER, UFS2, BtrFS, and others all use these tree style file systems with lots of indexing and things like copy-on-write abilities to avoid the need of defragmentation using these indexes.

This is why so little information exists. There are some volume managers like BtrFS and ZFS that offer integrity checks, but those are to reduce data corruption only.

Ntfs-3g-progs to assist NTFS on UNIX-like systems, as well as tools to assist vfat/fat12/fat16/exfat/fat32 offers some defragmentation abilities, but these are rudimentary and still, are not recommended.
 
Old 06-01-2016, 07:37 PM   #13
jpollard
Senior Member
 
Registered: Dec 2012
Location: Washington DC area
Distribution: Fedora, CentOS, Slackware
Posts: 4,912

Rep: Reputation: 1513Reputation: 1513Reputation: 1513Reputation: 1513Reputation: 1513Reputation: 1513Reputation: 1513Reputation: 1513Reputation: 1513Reputation: 1513Reputation: 1513
Didn't know that about NTFS... though I did know it was tricky.

That pretty much leaves only FAT/vFat filesystems that could be used (they are a MUCH simpler filesystem).

Unless, of course, you wanted to write your own filesystem (which is what fuse is for).
 
Old 06-02-2016, 03:55 AM   #14
TIBO
LQ Newbie
 
Registered: May 2015
Posts: 12

Original Poster
Rep: Reputation: Disabled
Thank you all for your answers !
I'm going to try and get on with it.
 
Old 06-02-2016, 05:07 AM   #15
hydrurga
LQ Guru
 
Registered: Nov 2008
Location: Pictland
Distribution: Linux Mint 20 MATE
Posts: 8,048
Blog Entries: 5

Rep: Reputation: 2924Reputation: 2924Reputation: 2924Reputation: 2924Reputation: 2924Reputation: 2924Reputation: 2924Reputation: 2924Reputation: 2924Reputation: 2924Reputation: 2924
Just to add some info into the mix, ext4, the most commonly used filesystem on Linux, *does* suffer from fragmentation, contrary to some comments above. However it is very limited due to the way the kernel filesystem driver runs things and thus is not normally problematic.

Multiple extents for the same inode, for example, are not necessarily contiguous and these can be defragmented. What I would recommend is that you download the e2fsprogs source code and have a look at the code for the following programs:

e4defrag - reduces fragmentation of an extent-based file, giving it more contiguous blocks and improving the file access speed.

filefrag - reports on how badly fragmented a file might be.

e2freefrag - scans the block bitmap to check how many free blocks are present as contiguous and aligned free space.

That should give you a start...
 
1 members found this post helpful.
  


Reply


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off



Similar Threads
Thread Thread Starter Forum Replies Last Post
ext3fs defragmentation tool - defrag traene Linux - Hardware 3 01-19-2008 08:12 PM
A file-system defragmentation tool on Linux tmcco Linux - Software 148 05-07-2007 11:29 AM
defragmentation tool czezz Linux - Software 1 02-04-2005 10:35 AM
HDD Defragmentation tool ? membrax Linux - Software 3 01-22-2003 05:30 AM

LinuxQuestions.org > Forums > Linux Forums > Linux - Software

All times are GMT -5. The time now is 06:04 AM.

Main Menu
Advertisement
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
Open Source Consulting | Domain Registration