LinuxQuestions.org
Help answer threads with 0 replies.
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - Software
User Name
Password
Linux - Software This forum is for Software issues.
Having a problem installing a new program? Want to know which application is best for the job? Post your question in this forum.

Notices



Reply
 
Search this Thread
Old 01-16-2011, 12:03 PM   #1
H_TeXMeX_H
Guru
 
Registered: Oct 2005
Location: $RANDOM
Distribution: slackware64
Posts: 12,928
Blog Entries: 2

Rep: Reputation: 1269Reputation: 1269Reputation: 1269Reputation: 1269Reputation: 1269Reputation: 1269Reputation: 1269Reputation: 1269Reputation: 1269
Portable sparse defrag by moving files using dd ?


I want to make a simple, portable sparse defragmentation script, but I'm not sure if the theory behind it is sound, so here's my theory:

Assumptions (all should be true, but check them):

dd can be used to create contiguous files, and is used to make swap files because of this.

Fragmentation occurs when files that exist increase in size so that they collide with the next file on the disk and must be split up.

Linux and Linux filesystems try to keep files sparse so that files have room to expand and the collision is less likely.

Theory:

For all files in the filesystem, I will use dd to make contiguous copies to replace the originals.

I find to find all files on the filesystem 'find / -type f', and for each one use:

Code:
mv file file.defrag
dd bs=4K if=file.defrag of=file
rm -f file.defrag
Requirements:

Space on the filesystem.

Maybe a live CD environment ?

And that should do it ... is it really that simple ? How can I know if this actually works ? Is there any way to see how fragmented files are in Linux ? This should theoretically work regardless of filesystem, as long as it is journaled and running Linux.

EDIT:
This is the usual way to sparse defrag a filesystem that doesn't have a defrag program, like JFS:
https://wiki.archlinux.org/index.php...ragmenting_JFS

It involves moving all files off the disk, recreating the filesystem, and moving them all back. I think it's excessive, and I don't have an extra HDD of large enough size.

Last edited by H_TeXMeX_H; 01-16-2011 at 12:06 PM.
 
Old 01-16-2011, 12:06 PM   #2
stress_junkie
Senior Member
 
Registered: Dec 2005
Location: Massachusetts, USA
Distribution: Ubuntu 10.04 and CentOS 5.5
Posts: 3,873

Rep: Reputation: 331Reputation: 331Reputation: 331Reputation: 331
I don't think so. I believe that the dd command, as used in your script, will just create files the same way as any other method. I think that you would have to use the bs=<filesize> count=1 somewhere in there to achieve your goal, at least in theory.

Don't forget that small files reside entirely in the inodes so they don't suffer from sparseness.

Single block files also do not suffer from sparseness.

I wonder about the practical value of defragmenting files in UNIX/Linux. I've never had a performance problem on systems that have been running for years. I should probably phrase that as "I've never seen disk/file performance degrade over time on UNIX/Linux systems".

These days it is barely worth doing even on Windows systems since disks are so fast. I tell my clients to only defrag their files once a year or less and only if they really feel compelled to do it at all. On Windows systems the performance degradation comes from the registry getting messed up over time, not from file fragmentation. (If you reinstall Windows the machine will usually run as fast as the day it was purchased even though the files are severely fragmented.)

I remember when Winchester disk (PC disks) data throughput was measured in KB/sec. That was slow enough to make file defragmentation worthwhile. I don't see the value of it today.

NOTE: THIS NEXT PART IS INCORRECT!!!
And another thing. Don't forget that modern disks reorganize disk writes in their buffer at the hardware level to optimize their access to the platters thereby reducing the benefit of defragmenting files on disk since they are probably being (...partially at least...) defragmented (...or more importantly, optimized...) automatically in the disk buffer. (I'm not really sure if this is completely correct. Actually, the more I think about it the more I think it is not correct.)

Last edited by stress_junkie; 01-17-2011 at 08:36 AM.
 
Old 01-17-2011, 03:10 AM   #3
JZL240I-U
Senior Member
 
Registered: Apr 2003
Location: Germany
Distribution: openSuSE 13.1 / 12.3_64-KDE, Ubuntu 14.04, Fedora 20, Mint 17, Chakra
Posts: 3,689

Rep: Reputation: Disabled
I think you crash already with the "mv". As I understand things the way you used it is a "rename" (look at the upper section of http://linux.die.net/man/1/mv). You'll at least have to move it to a different file system i.e. partition.

Last edited by JZL240I-U; 01-17-2011 at 03:12 AM.
 
1 members found this post helpful.
Old 01-17-2011, 06:54 AM   #4
jhuizer
LQ Newbie
 
Registered: Apr 2009
Posts: 12

Rep: Reputation: 0
Quote:
Originally Posted by stress_junkie View Post
And another thing. Don't forget that modern disks reorganize disk writes in their buffer at the hardware level to optimize their access to the platters thereby reducing the benefit of defragmenting files on disk since they are probably being (...partially at least...) defragmented (...or more importantly, optimized...) automatically in the disk buffer. (I'm not really sure if this is completely correct. Actually, the more I think about it the more I think it is not correct.)
I thought it was the Linux kernel reordering reads/writes for optimal performance.
 
Old 01-17-2011, 07:27 AM   #5
JZL240I-U
Senior Member
 
Registered: Apr 2003
Location: Germany
Distribution: openSuSE 13.1 / 12.3_64-KDE, Ubuntu 14.04, Fedora 20, Mint 17, Chakra
Posts: 3,689

Rep: Reputation: Disabled
Quote:
Originally Posted by jhuizer View Post
I thought it was the Linux kernel reordering reads/writes for optimal performance.
Depends on the file system (and its version, i.e. age). For ext4 it is moved back from the kernel to the file system in the coming / newest(?) version, of the others I don't know.
 
Old 01-17-2011, 08:30 AM   #6
stress_junkie
Senior Member
 
Registered: Dec 2005
Location: Massachusetts, USA
Distribution: Ubuntu 10.04 and CentOS 5.5
Posts: 3,873

Rep: Reputation: 331Reputation: 331Reputation: 331Reputation: 331
Quote:
Originally Posted by jhuizer View Post
I thought it was the Linux kernel reordering reads/writes for optimal performance.
Yes. You are correct.

I was confusing the way that a disk can reorder the read/writes in its buffer with file optimization. Actually the disk will only put the file fragments on the locations that the kernel tells it to place them. Therefore the disk buffer will not optimize file fragment placement on the disk.

What actually does happen in the disk buffer is a reordering of the read/write commands to optimize the head movement during execution of the list of commands. And this is only on newer disks.

I considered removing the erroneous part of my original post but I felt that it would be cheating. That is why I made those comments at the end.

Last edited by stress_junkie; 01-17-2011 at 08:33 AM.
 
1 members found this post helpful.
Old 01-17-2011, 11:59 AM   #7
H_TeXMeX_H
Guru
 
Registered: Oct 2005
Location: $RANDOM
Distribution: slackware64
Posts: 12,928
Blog Entries: 2

Original Poster
Rep: Reputation: 1269Reputation: 1269Reputation: 1269Reputation: 1269Reputation: 1269Reputation: 1269Reputation: 1269Reputation: 1269Reputation: 1269
One other way I was considering was indeed moving files off the disk onto another disk and moving them back.

I also agree that fragmentation will be experienced mostly with larger files, so I'll probably find files greater than the block size ... right ?

Still, why wouldn't it work with copying files within the filesystem, a new copy of the file is made, and the file is NOT fragmented when the copy is made ... or at least I think it isn't.

That's the problem, there's no tool to asses file fragmentation, if anyone knows of one, please list it. I know there's one for Window$, but that one is not very specific (like which exact files).
 
Old 01-17-2011, 12:02 PM   #8
JZL240I-U
Senior Member
 
Registered: Apr 2003
Location: Germany
Distribution: openSuSE 13.1 / 12.3_64-KDE, Ubuntu 14.04, Fedora 20, Mint 17, Chakra
Posts: 3,689

Rep: Reputation: Disabled
You might deign to read the link I provided...
 
Old 01-17-2011, 12:17 PM   #9
H_TeXMeX_H
Guru
 
Registered: Oct 2005
Location: $RANDOM
Distribution: slackware64
Posts: 12,928
Blog Entries: 2

Original Poster
Rep: Reputation: 1269Reputation: 1269Reputation: 1269Reputation: 1269Reputation: 1269Reputation: 1269Reputation: 1269Reputation: 1269Reputation: 1269
I did. I rename the file with 'mv' then copy it to a new file with the original name using 'dd' (this one should in theory not be fragmented), then I delete the one I moved.
 
Old 01-17-2011, 12:23 PM   #10
JZL240I-U
Senior Member
 
Registered: Apr 2003
Location: Germany
Distribution: openSuSE 13.1 / 12.3_64-KDE, Ubuntu 14.04, Fedora 20, Mint 17, Chakra
Posts: 3,689

Rep: Reputation: Disabled
If you rename a file, its internal structure / fragmentation will not be affected.

When you use "dd" my understanding is, that it faithfully reproduces the aforementioned structure, else you might as well use "cp" (why do you use "dd" and not "cp" by the way?). Perhaps you'd better ask here: http://www.linuxquestions.org/questi...ommand-362506/

Last edited by JZL240I-U; 01-17-2011 at 12:28 PM.
 
Old 01-18-2011, 11:06 AM   #11
H_TeXMeX_H
Guru
 
Registered: Oct 2005
Location: $RANDOM
Distribution: slackware64
Posts: 12,928
Blog Entries: 2

Original Poster
Rep: Reputation: 1269Reputation: 1269Reputation: 1269Reputation: 1269Reputation: 1269Reputation: 1269Reputation: 1269Reputation: 1269Reputation: 1269
I was hoping dd would only create files that are not fragmented, as it does when creating a swap file. I don't know if cp does this.

Either way, when I have more time, I will try to mess with a script like this and post the results if I can quantify them in some way.
 
Old 01-18-2011, 05:00 PM   #12
jefro
Guru
 
Registered: Mar 2008
Posts: 12,087

Rep: Reputation: 1521Reputation: 1521Reputation: 1521Reputation: 1521Reputation: 1521Reputation: 1521Reputation: 1521Reputation: 1521Reputation: 1521Reputation: 1521Reputation: 1521
Why not use tar?
 
Old 01-19-2011, 03:39 AM   #13
JZL240I-U
Senior Member
 
Registered: Apr 2003
Location: Germany
Distribution: openSuSE 13.1 / 12.3_64-KDE, Ubuntu 14.04, Fedora 20, Mint 17, Chakra
Posts: 3,689

Rep: Reputation: Disabled
Well, I offered a link to the one true and real dd thread on LQ...
 
Old 01-19-2011, 12:22 PM   #14
H_TeXMeX_H
Guru
 
Registered: Oct 2005
Location: $RANDOM
Distribution: slackware64
Posts: 12,928
Blog Entries: 2

Original Poster
Rep: Reputation: 1269Reputation: 1269Reputation: 1269Reputation: 1269Reputation: 1269Reputation: 1269Reputation: 1269Reputation: 1269Reputation: 1269
Quote:
Originally Posted by jefro View Post
Why not use tar?
How would I use it here ?
 
Old 05-16-2011, 05:19 AM   #15
H_TeXMeX_H
Guru
 
Registered: Oct 2005
Location: $RANDOM
Distribution: slackware64
Posts: 12,928
Blog Entries: 2

Original Poster
Rep: Reputation: 1269Reputation: 1269Reputation: 1269Reputation: 1269Reputation: 1269Reputation: 1269Reputation: 1269Reputation: 1269Reputation: 1269
It looks like 'filefrag' was the program I needed to test this theory. So far it seems that a mere 'cp' will work and will defragment files. 'dd' also works to about the same extent.
 
  


Reply


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off


Similar Threads
Thread Thread Starter Forum Replies Last Post
copy sparse files tincboy Linux - Newbie 14 07-07-2010 02:47 AM
MLDonkey/mlnet: creating sparse files - undesirable alexander_bosakov Linux - Software 0 02-26-2008 05:10 PM
disk defrag and temp files james2b Linux - General 27 09-03-2007 09:17 AM
reserving space on the disk for sparse files madhukirant Programming 1 08-17-2005 08:29 PM
defrag/temp files mindseye Linux - Newbie 1 01-18-2004 12:28 PM


All times are GMT -5. The time now is 07:32 AM.

Main Menu
Advertisement
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
identi.ca: @linuxquestions
Facebook: linuxquestions Google+: linuxquestions
Open Source Consulting | Domain Registration