Visit Jeremy's Blog.
Go Back > Forums > Linux Forums > Linux - Software
User Name
Linux - Software This forum is for Software issues.
Having a problem installing a new program? Want to know which application is best for the job? Post your question in this forum.


  Search this Thread
Old 11-18-2010, 10:10 AM   #1
Registered: Oct 2008
Distribution: Slackware
Posts: 82

Rep: Reputation: 12
find | xargs | shred seems really slow

What am I doing wrong here ? Shredding a directory with files is incredibly slow. If I create a file of similar size it is around 1000 times faster. Filesystem is ext4, OS is Slack current.

Looking at top it shows shred's status as 'D' which, according to the man page, means uninterruptible sleep ! Also /usr/bin/time shows around 10 minutes but that time was printed on stdout approx. 7 minutes before the command prompt reappeared !

[virgil@thunderbird2:~/q] $ du -sh 
336M	.
[virgil@thunderbird2:~/q] $ find . -type d | wc
    160     160   12038
[virgil@thunderbird2:~/q] $ find . -type f | wc
   6721    6721  687549
[virgil@thunderbird2:~/q] $ /usr/bin/time find . -type f -print0 | xargs -0 -r shred -zu
0.00user 0.00system 10:11.76elapsed 0%CPU (0avgtext+0avgdata 3616maxresident)k
0inputs+0outputs (0major+285minor)pagefaults 0swaps

[virgil@thunderbird2:~] $ dd if=/dev/urandom of=qwerty bs=336M count=1
1+0 records in
1+0 records out
352321536 bytes (352 MB) copied, 51.6693 s, 6.8 MB/s
[virgil@thunderbird2:~] $ ls -l qwerty 
-rw-r--r-- 1 virgil virgil 352321536 2010-11-18 15:13 qwerty
[virgil@thunderbird2:~] $ /usr/bin/time shred -zu qwerty 
0.92user 0.78system 0:16.94elapsed 10%CPU (0avgtext+0avgdata 2912maxresident)k
0inputs+2752528outputs (0major+235minor)pagefaults 0swaps
Old 11-19-2010, 07:17 AM   #2
David the H.
Bash Guru
Registered: Jun 2004
Location: Osaka, Japan
Distribution: Debian sid + kde 3.5 & 4.4
Posts: 6,826

Rep: Reputation: 1973Reputation: 1973Reputation: 1973Reputation: 1973Reputation: 1973Reputation: 1973Reputation: 1973Reputation: 1973Reputation: 1973Reputation: 1973Reputation: 1973
I don't know exactly why you're showing such performance, although I suppose any command that has to work through the kernel's I/O system will likely be slower than one like dd that can do direct disk operations. Not to mention that shred needs to overwrite the disk several times in order to destroy data, as opposed to simply creating a file, which takes only a single write operation. There may be buffering effects going on as well.

The find | xargs pipe chain can only slow it down even more. But you really shouldn't need to use xargs here anyway. find has the equivalent ability to apply commands on it's own, using the -exec option.
find . -type f -exec shred -zu '{}' \+
The \+ at the end of the -exec command makes it perform the same way xargs does, with as many files as possible concatenated into a single run. If your command can't handle the multiple inputs, you'd have to use \; instead, which will make it run separately for each file find feeds it.
Old 11-19-2010, 09:33 AM   #3
Registered: Oct 2008
Distribution: Slackware
Posts: 82

Original Poster
Rep: Reputation: 12
Thanks for the informative reply. Using find this way speeds it up considerably. I don't know what has changed but this slow deletion is a recent thing. I run a cron job that tidies up the log files and also shreds all the files in the squid proxy cache. This directory is always around 93M in size so the change in time the script was running was quite noticeable. I did change to ext4 from ext3 around a month ago, I wonder if this is the reason. I'll have to investigate further.
Old 11-19-2010, 09:36 AM   #4
LQ Guru
Registered: Oct 2005
Location: $RANDOM
Distribution: slackware64
Posts: 12,928
Blog Entries: 2

Rep: Reputation: 1288Reputation: 1288Reputation: 1288Reputation: 1288Reputation: 1288Reputation: 1288Reputation: 1288Reputation: 1288Reputation: 1288
Also make sure the HDD is not failing.


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off

Similar Threads
Thread Thread Starter Forum Replies Last Post
Removing directory that is too large to use find and xargs Sometime Linux - Newbie 1 02-21-2008 07:11 AM
File name expansion with {} in gnu find and xargs anamericanjoe Linux - Software 2 09-16-2006 03:31 PM
clarification on find + xargs rm command dtra Linux - General 2 05-17-2006 05:56 AM
du on specific files from find command (use xargs?) fireman949 Linux - Software 2 07-10-2005 01:42 AM
error with find . | xargs cbonar Linux - Newbie 7 12-09-2004 11:22 AM > Forums > Linux Forums > Linux - Software

All times are GMT -5. The time now is 10:18 AM.

Main Menu
Write for LQ is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
Facebook: linuxquestions Google+: linuxquestions
Open Source Consulting | Domain Registration