LinuxQuestions.org
Visit Jeremy's Blog.
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - Newbie
User Name
Password
Linux - Newbie This Linux forum is for members that are new to Linux.
Just starting out and have a question? If it is not in the man pages or the how-to's this is the place!

Notices


Reply
  Search this Thread
Old 11-02-2008, 04:31 AM   #1
nbdr
LQ Newbie
 
Registered: Nov 2008
Posts: 6

Rep: Reputation: 0
Quicker way to delete folders than rm -r folder_name


Hello,

I need to delete 100,000 of files in a massive directory tree daily.

Using rm -r folder_name takes hours.

Is there another linux command that deletes folder trees faster?

Thanks,
nbdr
 
Old 11-02-2008, 07:06 AM   #2
ilikejam
Senior Member
 
Registered: Aug 2003
Location: Glasgow
Distribution: Fedora / Solaris
Posts: 3,109

Rep: Reputation: 96
Hi.

Nope. rm's as good as it gets. You might want to look into using a different filesystem, though.

Dave.
 
Old 11-02-2008, 07:20 AM   #3
syg00
LQ Veteran
 
Registered: Aug 2003
Location: Australia
Distribution: Lots ...
Posts: 16,001

Rep: Reputation: 2219Reputation: 2219Reputation: 2219Reputation: 2219Reputation: 2219Reputation: 2219Reputation: 2219Reputation: 2219Reputation: 2219Reputation: 2219Reputation: 2219
Which can be read (at least) two ways.
Me, I might put them in a separate partition, and just do a mkfs on it. Bet it wouldn't take too long ...
 
Old 11-02-2008, 08:37 AM   #4
pixellany
LQ Veteran
 
Registered: Nov 2005
Location: Annapolis, MD
Distribution: Arch/XFCE
Posts: 17,802

Rep: Reputation: 738Reputation: 738Reputation: 738Reputation: 738Reputation: 738Reputation: 738Reputation: 738
Quote:
Originally Posted by nbdr View Post
Hello,

I need to delete 100,000 of files in a massive directory tree daily.

Using rm -r folder_name takes hours.

Is there another linux command that deletes folder trees faster?

Thanks,
nbdr
Welcome to LQ!!

What you need is some way of defining which folders to delete. This could be something simple like having them all in one place, or it could be a unique string in the filename or extension.

Take a look at the "find" command, with all its options.
 
Old 11-02-2008, 01:21 PM   #5
nbdr
LQ Newbie
 
Registered: Nov 2008
Posts: 6

Original Poster
Rep: Reputation: 0
Quote:
Originally Posted by pixellany View Post
Welcome to LQ!!

What you need is some way of defining which folders to delete. This could be something simple like having them all in one place, or it could be a unique string in the filename or extension.

Take a look at the "find" command, with all its options.

Thanks for the answers.

I know which folders to delete. they are all under 'cache' folder. What do I do then that is faster than rm -r cache/ ?

-Nbdr
 
Old 11-02-2008, 01:26 PM   #6
i92guboj
Gentoo support team
 
Registered: May 2008
Location: Lucena, Córdoba (Spain)
Distribution: Gentoo
Posts: 4,074

Rep: Reputation: 387Reputation: 387Reputation: 387Reputation: 387
Nothing.

If rm could be faster, it would be faster. It has had many decades to improve.

Most of the stuff is done by the fs driver, so, the only way to improve the performance would be to look into the file system (change filesystem, tune it, change options when formatting, etc. etc. etc.).
 
Old 11-02-2008, 01:50 PM   #7
john test
Member
 
Registered: Jul 2008
Distribution: ubuntu 9.10
Posts: 527
Blog Entries: 1

Rep: Reputation: 35
probably
Code:
rm -rf /cache
would be somewhat faster.
 
Old 11-02-2008, 01:54 PM   #8
Quakeboy02
Senior Member
 
Registered: Nov 2006
Distribution: Debian Squeeze 2.6.32.9 SMP AMD64
Posts: 3,318

Rep: Reputation: 126Reputation: 126
Quote:
Originally Posted by nbdr View Post
I need to delete 100,000 of files in a massive directory tree daily.
If it's just junk, why are you even saving it? What is it, anyway, if I can ask?
 
Old 11-02-2008, 02:11 PM   #9
H_TeXMeX_H
LQ Guru
 
Registered: Oct 2005
Location: $RANDOM
Distribution: slackware64
Posts: 12,928
Blog Entries: 2

Rep: Reputation: 1288Reputation: 1288Reputation: 1288Reputation: 1288Reputation: 1288Reputation: 1288Reputation: 1288Reputation: 1288Reputation: 1288
What filesystem are you using ? If it's ext3 then I can see why it takes hours, I bet the same would only take minutes with JFS (fastest delete speed) or XFS.
 
Old 11-02-2008, 03:12 PM   #10
i92guboj
Gentoo support team
 
Registered: May 2008
Location: Lucena, Córdoba (Spain)
Distribution: Gentoo
Posts: 4,074

Rep: Reputation: 387Reputation: 387Reputation: 387Reputation: 387
Quote:
Originally Posted by H_TeXMeX_H View Post
What filesystem are you using ? If it's ext3 then I can see why it takes hours, I bet the same would only take minutes with JFS (fastest delete speed) or XFS.
That's not my experience with ext3 at all.

Code:
$ time for dir1 in $(seq 1 1000); do mkdir $dir1; for dir2 in $(seq 1 100); do mkdir $dir1/$dir2; done; done

real    5m14.906s
user    1m19.725s
sys     2m56.671s
$ time rm -rf *

real    0m9.218s
user    0m0.584s
sys     0m6.224s
Under ten seconds for 100,000 directories. This is a sempron 3000+ (relatively old machine) with a sata disk (not sata 2). Filesystem is ext3, it's formated with -O dir_index, though. Creation of directories is not that fast, however that's to be expected.

Out of curiosity, I tested a loopback fs formated and mounted with the standard options (no dir_index), just to be fair. The results are very similar.

Code:
real    5m55.984s
user    1m44.103s
sys     3m1.987s
$ time rm -rf *

real    0m10.178s
user    0m0.583s
sys     0m7.564s
A really small difference.

From my experience, I know that ext3 is a very stable and fast filesystem overall even if people usually don't like to admit it because of I don't know what reason. Sure that some other fs's do X thing better, but they also do other things *much* worse. I find that ext3 does everything adequately.



If the OP really find that deleting 100,000 files takes that long, there are a number of probably causes.
  • Defective or experimental fs (not ext3), like reiser4 or ext4. I don't know if reiserfs (3.x) can have problems with this, but I know from first hand that it does have serious problems with fragmentation.
  • Defective hardware, look on the dmesg output for I/O errors when doing fs operations.
  • Your cpu is being hogged by something else. Check top or htop.

There might be other possible problems. But rm is not one of them.
 
Old 11-02-2008, 09:10 PM   #11
chrism01
LQ Guru
 
Registered: Aug 2004
Location: Sydney
Distribution: Centos 6.9, Centos 7.3
Posts: 17,417

Rep: Reputation: 2397Reputation: 2397Reputation: 2397Reputation: 2397Reputation: 2397Reputation: 2397Reputation: 2397Reputation: 2397Reputation: 2397Reputation: 2397Reputation: 2397
As mentioned, rm is pretty quick. What might be slowing it down is updating the dir files as it goes. Try re-mounting with -noatime.
Alternately, as said, make it a separate partition and use mkfs.
 
Old 11-02-2008, 10:38 PM   #12
syg00
LQ Veteran
 
Registered: Aug 2003
Location: Australia
Distribution: Lots ...
Posts: 16,001

Rep: Reputation: 2219Reputation: 2219Reputation: 2219Reputation: 2219Reputation: 2219Reputation: 2219Reputation: 2219Reputation: 2219Reputation: 2219Reputation: 2219Reputation: 2219
Hadn't seen post by i92guboj - I did some tests too. I just created 100000 copies of a small (few hundred bytes) file. Took just less than 10 and a half minutes.
Rebooted and "rm-rf ..." - less than 10 seconds.

Hardware RAID5 on an old idle quad (P-III based) Xeon server. EXT3 mounted noatime, nodiratime - because I always have them that way.
 
Old 11-03-2008, 01:27 AM   #13
nbdr
LQ Newbie
 
Registered: Nov 2008
Posts: 6

Original Poster
Rep: Reputation: 0
Thank you all

Its a cache of a website that hosted on a shared server.

I think that the files are stored on a storage cluster. Don't have any control of the file system or other stuff. The problem is that I exceed the 500,000 files limit every few days and delete manually until I optimize the caching.

Thanks again,
Nbdr
 
Old 11-03-2008, 01:31 AM   #14
jay73
LQ Guru
 
Registered: Nov 2006
Location: Belgium
Distribution: Ubuntu 11.04, Debian testing
Posts: 5,019

Rep: Reputation: 130Reputation: 130
So if you do that daily, wouldn't it make sense to set up a cron job or something that takes care of it for you?
 
Old 11-03-2008, 02:32 AM   #15
i92guboj
Gentoo support team
 
Registered: May 2008
Location: Lucena, Córdoba (Spain)
Distribution: Gentoo
Posts: 4,074

Rep: Reputation: 387Reputation: 387Reputation: 387Reputation: 387
Quote:
Originally Posted by jay73 View Post
So if you do that daily, wouldn't it make sense to set up a cron job or something that takes care of it for you?
++

That's the way to go. Just create a cron job. He might consider using an higher niceness so it doesn't hit the cpu so badly, though, sincerely, in a cluster I don't think that cpu is the problem. I am rather inclined to think that's something to do with the fs or the hardware.

Last edited by i92guboj; 11-03-2008 at 02:36 AM.
 
  


Reply


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off



Similar Threads
Thread Thread Starter Forum Replies Last Post
Can't delete folders from Windows Tenover Linux - Networking 8 12-24-2005 12:55 AM
Can't delete Folders. duffmckagan SUSE / openSUSE 1 09-19-2005 12:09 PM
HELP: can't delete files/folders :( kevingpo General 1 12-07-2004 08:51 PM
Can you delete folders after ./configure ? iLLuSionZ Linux - Newbie 2 11-19-2003 08:39 PM
Am I able to delete folders after install? SuSE_Surfer Linux - General 6 10-18-2003 12:41 AM

LinuxQuestions.org > Forums > Linux Forums > Linux - Newbie

All times are GMT -5. The time now is 04:56 AM.

Main Menu
Advertisement
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
Facebook: linuxquestions Google+: linuxquestions
Open Source Consulting | Domain Registration