LinuxQuestions.org
Latest LQ Deal: Complete CCNA, CCNP & Red Hat Certification Training Bundle
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - Newbie
User Name
Password
Linux - Newbie This Linux forum is for members that are new to Linux.
Just starting out and have a question? If it is not in the man pages or the how-to's this is the place!

Notices


Reply
  Search this Thread
Old 05-26-2010, 09:58 PM   #1
Osirix
LQ Newbie
 
Registered: Jul 2006
Posts: 10

Rep: Reputation: 0
How to process billions of small files using Bash shell commands with limited memory?


I need to process billions of small files using bash shell commands with limited memory size (256MB). If any of those files contain certain "keywords", the file will be removed.

I tried with command:
find . -type f -exec grep -i -l -H "keyword" '{}' + | xargs rm -rf

where all files are located within the current directory. But the command above failed in "out of memory".

May I know any tip or tricks from you guru in handling such a task?
 
Old 05-26-2010, 10:07 PM   #2
grail
LQ Guru
 
Registered: Sep 2009
Location: Perth
Distribution: Manjaro
Posts: 9,506

Rep: Reputation: 2890Reputation: 2890Reputation: 2890Reputation: 2890Reputation: 2890Reputation: 2890Reputation: 2890Reputation: 2890Reputation: 2890Reputation: 2890Reputation: 2890
You could put it in a loop so only processing one file at a time (assuming time to do so is not an issue):
Code:
while read file
do
    if grep -q -i -l -H "keyword" "$file"
    then
        rm -rf "$file"
    fi
done< <(find . -type f)
There is a space between the two <
 
  


Reply


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off



Similar Threads
Thread Thread Starter Forum Replies Last Post
Graphic status bar when transfering files using shell commands? k0z1 Linux - Newbie 2 06-16-2009 11:49 AM
copy directories and files recursively using C shell or bash shell bostonuser Programming 7 06-06-2008 01:24 AM
how to export a bash shell variable before init.d process start pnsidhu Linux - Newbie 1 06-04-2008 08:21 AM
LXer: Billions and billions of...lines of proprietary code to go open source LXer Syndicated Linux News 0 11-27-2007 01:40 PM
making .tif files into animated .gif files (bash shell, Red Hat 7.2) illiniguy3043 Linux - Newbie 1 06-01-2004 04:04 PM


All times are GMT -5. The time now is 06:55 PM.

Main Menu
Advertisement
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
Facebook: linuxquestions Google+: linuxquestions
Open Source Consulting | Domain Registration