LinuxQuestions.org
View the Most Wanted LQ Wiki articles.
Go Back   LinuxQuestions.org > Forums > Non-*NIX Forums > Programming
User Name
Password
Programming This forum is for all programming questions.
The question does not have to be directly related to Linux and any language is fair game.

Notices

Reply
 
Search this Thread
Old 05-04-2010, 12:00 AM   #1
bdoremus
LQ Newbie
 
Registered: May 2010
Posts: 12

Rep: Reputation: 0
Question Bash: recursion = bad?


Hello all,
new Bash programmer here. I've dealt with other languages in the past, but I honestly think that's not to my advantage right now.

GOAL:
Starting in a directory, search it and all sub directories for any files containing a certain text string, then copy those files to another directory. There's about 1GB worth of files (all email messages) I want to search through.

PROBLEM:
Since I don't know the depth of the folders, my instinct was to create a recursive script. Unfortunately, I have to run this on a server... and I'm guessing it'll completely kill everything.
I'm hoping there's a cleaner solution?

Code:
#!/bin/bash

#navigate to the backups
cd /mnt/home/user/.mailbox/

for f in *; do

   if [ -f "$f" ]; then # it was a file
      #here is where I'd check for the string
      #if it matched, I'd copy it to the directory

   else # it was a folder
      cd "$f"
      /bin/emailSearch.sh # the script calls itself in the new subdirectory
      cd ../
   fi
done
Help?!
 
Old 05-04-2010, 12:23 AM   #2
SuperJediWombat!
Member
 
Registered: Apr 2009
Location: Perth, Australia
Distribution: Ubuntu/CentOS
Posts: 208

Rep: Reputation: 50
Code:
man find
Something like this will do. If you are expecting to match a large volume of files, you may want to look at piping find into xargs rather than using -exec.
Code:
find /mnt/home/user/.mailbox/ -iname "string" -type f -exec mv '{}' /mnt/home/user/thefolder/ \;

Last edited by SuperJediWombat!; 05-04-2010 at 12:33 AM.
 
Old 05-04-2010, 12:51 AM   #3
bdoremus
LQ Newbie
 
Registered: May 2010
Posts: 12

Original Poster
Rep: Reputation: 0
doesn't find -iname just search for the file name? I need to check text inside the file itself, or perhaps I'm reading the man wrong..?

The check is rather complicated, too. If any part of the "to" field matches three or more of a set of 10 users, it's considered a hit. I can use regular expressions to accomplish this, but again I don't think find can do this inside of a file.

Or am I missing something?
 
Old 05-04-2010, 01:25 AM   #4
catkin
LQ 5k Club
 
Registered: Dec 2008
Location: Tamil Nadu, India
Distribution: Servers: Debian Squeeze and Wheezy. Desktop: Slackware64 14.0. Netbook: Slackware 13.37
Posts: 8,551
Blog Entries: 28

Rep: Reputation: 1176Reputation: 1176Reputation: 1176Reputation: 1176Reputation: 1176Reputation: 1176Reputation: 1176Reputation: 1176Reputation: 1176
You could run the script via the nice command to reduce its impact on the server and if execution time is not important you could add a sleep command in the script too.

You could use the file command to restrict the search to appropriate file types.

You could use find to run the script in each directory but that gains little over recursion.
 
Old 05-04-2010, 01:40 AM   #5
A.Thyssen
Member
 
Registered: May 2006
Location: Brisbane, Australia
Posts: 119

Rep: Reputation: 32
Have a look at grep -R option
This is a recursive grep.

If you combine it with the -l option you get a list of all files containing the search string.
 
Old 05-04-2010, 02:31 AM   #6
SuperJediWombat!
Member
 
Registered: Apr 2009
Location: Perth, Australia
Distribution: Ubuntu/CentOS
Posts: 208

Rep: Reputation: 50
My mistake, I misread your post and thought that you only needed to search the filenames.
Code:
find . -type f | xargs grep -s -l "test" | xargs -I {} cp {} ~/

Last edited by SuperJediWombat!; 05-05-2010 at 07:39 AM.
 
Old 05-04-2010, 02:44 AM   #7
grail
Guru
 
Registered: Sep 2009
Location: Perth
Distribution: Manjaro
Posts: 7,537

Rep: Reputation: 1907Reputation: 1907Reputation: 1907Reputation: 1907Reputation: 1907Reputation: 1907Reputation: 1907Reputation: 1907Reputation: 1907Reputation: 1907Reputation: 1907
+1 to A. Thyssen - and if you use the -f option with the name of the 10 users in a file you can cover that part too
 
Old 05-04-2010, 11:14 PM   #8
bdoremus
LQ Newbie
 
Registered: May 2010
Posts: 12

Original Poster
Rep: Reputation: 0
Awesome, thanks guys!

I was able to rewrite it using grep -rl; definitely cleaned things up a lot!

@grail: I wasn't able to find any documentation on the "-f" option; maybe I'm not looking in the right place (man grep)?

@Super: I'm going to come back to this; it's taking me a while to decipher. Thanks!
 
Old 05-05-2010, 12:05 AM   #9
SuperJediWombat!
Member
 
Registered: Apr 2009
Location: Perth, Australia
Distribution: Ubuntu/CentOS
Posts: 208

Rep: Reputation: 50
Try this
Quote:
grep -R -s -l "test" /mnt/home/user/.mailbox/ | xargs -I {} cp {} /mnt/home/user/movedir/
 
Old 05-05-2010, 03:02 AM   #10
grail
Guru
 
Registered: Sep 2009
Location: Perth
Distribution: Manjaro
Posts: 7,537

Rep: Reputation: 1907Reputation: 1907Reputation: 1907Reputation: 1907Reputation: 1907Reputation: 1907Reputation: 1907Reputation: 1907Reputation: 1907Reputation: 1907Reputation: 1907
Yes to 'man grep':

Code:
-f FILE, --file=FILE
	      Obtain patterns from FILE, one per line.	The  empty  file  con-
	      tains zero patterns, and therefore matches nothing.
 
Old 05-05-2010, 09:01 PM   #11
bdoremus
LQ Newbie
 
Registered: May 2010
Posts: 12

Original Poster
Rep: Reputation: 0
rightyo- I've restructured my program after learning about the tools you all mentioned here; thanks!
 
Old 05-05-2010, 09:33 PM   #12
SuperJediWombat!
Member
 
Registered: Apr 2009
Location: Perth, Australia
Distribution: Ubuntu/CentOS
Posts: 208

Rep: Reputation: 50
Can you post it?
 
Old 05-05-2010, 09:49 PM   #13
bdoremus
LQ Newbie
 
Registered: May 2010
Posts: 12

Original Poster
Rep: Reputation: 0
It's still broken, but with other problems

In essence, what I did was:
1) run a find to get a list of all files in the applicable folders (current problem, see other post)
allFiles=$(find $src -print)
2) grep through the files to see if they had the string I wanted, and save the file paths of the matches to a variable
files=$(grep -il -E "^To:.*($bod|$admin)" $allFiles)
3) iterate through the matches to modify the files,
for f in $files; do
...

4) then copy them over
cp "$fSrc" "$dest$fName"

Thanks again! I've learned a ton so far because of this little project!
 
Old 05-06-2010, 09:47 AM   #14
tuxdev
Senior Member
 
Registered: Jul 2005
Distribution: Slackware
Posts: 2,012

Rep: Reputation: 111Reputation: 111
Code:
allFiles=$(find $src -print)
It's almost never correct to put the results of find into a variable like thatk..

Code:
files=$(grep -il -E "^To:.*($bod|$admin)" $allFiles)
..because properly quoting allFiles is one big long string, and not quoting leads to word-splitting problems.

Code:
for f in $files; do
Again, not quoting means word-splitting, and that's almost always a Bad Thing.

You can also use this sort of loop:
Code:
while IFS="" read -r -d "" file ; do
   if grep -iq -E "^To:.*($bod|$admin)" "$file ; then
      cp ...
   fi
done < <(find "$src" -print0)
 
Old 05-06-2010, 11:47 AM   #15
catkin
LQ 5k Club
 
Registered: Dec 2008
Location: Tamil Nadu, India
Distribution: Servers: Debian Squeeze and Wheezy. Desktop: Slackware64 14.0. Netbook: Slackware 13.37
Posts: 8,551
Blog Entries: 28

Rep: Reputation: 1176Reputation: 1176Reputation: 1176Reputation: 1176Reputation: 1176Reputation: 1176Reputation: 1176Reputation: 1176Reputation: 1176
Quote:
Originally Posted by tuxdev View Post
Code:
while IFS="" read -r -d "" file ; do
   if grep -iq -E "^To:.*($bod|$admin)" "$file ; then
      cp ...
   fi
done < <(find "$src" -print0)
As someone pointed out in another thread (CTOP) there is no need for the IFS="" when each line is being read into a single variable, in this case $file.
 
  


Reply

Tags
bash, folder, programming, recursion, search


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off


Similar Threads
Thread Thread Starter Forum Replies Last Post
New to Bash | Need help with recursion. VauxhallVXR Programming 2 05-30-2009 05:47 PM
using recursion in bash script drkstr Linux - Software 4 07-09-2006 08:48 PM
Bash Local Variable Recursion With Array jshivers Programming 0 06-16-2006 04:31 PM
tar: '--no-recursion' option doesn't prevent recursion Earl Parker II Slackware 12 08-17-2004 02:49 AM
Writing bash script with recursion.. ray5_83 Programming 4 08-04-2004 05:44 PM


All times are GMT -5. The time now is 07:09 AM.

Main Menu
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
identi.ca: @linuxquestions
Facebook: linuxquestions Google+: linuxquestions
Open Source Consulting | Domain Registration