need script to copy all files containing a certain string to a directory

konsolebox · 04-15-2013, 08:14 AM

This is a very old thread and should have been not revived, but since we're here already, might as well give a suggestion.

Quote:

Originally Posted by dc6463

thanks all for your help. that was quick.

actually, the code in the first post from marghorp worked fine -on os x at least.

now, my new problem is how to make this work on a folder of 10,000 text files without getting the 'Argument list too long." our mail server throws the backup of all outgoing mails for all users into one folder and i need a way to restore for just one user.

this script works great, but i have to split up the folder first.

any other ideas?

thanks again!

The simple concept for that actually is to pipe output to 'while read LINE; do ...; done'. A method not well known in those days.. at least not until 2006.

For a concept material this could be an example.

Code:

#!/bin/bash

PATTERN_FILES=(*.txt)  # pathname pattern could be a path like /somewhere/*.txt

SOURCE_DIR='/path/to/directory/containing/source/files'
DEST_DIR='/path/to/target/directory'
EXTRA_GREP_OPTS=('-r')  # add -r to recurse
EXTRA_CP_OPTS=('-a')

GREP_COMMAND=("grep") I=1
for A in "${PATTERN_FILES[@]}"; do
	GREP_COMMAND[I++]="-f"
	GREP_COMMAND[I++]=$A
done

while read FILE; do
	cp "${EXTRA_CP_OPTS[@]}" "$FILE" "$DEST_DIR"
done < <(exec "${GREP_COMMAND[@]}" "${EXTRA_GREP_OPTS[@]}" -l "$SOURCE_DIR")

An advantage to this is that a buffer is not needed to be allocated in for ...; do ...; done before the strings inside the big string gets parsed with IFS.

NevemTeve · 04-15-2013, 08:25 AM

Code:

find . -maxdepth 1 -name 'name-mask' | \
xargs grep -e 'regexp-to-find' -l -- | \
xargs cp -t -- target

konsolebox · 04-15-2013, 08:27 AM

@NevemTeve The problem was actually about not having "Argument list too long."

Edit: Perhaps it could be a problem for grep I haven't/can't check. But even with cp, the number of file arguments that could be passed to it is sometimes limited in some systems.

Also I don't think that adds the solution for getting patterns from multiple files.

NevemTeve · 04-15-2013, 08:56 AM

You aren't being clear enough; did you get error message 'Argument list too long', or not? If you did get, you should use find + xargs. See my previous post for example.

konsolebox · 04-15-2013, 09:11 AM

Quote:

Originally Posted by NevemTeve

You aren't being clear enough; did you get error message 'Argument list too long',

No I did not, but that's the concern of the OP. Is the statement "Argument list too long" not enough to give the idea of the problem?

Do you know how a command after xargs gets executed?

And as one could obviously think, the error could generate depending on the system, and perhaps on the version of grep/cp, and perhaps on the number of files involved with it.

Quote:

If you did get, you should use find + xargs. See my previous post for example.

Yet what part of that makes use of grep -f which basically is what's needed? See the OP's post about grep -f *.txt. And using a glob pattern on -f won't work that's why you had to fix it with the script.

Or perhaps you just don't understand the concept of the code I posted?

NevemTeve · 04-15-2013, 09:39 AM

I'll admit I'm not really interested in the code you posted, I'm only trying to help to solve the problem of the OP.

'grep -l -f *.txt' is a very strange idea, should be replaced with one of these:

Code:

1. grep -l -f file-containing-patters.txt -- *.txt
2. grep -l -e 'pattern' -- *.txt

Now when there are too many files, it stops working, so you switch to find+xargs:

Code:

find . -maxdepth 1 -name '*.txt' | \
xargs grep -l -e 'patter'--

PS: okay, I read back and found out that the original question was asked in 2004. Sorry, I should have checked it earlier.

David the H. · 04-18-2013, 01:56 PM

The only things I have to say are:

1) That xargs should not be used without null separators between the input files. Otherwise files with whitespace in them will error out.

Code:

find <whatever> -print0 | xargs -0 <whatever>
grep -l -Z <whatever> | xargs -0 <whatever>

2) Don't Read Lines With For -- for pretty much the same reason, as well as possibly overrunning the ARG_MAX setting of your system. The user in post #14, which reopened the thread, actually grabbed one of the less-sane options here.

A while+read loop would be much safer, or a for loop with an array of files. But in any case the input files need to be processed in a way that handles whitespace safely.

How can I find and deal with file names containing newlines, spaces or both?
http://mywiki.wooledge.org/BashFAQ/020

How can I recursively search all files for a string?
http://mywiki.wooledge.org/BashFAQ/008