bash, cp and filename (with spaces) woes

dfidler · 06-09-2006, 10:14 AM

Hey All,

I am trying to get a script to move files from one dir to another when they are older than 180 days. What I have so far:

Quote:

#Modify this to change the destination of the various components
BASE_DIR="/sumfiles
TARGET_DIR="/backup-sumfiles
AGE=180

echo "Moving all files older than $AGE days from $BASE_DIR -> $TARGET_DIR"
echo
# Make sure that our target directory exists
mkdir -p $TARGET_DIR

# Move the files
for i in `find $BASE_DIR -type f -mtime +$AGE`; do
cp "$i" $TARGET_DIR/ --parents -p
if (( $? != 0 )); then
echo "FAILED TO COPY $i; BAILING OUT"
exit 1;
fi
rm "$i"
echo "Removed: \"$i\""
done

This works great, right up until I hit a filename that contains spaces. Just quoting my $i doesn't seem to work (now, why don't all mtools have a -0 option? for the love of god!) I would *LOVE* to be able to just move the files, but I need to maintain the directory structure under which the files reside (hence --parents).

Anyone have any suggestions on how I can make it that last foot-and-a-half?

I am sure that I could use something like:

Quote:

find $BASE_DIR -type f -mtime $AGE -print0 | xargs -0 -l1 cp --parents -p "{}" $TARGET_DIR/ && rm {} \;

But I couldn't figure out how to get xargs to like the &&.

Any suggestions? I'd prefer to use the for loop from the first method as it is cleaner, easier to read and way easier to comment for the next guy that comes along.

note: maintaining the file attribs and directory structure are the key requirements here.

Dave.

MensaWater · 06-09-2006, 10:50 AM

This is how I did something similar for Oracle binary files as they contain spaces in their names:

Code:

#!/bin/ksh
# Simple script to change owner of files from original owner to new owner.
# jlightner 23-Aug-2005
#
# Usage:  chownit olduid newuid
#
find / -user $1 -fsonly vxfs |awk '{print "\""$0"\""}'|xargs chown -h $2

What I found was I had to escape the quotes with the \ but then quote the escape characters themselves. $0 in awk means "the entire line" so what the abvoe does is send the entire line (file name) with a quote at the beginning and a quote at the end to xargs. All the other quotes in that print statement are just so it will use the escape as an escape rather than a litteral.

(REGEXP is such fun!).

P.S. Rant about Oracle: What kinds of morons use spaces in their file names AND name a directory "core"? grrrr

dfidler · 06-09-2006, 11:10 AM

Quote:

Originally Posted by jlightner

This is how I did something similar for Oracle binary files as they contain spaces in their names:

Code:

#!/bin/ksh
# Simple script to change owner of files from original owner to new owner.
# jlightner 23-Aug-2005
#
# Usage:  chownit olduid newuid
#
find / -user $1 -fsonly vxfs |awk '{print "\""$0"\""}'|xargs chown -h $2

What I found was I had to escape the quotes with the \ but then quote the escape characters themselves. $0 in awk means "the entire line" so what the abvoe does is send the entire line (file name) with a quote at the beginning and a quote at the end to xargs. All the other quotes in that print statement are just so it will use the escape as an escape rather than a litteral.

(REGEXP is such fun!).

P.S. Rant about Oracle: What kinds of morons use spaces in their file names AND name a directory "core"? grrrr

re: Oracle rant; LOL

As for your code, you prob could have done:

Code:

find / -user $1 -fsonly vxfs -print0 | xargs -0 chown -h $2

-print0 delimits the return vals from find with the null char while -0 tells xargs to expect null delimited strings

As far as your awking goes, I will give it a shot. I tried using sed/tr to replace the spaces in my strings with "\ " before but that required the use of echo; perhaps this will meet with more success...

Thanks.

jschiwal · 06-09-2006, 11:12 AM

The problem is takes place before the copy command. When find returns a file name with a space, it is in the top of the for loop where it is read as two separate arguments.

You can temporarily change the value of the the IFS variable so that this doesn't happen.
[code]
ifs=$IFS
IFS='\
'
for file in $(find $BASE_DIR -type f -mtime +$AGE); do
cp -p --parent "$file" $BACKUP_DIR && rm "$file" || echo failed to backup "$file"
done
IFS=ifs

It doesn't make sense to me to abort the backup because a single copy failed.
You may still have a problem with certain evil characters, such as "!" which have a special meaning to the shell, and are evaluated in double quotes.

MensaWater · 06-09-2006, 12:02 PM

The -0 option isn't available in HP-UX xargs which is where I came up with this script originally. Thanks for the tip though - It may come in handy if I need to do something similar in Linux (as is likely - we're looking at migrating our Oracle Apps tier to Linux from HP-UX).

dfidler · 06-09-2006, 12:06 PM

Quote:

Originally Posted by jlightner

The -0 option isn't available in HP-UX xargs which is where I came up with this script originally. Thanks for the tip though - It may come in handy if I need to do something similar in Linux (as is likely - we're looking at migrating our Oracle Apps tier to Linux from HP-UX).

Ah, my bad

dfidler · 06-09-2006, 12:15 PM

Quote:

Originally Posted by jschiwal

The problem is takes place before the copy command. When find returns a file name with a space, it is in the top of the for loop where it is read as two separate arguments.

You can temporarily change the value of the the IFS variable so that this doesn't happen.
[code]
ifs=$IFS
IFS='\
'
for file in $(find $BASE_DIR -type f -mtime +$AGE); do
cp -p --parent "$file" $BACKUP_DIR && rm "$file" || echo failed to backup "$file"
done
IFS=ifs

So, if this works, do you prefer prayer or hymns?

Quote:

Originally Posted by jschiwal

It doesn't make sense to me to abort the backup because a single copy failed.

Yeah, except that I am writing this knowing that people that know nothing about the script will be running it. It is not critical if we miss a file, but it could be a bad thing under certain circumstances; "side-effects = bad". I'd rather it puke and have the know-nots call the know-whats because they can't proceed. Nothing ticks me off more than asking someone, "So, after the script ran, did you read the log file for failures?" The response is always, "Well, no... the script completed just fine". *sigh*

Quote:

Originally Posted by jschiwal

You may still have a problem with certain evil characters, such as "!" which have a special meaning to the shell, and are evaluated in double quotes.

Thanks, that is a good warning; I'll do some testing for the different use cases.

Cheers,
Dave.

d1663m · 04-10-2007, 02:53 PM

Props to this post, this is a very elegant solution to weird characters in filenames:
http://www.macgeekery.com/tips/cli/h...spaces_in_bash

Code:

find . -type f -print | while read i; do touch "${i}"; done

'for' splits on spaces. Period. Regardless of quoting.
'read' does not.

MensaWater · 04-10-2007, 03:01 PM

Of course although I put mine in a script to be used later and commented it and put in the shell interpreter my original solution really was one line also.

I'll admit it had a few more characters than the above line though.

On the plus side the OP didn't have to wait nearly a year for my answer.

cfaj · 04-10-2007, 11:12 PM

Quote:

Originally Posted by d1663m

Props to this post, this is a very elegant solution to weird characters in filenames:
http://www.macgeekery.com/tips/cli/h...spaces_in_bash

Code:

find . -type f -print | while read i; do touch "${i}"; done

That will fail if any filenames have leading or trailing spaces or end in a backslash. (Not to mention filenames containing newlines.)

Quote:

'for' splits on spaces. Period. Regardless of quoting.

It does not split on spaces if there is no space in the value of $IFS.

Quote:

'read' does not.

It does if more than one variable is given as an argument (and IFS contains a space).