LinuxQuestions.org
Latest LQ Deal: Complete CCNA, CCNP & Red Hat Certification Training Bundle
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - Newbie
User Name
Password
Linux - Newbie This Linux forum is for members that are new to Linux.
Just starting out and have a question? If it is not in the man pages or the how-to's this is the place!

Notices


Reply
  Search this Thread
Old 04-23-2013, 10:55 AM   #1
rlich
LQ Newbie
 
Registered: Apr 2013
Posts: 6

Rep: Reputation: Disabled
Strange problem with CP and wildcards


I'm trying to copy files using wildcards, but can't get it to work:

cp -f /Src/FULL_\^RRRR\^_IP_2013-03* /Dest/

When I do the same wildcard with find, I see all the files I am after. The file names and the directory in which the files rest have spaces in them. Could this be the problem. Thanks.
 
Old 04-23-2013, 12:09 PM   #2
Diantre
Member
 
Registered: Jun 2011
Distribution: Slackware
Posts: 498

Rep: Reputation: 214Reputation: 214Reputation: 214
Quote:
Originally Posted by rlich View Post
The file names and the directory in which the files rest have spaces in them. Could this be the problem.
Certainly. Enclose all filename parameters to cp in single or double quotes and try again.
 
Old 04-23-2013, 12:24 PM   #3
shivaa
Senior Member
 
Registered: Jul 2012
Location: Grenoble, Fr.
Distribution: Sun Solaris, RHEL, Ubuntu, Debian 6.0
Posts: 1,800
Blog Entries: 4

Rep: Reputation: 286Reputation: 286Reputation: 286
Quote:
/Src/FULL_\^RRRR\^_IP_2013-03*
Problem is with this line. Either remove escape sequence or share name of any one of the file that you want to copy to Dest.
 
Old 04-23-2013, 12:40 PM   #4
David the H.
Bash Guru
 
Registered: Jun 2004
Location: Osaka, Japan
Distribution: Debian sid + kde 3.5 & 4.4
Posts: 6,823

Rep: Reputation: 1959Reputation: 1959Reputation: 1959Reputation: 1959Reputation: 1959Reputation: 1959Reputation: 1959Reputation: 1959Reputation: 1959Reputation: 1959Reputation: 1959
I'm confused. Could you show us exactly what the filenames look like? Also point out which ones you want to copy and which ones to exclude.

Spaces in filenames aren't a problem when using globbing like this, btw, since the filename expansion happens after the word-splitting step is complete. This is almost certainly due to incorrectly escaping the characters.

Last edited by David the H.; 04-23-2013 at 12:42 PM.
 
Old 04-24-2013, 06:29 AM   #5
rlich
LQ Newbie
 
Registered: Apr 2013
Posts: 6

Original Poster
Rep: Reputation: Disabled
Figured out the problem, but to answer the questions to my question, an actual file name example follows:


FULL_^RRRR^_IP_2013-03-01 someotherstuff.someextension

So, I figured that if I executed the following command that I would be able to copy all files with a March 2013 timestamp:

cp -f /Src/FULL_^RRRR^_IP_2013-03* /Dest/

Turns out that this works, but not with 6,000 plus files. I recall that I encountered this "too many files to copy" problem in the past. I suppose I could simply use a for loop with find to get what I want. It bothers me that there is a limit to the number of files I can copy at onces AND the fact that there is no error message indicating this. So, here is my attempt at the for loop, but it is having problems with spaces. Anyone know how to copy multiple files with for loop, taking into acount spaces in both the file name and at least one directory along the path. Nothing seems to be working.

Thanks.

Last edited by rlich; 04-24-2013 at 06:32 AM.
 
Old 04-24-2013, 06:55 AM   #6
jpollard
Senior Member
 
Registered: Dec 2012
Location: Washington DC area
Distribution: Fedora, CentOS, Slackware
Posts: 4,696

Rep: Reputation: 1261Reputation: 1261Reputation: 1261Reputation: 1261Reputation: 1261Reputation: 1261Reputation: 1261Reputation: 1261Reputation: 1261
Quote:
Originally Posted by rlich View Post
Figured out the problem, but to answer the questions to my question, an actual file name example follows:


FULL_^RRRR^_IP_2013-03-01 someotherstuff.someextension

So, I figured that if I executed the following command that I would be able to copy all files with a March 2013 timestamp:

cp -f /Src/FULL_^RRRR^_IP_2013-03* /Dest/

Turns out that this works, but not with 6,000 plus files. I recall that I encountered this "too many files to copy" problem in the past. I suppose I could simply use a for loop with find to get what I want. It bothers me that there is a limit to the number of files I can copy at onces AND the fact that there is no error message indicating this. So, here is my attempt at the for loop, but it is having problems with spaces. Anyone know how to copy multiple files with for loop, taking into acount spaces in both the file name and at least one directory along the path. Nothing seems to be working.

Thanks.
It is not directly a limit on cp, but a limit on the memory allowed for use in parameters to a process - hence the "too many files..." due to the bash parameter expansion. You can expand the number of files by shortening the name... Do a cd into /Src - and this will free up four bytes per file name...

The reason find works is that the search expression is evaluated by the find utility itself, and does not pass the entire list as a parameter list.

Use find instead of the for loop - it does the same thing, and is optimized for doing it. The disadvantage for both methods is that you have to spawn off a "cp" process for every file.

Last edited by jpollard; 04-24-2013 at 06:57 AM.
 
Old 04-24-2013, 07:46 AM   #7
rlich
LQ Newbie
 
Registered: Apr 2013
Posts: 6

Original Poster
Rep: Reputation: Disabled
jpollard - helpful advise, thanks. When you write to use find, how would I do that? Pipe the find results to cp? How would that look?

find / -name *** | cp something?

Thanks.
 
Old 04-24-2013, 08:21 AM   #8
rlich
LQ Newbie
 
Registered: Apr 2013
Posts: 6

Original Poster
Rep: Reputation: Disabled
Figured it out:

find . -name filename -exec cp {} Destination \;

It's great because it works with spaces in files!
 
Old 04-25-2013, 11:09 AM   #9
David the H.
Bash Guru
 
Registered: Jun 2004
Location: Osaka, Japan
Distribution: Debian sid + kde 3.5 & 4.4
Posts: 6,823

Rep: Reputation: 1959Reputation: 1959Reputation: 1959Reputation: 1959Reputation: 1959Reputation: 1959Reputation: 1959Reputation: 1959Reputation: 1959Reputation: 1959Reputation: 1959
Is that 6000 files just for March?

You could run a loop and processing it by each day individually.

Code:
for day in {01..31}; do

    cp -f "/Src/FULL_^RRRR^_IP_2013-03-$day"* /Dest/

done
Notice also how you can quote-protect the fixed parts of the string while leaving the globbing characters open for expansion. It's not really needed here, however, since there are no reserved characters in the string. But it never hurts.

You may have to get a bit fancier to avoid the unnecessary days on short months, however. Perhaps we should use an intermediate array instead.

Code:
shopt -s nullglob

for day in {01..31}; do

    array=( "/Src/FULL_^RRRR^_IP_2013-03-$day"* )

    [[ -n $array ]] && cp -f "${array[@]}" /Dest/

done
But that's getting to be a lot of work, and there's still no guarantee that you won't hit ARG_MAX, so I suppose find might still be best choice.

I recommend modifying it to this, however:

Code:
fileprefix='/Src/FULL_^RRRR^_IP_'
filedate='2013-03'
destination='/destdir'

find . -maxdepth 1 -name "$fileprefix$filedate*" -exec cp -t "$destination" '{}' +
The '+' ending for the -exec option enables xargs-style batch processing mode. It will run only as many instances of the command as necessary to avoid the ARG_MAX limit. The main limitation is that the '{}' brackets must then come at the end of the command (it expands to the full list of filenames), so for cp or mv you need to use the -t target directory option to move it to the front.

It's also recommended to avoid hard-coding data strings into your commands. Set variables at the top of your script (or pass them to it from outside), and use those in the actual commands.

Last edited by David the H.; 04-25-2013 at 11:11 AM.
 
Old 05-07-2013, 06:59 AM   #10
rlich
LQ Newbie
 
Registered: Apr 2013
Posts: 6

Original Poster
Rep: Reputation: Disabled
thanks for the follow-up information. Everything works great now.
 
  


Reply


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off



Similar Threads
Thread Thread Starter Forum Replies Last Post
problem with wildcards amr_salah944 Linux - Newbie 13 02-16-2012 03:38 AM
need help with wildcards liorpana Programming 2 05-12-2010 08:45 AM
using wildcards nadroj Linux - General 5 01-28-2007 08:39 PM
copy problem using wildcards plisken Linux - General 11 11-16-2005 08:34 AM
strange, strange alsa problem: sound is grainy/pixellated? fenderman11111 Linux - Software 1 11-01-2004 05:16 PM


All times are GMT -5. The time now is 07:58 PM.

Main Menu
Advertisement
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
Facebook: linuxquestions Google+: linuxquestions
Open Source Consulting | Domain Registration