LinuxQuestions.org
Visit the LQ Articles and Editorials section
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - Newbie
User Name
Password
Linux - Newbie This Linux forum is for members that are new to Linux.
Just starting out and have a question? If it is not in the man pages or the how-to's this is the place!

Notices

Reply
 
Search this Thread
Old 04-09-2008, 09:15 AM   #1
waterdragon900
LQ Newbie
 
Registered: Apr 2008
Posts: 5

Rep: Reputation: 0
Question Selectively moving files from a tree with a specific string anywhere in the path


Please forgive a forum newcomer if I have misposted or duplicated an existing thread - extensive searching could not find an answer to this problem.

I need to selectively move (not just copy) certain files from one Linux server to another. The selection criteria for the files to be moved is that they should have /Photos/ anywhere in the pathname. In other words, there are many directories all called "Photos" within the data tree on one server which need to be moved recursively to the other - i.e. including any files or trees within them and maintaining any preceding (higher) tree hierarchy. Before anyone suggests simply using the filename extension of .jpg or similar: I can not use this method because there are many other graphic files on the server which must not be moved!

I have tried adapting many ideas I found in forums and other help sites using various combinations of rsync, cp, scp and grep but nothing seems to work for me. I am particularly bemused by how to implement wildcards when handling pathnames. Never know whether to use quotes etc. And I freely confess that man pages frequently baffle me.

My level of knowledge of Linux is very basic - I can write simple files using vi and I have successfully installed and set up a simple data server running Samba on a purely internal LAN. That's about the limit of my knowledge - hence why I am posting to the Newbie forum.
 
Old 04-09-2008, 09:32 AM   #2
TB0ne
Guru
 
Registered: Jul 2003
Location: Birmingham, Alabama
Distribution: SuSE, RedHat, Slack,CentOS
Posts: 14,470

Rep: Reputation: 2538Reputation: 2538Reputation: 2538Reputation: 2538Reputation: 2538Reputation: 2538Reputation: 2538Reputation: 2538Reputation: 2538Reputation: 2538Reputation: 2538
Try using a find command with backticks (the ones at the top-left of the keyboard, with the tilde symbol). Something like this:

- Try the find command first, to make sure it returns what you're looking for, such as "find ./ -name Photos". See what it barfs back to the screen, and make sure you're getting what you want. If not, refine the command until you do.

- Use the backticks to use that command, in conjunction with another command, such as:

scp `find ./ -name Photos` username@10.11.12.13:/Photos/

Or you can make a tarball of them, and just SCP/FTP that later:

tar cvfz Photos.tgz `find ./ -name Photos`

Hope this helps.
 
Old 04-11-2008, 04:21 AM   #3
waterdragon900
LQ Newbie
 
Registered: Apr 2008
Posts: 5

Original Poster
Rep: Reputation: 0
Thumbs up Thanks but now have problem with white spaces

Many thanks to TB0ne for the helpful reply. That solution worked well and taught me about backticks which I never heard of before!

Now I have to work out how to deal with the whitespaces in the pathnames which unfortunately defeat this solution.

I found some small information on backticks and other things which may help other newbies here:-
http://geektimelinux.com/2003/12/05/OLD124/
and on filename handling here:-
http://www.linuxquestions.org/questi...e-path-634180/

but I am still not succeeding in handling the whitespaces.

Since the command strings are beginning to get rather long, I guess it's time I learned to write an executable script which is new territory for me.

I'll keep at it but if anyone has any ideas to offer, I would be very grateful!
 
Old 04-11-2008, 08:21 AM   #4
TB0ne
Guru
 
Registered: Jul 2003
Location: Birmingham, Alabama
Distribution: SuSE, RedHat, Slack,CentOS
Posts: 14,470

Rep: Reputation: 2538Reputation: 2538Reputation: 2538Reputation: 2538Reputation: 2538Reputation: 2538Reputation: 2538Reputation: 2538Reputation: 2538Reputation: 2538Reputation: 2538
Quote:
Originally Posted by waterdragon900 View Post
Many thanks to TB0ne for the helpful reply. That solution worked well and taught me about backticks which I never heard of before!

Now I have to work out how to deal with the whitespaces in the pathnames which unfortunately defeat this solution.
If it's just spaces you're concerned about, you can always just qualify the space as a character, with the \

Such as

find ./ -name "Photos\ "

Which will tell the find to look for Photos, with a space after it. If you have very convoluted paths, with numerous spaces, it'll get more complicated, and you might have to bang out a shell script to get it done.

You might also want to check out chcase here

http://www.primaledge.ca/chcase.html

Which will let you rename files recursively, in bulk. Once you remove the spaces, it might be easier to copy or TAR them up.

You could also try to use your GUI based file manager (Konqueror or the like), to search for them, then do a group select, and send them to a TAR file, spaces and all.....
 
Old 04-17-2008, 10:10 AM   #5
waterdragon900
LQ Newbie
 
Registered: Apr 2008
Posts: 5

Original Poster
Rep: Reputation: 0
Unhappy Still problems with white space handling

Thanks once again to TB0ne for the advice. chcase looks useful for future use but I am not allowed the luxury of ripping out the white spaces on the existing server as the users will have something to say about that!
The use of a GUI is also not an option since there are literally thousands of file trees to go through Ė each of which may contain a Photos directory. (Plus I havenít installed any GUI on the machine; it being a server.)

So to the problem as it remains now. I have substituted letters a, b, c, etc in place of the original characters in the file paths below but the white spaces are in the exact places that came from my session dump. This was done on a small sample of file trees copied into a safe testing area away from live data on the live server! I used sed as suggested here:- http://mindspill.net/computing/linux...-with-sed.html

Using find with sed to show where a directory called Photos is gives a result thus:-

myname@myserver:~$ echo `find /aaaa/bbbbbbbbbb/ccccccccccccccc -name Photos | sed 's/ /\\ /g'`
/aaaa/bbbbbbbbbb/ccccccccccccccc/ddddddddd/eeeee ffff ffff fff ffffff fff ffffff ffff/gg ggg/Photos

Trying to use this result for cp comes up with errors which I assume is because it canít handle the white spaces thus:-

myname@myserver:~$ cp -r `find /aaaa/bbbbbbbbb/cccccccccccccccc -name Photos | sed 's/ /\\ /g'` /aaaa/bbbbbbbbbb/dump/
cp: cannot stat `/aaaa/bbbbbbbbb/cccccccccccccccc/ddddddddd/eeeee': No such file or directory
cp: cannot stat `ffff': No such file or directory
cp: cannot stat `ffff': No such file or directory
cp: cannot stat `fff': No such file or directory
cp: cannot stat `ffffff': No such file or directory
cp: cannot stat `fff': No such file or directory
cp: cannot stat `ffffff': No such file or directory
cp: cannot stat `ffff/gg': No such file or directory
cp: cannot stat `ggg/Photos*': No such file or directory

I wondered if the output of the find needed to have escapes before each white space and so tried this:-

myname@myserver:~$ echo `find /aaaa/bbbbbbbbbb/ccccccccccccccc -name Photos | sed 's/ /\\\ /g'`
/aaaa/bbbbbbbbbb/ccccccccccccccc/ddddddddd/eeeee\ ffff\ ffff\ fff\ ffffff\ fff\ ffffff\ ffff/gg\ ggg/Photos

So far so good but still it seems that cp canít cope:-

myname@myserver:~$ cp -r `find /aaaa/bbbbbbbbb/cccccccccccccccc -name Photos | sed 's/ /\\\ /g'` /aaaa/bbbbbbbbbb/dump/
cp: cannot stat `/aaaa/bbbbbbbbb/cccccccccccccccc/ddddddddd/eeeee\\': No such file or directory
cp: cannot stat `ffff\\': No such file or directory
cp: cannot stat `ffff\\': No such file or directory
cp: cannot stat `fff\\': No such file or directory
cp: cannot stat `ffffff\\': No such file or directory
cp: cannot stat `fff\\': No such file or directory
cp: cannot stat `ffffff\\': No such file or directory
cp: cannot stat `ffff/gg\\': No such file or directory
cp: cannot stat `ggg/Photos*': No such file or directory

yet if I try to copy the files by manually typing in the path, it works OK thus:-

myname@myserver:~$ cp -r /aaaa/bbbbbbbbb/cccccccccccccccc/ddddddddd/eeeee\ ffff\ ffff\ fff\ ffffff\ fff\ ffffff\ ffff/gg\ ggg/Photos/ /aaaa/bbbbbbbbbb/dump/

Clearly I still havenít cracked this problem and so canít progress to writing a script to work on the real data. Anyone any ideas what Iím doing wrong? Or is there a completely different approach which might work?
 
Old 08-11-2008, 03:22 PM   #6
gd2shoe
Member
 
Registered: Jun 2004
Location: Northern CA
Distribution: Debian
Posts: 835

Rep: Reputation: 49
I hope I am not too late to help. It sounds like you need to tweak the IFS (Internal Field Separator) environment variable. It contains characters which are interpreted as separating items on the command line. something like:
Quote:
ifs=$IFS
IFS="\n"
yourCommand
IFS=$ifs
I believe you can also do something such as:
Quote:
IFS="\n" yourCommand
Hope this helps.
 
Old 08-11-2008, 07:04 PM   #7
chrism01
Guru
 
Registered: Aug 2004
Location: Sydney
Distribution: Centos 6.5, Centos 5.10
Posts: 16,261

Rep: Reputation: 2028Reputation: 2028Reputation: 2028Reputation: 2028Reputation: 2028Reputation: 2028Reputation: 2028Reputation: 2028Reputation: 2028Reputation: 2028Reputation: 2028
To be honest, I'd use Perl for this.
You'd have to learn it obviously, but if this is the sort of thing you are likely to be dong again ie non-trivial work, it'll pay off big time.
Perl has built-in cmds to traverse dirs and read file-by-file at the system level ie spaces in the dir names or file names are no problem. (but simpler to learn than C)
It will also do all the moving for you across the LAN.
It'd be dead easy to tell it to produce a list that you can check before enabling the move cmd.

Perl docs: http://perldoc.perl.org/ includes tutorials, or ask here/prog forum.
 
Old 08-18-2008, 09:21 AM   #8
waterdragon900
LQ Newbie
 
Registered: Apr 2008
Posts: 5

Original Poster
Rep: Reputation: 0
Just returned from leave to find posts by gd2shoe and chrism01. My sincere thanks to both.
To be honest, I have had to put this matter to one side as I was making no headway. Hopefully these new ideas will help when I get back on to this problem which still requires a solution. Could be some time before I do this due to pressure of other urgent matters....
I'll post here again if I get anywhere with this.
 
Old 08-18-2008, 09:57 AM   #9
BedriddenTech
Member
 
Registered: Aug 2003
Distribution: Slackware 9.1, 10; FreeBSD 5.2.1; Fedora Core 2; SuSE 9.1
Posts: 44

Rep: Reputation: 15
I guess you are completely missing find's `-exec' option, which will handle spaces without problems. You can use it like this:
Code:
find . -name \*Photos\* -exec scp '{}' user@host:/Photos \;
Remember the single quotes around the curly braces (which find replaces by the name of the file) and the escaped semi-colon.

Have a look at find's manpage to tune this command; for example using `-type d' to select only directories. (Give `scp' the `-r' option, then.)

The only drawback is that scp will propably ask you for your password for every directory. There are two solutions to this:
  1. Use ssh-agent
  2. Use a shell script to construct a quoted list for scp
 
  


Reply

Tags
files, find, moving, quoting, scp, selective, ssh


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off


Similar Threads
Thread Thread Starter Forum Replies Last Post
output the path for files whose names include string "string" (case insensitive) sean_zhang Linux - Newbie 1 03-04-2008 11:59 PM
AT91RM9200 specific directory in kernel source tree kondrara Linux - Embedded & Single-board computer 2 02-20-2008 05:46 PM
make only specific module is kernel source tree saranjothy Linux - Kernel 2 08-16-2006 10:05 AM
How to find those files having specific string??? saeed Red Hat 4 07-19-2006 11:50 AM
Moving root directory tree to new hardrive paul.nel Linux - General 3 01-22-2004 12:28 AM


All times are GMT -5. The time now is 11:20 PM.

Main Menu
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
identi.ca: @linuxquestions
Facebook: linuxquestions Google+: linuxquestions
Open Source Consulting | Domain Registration