LinuxQuestions.org
Share your knowledge at the LQ Wiki.
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - General
User Name
Password
Linux - General This Linux forum is for general Linux questions and discussion.
If it is Linux Related and doesn't seem to fit in any other forum then this is the place.

Notices


Reply
  Search this Thread
Old 07-28-2008, 04:55 PM   #1
WingnutOne
Member
 
Registered: Sep 2007
Location: Kansas City
Distribution: Mixed, mostly RH / Fedora
Posts: 76

Rep: Reputation: 16
Question recursive scp w/o following links?


What's the best alternative to scp if you want to recursively copy a directory from one machine to another but NOT follow links?
Using rsync or cpio over an nfs mount is possible, but extremely slow.

Fedora 8-9

thanks!

Last edited by WingnutOne; 07-28-2008 at 05:15 PM. Reason: clarity
 
Old 07-28-2008, 05:01 PM   #2
trickykid
LQ Guru
 
Registered: Jan 2001
Posts: 24,149

Rep: Reputation: 235Reputation: 235Reputation: 235
If you're going over an NFS mount, why are you even using scp or rsync? You should be able to treat it like a local filesystem and just use cp or mv, etc, then eliminate the copying of links.
 
Old 07-28-2008, 05:42 PM   #3
WingnutOne
Member
 
Registered: Sep 2007
Location: Kansas City
Distribution: Mixed, mostly RH / Fedora
Posts: 76

Original Poster
Rep: Reputation: 16
Quote:
Originally Posted by trickykid View Post
If you're going over an NFS mount, why are you even using scp or rsync? You should be able to treat it like a local filesystem and just use cp or mv, etc, then eliminate the copying of links.
For this purpose, mv and cp work about the same as rsync or cpio. (They both use the NFS mount too.)
Copying lots of small files over an NFS mount slows your progress down to a crawl. I don't know how or why, but it's like there's a set minimum amount of time required to copy any given file, no matter how small. For instance: a single 1MB file might only take a few seconds to copy, but 1000 10-byte files (~10KB total) might take several minutes. I don't know why it happens, but I've seen it numerous times on several systems. (If anyone does know why, I'd like to hear about it.)

wn

Last edited by WingnutOne; 07-28-2008 at 05:43 PM. Reason: clearer
 
Old 07-28-2008, 10:35 PM   #4
frob23
Senior Member
 
Registered: Jan 2004
Location: Roughly 29.467N / 81.206W
Distribution: OpenBSD, Ubuntu, FreeBSD
Posts: 1,449

Rep: Reputation: 48
This is the way I would do it. Start on the machine you want the files to end up on.

Code:
cd /destination/directory
ssh user@remote.host "cd /original/directory; tar cf - ./" | tar xvf -
Since tar doesn't follow symbolic links by default, this should work the way you want. You may add other flags as you want... if both machines are fast and the link is slow... compression would be good (either pipe it through bzip2 or use the flags in tar)... I would discourage -p (preserve permissions) unless the user IDs are the same on both machines... if not, and you want to keep permissions, remember to change the owners and groups after moving the files.

Note: you can replace tar with cpio if that is your preferred method... using the same technique.

Edit: You can make this as complex as you want, using find/cpio/pax/bzip2 and pipes... you can actually manage some very complex selections using ssh like this.

Last edited by frob23; 07-28-2008 at 10:39 PM.
 
Old 07-29-2008, 01:49 AM   #5
trickykid
LQ Guru
 
Registered: Jan 2001
Posts: 24,149

Rep: Reputation: 235Reputation: 235Reputation: 235
Yeah, I'd agree with frob23, create one big file to copy. Smaller files eat up resources when moving, copying, deleting, etc. You have to realize the drive has to read more blocks to get each inode info about each file, etc. A bunch of small files is a backup administrators worst nightmare. Take for example a 2TB database dump, I could backup the whole thing in a couple of hours. The same amount of data as small misc files could take a day or longer. Don't underestimate them cause they're small, bigger files are easier to deal with when it comes to filesystems.
 
Old 07-29-2008, 10:59 AM   #6
WingnutOne
Member
 
Registered: Sep 2007
Location: Kansas City
Distribution: Mixed, mostly RH / Fedora
Posts: 76

Original Poster
Rep: Reputation: 16
Thanks for the info. I'm using the tar idea (with -z option) and it's much faster than messing with all the individual files.

This has been reduced to a matter of curiosity now, but I'd still like to learn why moving small files through an NFS mount takes so much longer than transferring the same files via ssh.

Thanks again,

wn
 
Old 09-09-2008, 03:23 AM   #7
julbra
LQ Newbie
 
Registered: Sep 2008
Posts: 1

Rep: Reputation: 0
Thumbs up

Quote:
Originally Posted by frob23 View Post
Code:
cd /destination/directory
ssh user@remote.host "cd /original/directory; tar cf - ./" | tar xvf -
Genius!

Never thought of using tar and ssh in that combination.
 
  


Reply


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off



Similar Threads
Thread Thread Starter Forum Replies Last Post
scp does not work and gives the following error message: scp: FATAL: Executing ssh1 i akay Linux - Networking 16 09-29-2008 12:41 AM
Symbolic Links in Net::FTP::Recursive innuendo_98 Programming 0 07-07-2006 11:08 AM
links (hard links and soft links..) sachitha Programming 1 08-10-2005 01:10 PM
SCP without soft links henriette Linux - Networking 1 08-27-2004 03:31 PM
Links Section updated - Please add your Linux links. jeremy Linux - General 2 11-24-2001 12:35 PM

LinuxQuestions.org > Forums > Linux Forums > Linux - General

All times are GMT -5. The time now is 05:10 PM.

Main Menu
Advertisement
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
Facebook: linuxquestions Google+: linuxquestions
Open Source Consulting | Domain Registration