[SOLVED] Does this command seem safe?

damgar · 09-02-2012, 04:04 PM

I have a 200GB directory I want to copy to a new machine over my LAN. It's a slow process. I found this command:

Code:

ssh user@source.server ‘tar -cz -C /source/path/ *’ | tar -zxv

that is supposed to do the copy as a single stream rather than one file at a time. My question since I'm not a tar expert is does te command look safe? It is meant to be run from the destination directory.

turtleli · 09-02-2012, 04:48 PM

I don't really know, but have you tried:

Code:

scp -C -r user@source.server:/remote/path /local/path

The -C means compression. I use this command to copy large directories from the HPC I have access to.

EDIT: should probably say this still transfers files one by one.

GazL · 09-02-2012, 05:12 PM

I'd be wary of the wildcard, especially so given the use of -C on the tar as it will probably be expanded in the home directory of user before the tar changes directory.

Using '.' will be less prone to errors and will also ensure that you don't miss any hidden files which the wildcard can sometimes miss.

e.g.

Code:

ssh user@source.server ‘tar -C /source/path -cz .' | tar -C /dest/path -xzv

damgar · 09-02-2012, 06:53 PM

I tested the command I posted on a small directory that has one single subdirectory (that has several more layers of subdirectories as well) and bunch of individual files. All of the individual files made the copy, but the subdirectory and it's contents did not. It did copy 2 files very fast though! lol Looking online it looks like tar should automatically be recursive and keep the subdirectory structures? I'm not sure exactly what went wrong here.

I went ahead with rsync + ssh, and 2.5 hours into the copy I can say two things for sure: I should have picked up that gigabit switch that I forgot about yesterday, and I should maybe have researched this a little further in advance!

damgar · 09-02-2012, 06:55 PM

Quote:

Originally Posted by turtleli

I don't really know, but have you tried:

Code:

scp -C -r user@source.server:/remote/path /local/path

The -C means compression. I use this command to copy large directories from the HPC I have access to.

EDIT: should probably say this still transfers files one by one.

I tried various incantations of scp but they all bogged down with the first 5GB file. That's kind of the problem with this particular directory. It's huge, and it has a large number of very large files combine with thousands of very small files. When you hit one of the large ones things just get SLOW! LOL

saulgoode · 09-02-2012, 08:06 PM

Quote:

Originally Posted by damgar

Code:

ssh user@source.server ‘tar -cz -C /source/path/ *’ | tar -zxv

I recall having problems with this once when using it to transfer a WINE directory which contained some circular links (the target just grew and grew until it ran out of space).

Other than that, it would seem pretty safe; perhaps add the -j option to make sure you don't archive non-local directories.

lopid · 09-05-2012, 06:04 PM

Code:

rsync -avP -e 'ssh -o ControlPersist=yes' user@source.server:/source/path .

Since it's a LAN, I wouldn't bother with compression.