Copying Large Files on the Network

bichonfrise74 · 02-19-2009, 02:34 PM

Hi,

I want to copy a large file from Server A to Server B using gzip because I heard that it is faster than scp.

I have this command, but it does not seem to be working... The effect of this is that it just copies the files from Server A to Server B without ever ending.

Can someone take a look?

gzip -c <file> | ssh host "gunzip -c - > /home/myname/<file>"

Thanks.

mjlouis · 02-19-2009, 03:27 PM

Quote:

Originally Posted by bichonfrise74

Hi,

I want to copy a large file from Server A to Server B using gzip because I heard that it is faster than scp.

I have this command, but it does not seem to be working... The effect of this is that it just copies the files from Server A to Server B without ever ending.

Can someone take a look?

gzip -c <file> | ssh host "gunzip -c - > /home/myname/<file>"

Thanks.

Depends what type of file it is. If it's something already compressed like an mpeg video then the gzip is not worth the effort - just scp it. In fact, unless we're talking about a slow network or an obscenely huge file, you might as well just scp it for a single file.
On the other hand, if it's loads of small files, say adding up to 1GB then yell and I'll re-look at your string.

theYinYeti · 02-19-2009, 03:40 PM

Here's something I often do at work where network is very slow:

Code:

cd /local/prefix; tar -czf - SOURCE | ssh remoteuser@remotehost 'cd /remote/prefix; tar -xpzf -'

or simply I use rsync.

Yves.

billymayday · 02-19-2009, 03:44 PM

Why not just use scp -C ?

theYinYeti · 02-19-2009, 03:53 PM

That I use too. Works well.

bichonfrise74 · 02-20-2009, 11:49 AM

Hi,

I've used tar in combination with ssh and it works... the one that I'm trying to figure out is to use gzip to work with ssh.

For some reason, if I try to run gzip with ssh, it does not work. I'm pretty sure it is just a typo but I can't figure it out.

Also, I've used scp and it works fine... but I heard rumors that gzip with ssh is faster than scp so I want to test it out.

mjlouis · 02-20-2009, 12:00 PM

Quote:

Originally Posted by bichonfrise74

Hi,

I've used tar in combination with ssh and it works... the one that I'm trying to figure out is to use gzip to work with ssh.

For some reason, if I try to run gzip with ssh, it does not work. I'm pretty sure it is just a typo but I can't figure it out.

Also, I've used scp and it works fine... but I heard rumors that gzip with ssh is faster than scp so I want to test it out.

That's a fair enough answer!
I suggest taking a look at the contents of /home/myname/<file> as the file might contain error messages that tell you what's wrong. Alternatively, just don't redirect the output and it should chuck its error right on the terminal for you to see.

bichonfrise74 · 02-20-2009, 12:48 PM

Thanks...

The file was corrupt!!! When I tried the process on a different file, it went okay. Also, I found out that gzip with ssh combination is slower as compared to a simple scp.