LinuxQuestions.org
View the Most Wanted LQ Wiki articles.
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - Newbie
User Name
Password
Linux - Newbie This Linux forum is for members that are new to Linux.
Just starting out and have a question? If it is not in the man pages or the how-to's this is the place!

Notices

Reply
 
Search this Thread
Old 02-19-2009, 02:34 PM   #1
bichonfrise74
LQ Newbie
 
Registered: Jul 2008
Posts: 24

Rep: Reputation: 15
Copying Large Files on the Network


Hi,

I want to copy a large file from Server A to Server B using gzip because I heard that it is faster than scp.

I have this command, but it does not seem to be working... The effect of this is that it just copies the files from Server A to Server B without ever ending.

Can someone take a look?

gzip -c <file> | ssh host "gunzip -c - > /home/myname/<file>"

Thanks.
 
Old 02-19-2009, 03:27 PM   #2
mjlouis
Member
 
Registered: Nov 2008
Posts: 48

Rep: Reputation: 16
Quote:
Originally Posted by bichonfrise74 View Post
Hi,

I want to copy a large file from Server A to Server B using gzip because I heard that it is faster than scp.

I have this command, but it does not seem to be working... The effect of this is that it just copies the files from Server A to Server B without ever ending.

Can someone take a look?

gzip -c <file> | ssh host "gunzip -c - > /home/myname/<file>"

Thanks.
Depends what type of file it is. If it's something already compressed like an mpeg video then the gzip is not worth the effort - just scp it. In fact, unless we're talking about a slow network or an obscenely huge file, you might as well just scp it for a single file.
On the other hand, if it's loads of small files, say adding up to 1GB then yell and I'll re-look at your string.
 
Old 02-19-2009, 03:40 PM   #3
theYinYeti
Senior Member
 
Registered: Jul 2004
Location: France
Distribution: Arch Linux
Posts: 1,897

Rep: Reputation: 61
Here's something I often do at work where network is very slow:
Code:
cd /local/prefix; tar -czf - SOURCE | ssh remoteuser@remotehost 'cd /remote/prefix; tar -xpzf -'
or simply I use rsync.

Yves.
 
Old 02-19-2009, 03:44 PM   #4
billymayday
Guru
 
Registered: Mar 2006
Location: Sydney, Australia
Distribution: Fedora, CentOS, OpenSuse, Slack, Gentoo, Debian, Arch, PCBSD
Posts: 6,678

Rep: Reputation: 122Reputation: 122
Why not just use scp -C ?
 
Old 02-19-2009, 03:53 PM   #5
theYinYeti
Senior Member
 
Registered: Jul 2004
Location: France
Distribution: Arch Linux
Posts: 1,897

Rep: Reputation: 61
That I use too. Works well.
 
Old 02-20-2009, 11:49 AM   #6
bichonfrise74
LQ Newbie
 
Registered: Jul 2008
Posts: 24

Original Poster
Rep: Reputation: 15
Hi,

I've used tar in combination with ssh and it works... the one that I'm trying to figure out is to use gzip to work with ssh.

For some reason, if I try to run gzip with ssh, it does not work. I'm pretty sure it is just a typo but I can't figure it out.

Also, I've used scp and it works fine... but I heard rumors that gzip with ssh is faster than scp so I want to test it out.
 
Old 02-20-2009, 12:00 PM   #7
mjlouis
Member
 
Registered: Nov 2008
Posts: 48

Rep: Reputation: 16
Quote:
Originally Posted by bichonfrise74 View Post
Hi,

I've used tar in combination with ssh and it works... the one that I'm trying to figure out is to use gzip to work with ssh.

For some reason, if I try to run gzip with ssh, it does not work. I'm pretty sure it is just a typo but I can't figure it out.

Also, I've used scp and it works fine... but I heard rumors that gzip with ssh is faster than scp so I want to test it out.
That's a fair enough answer!
I suggest taking a look at the contents of /home/myname/<file> as the file might contain error messages that tell you what's wrong. Alternatively, just don't redirect the output and it should chuck its error right on the terminal for you to see.
 
Old 02-20-2009, 12:48 PM   #8
bichonfrise74
LQ Newbie
 
Registered: Jul 2008
Posts: 24

Original Poster
Rep: Reputation: 15
Thanks...

The file was corrupt!!! When I tried the process on a different file, it went okay. Also, I found out that gzip with ssh combination is slower as compared to a simple scp.
 
  


Reply


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off


Similar Threads
Thread Thread Starter Forum Replies Last Post
using mmap for copying large files rohanak Programming 4 05-18-2008 06:28 AM
Copying large files to FAT32 sketch7 Linux - General 11 10-22-2007 10:13 AM
NFS Trouble on copying large files (400Mb+) Yalla-One Slackware 2 03-17-2007 08:29 AM
Samba hangs up at copying many large files ogcu Linux - Networking 1 05-19-2004 01:29 AM
Error copying large files from cdrom patrickyeo Linux - Hardware 1 11-03-2002 06:46 PM


All times are GMT -5. The time now is 03:26 AM.

Main Menu
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
identi.ca: @linuxquestions
Facebook: linuxquestions Google+: linuxquestions
Open Source Consulting | Domain Registration