LinuxQuestions.org
Share your knowledge at the LQ Wiki.
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - Newbie
User Name
Password
Linux - Newbie This Linux forum is for members that are new to Linux.
Just starting out and have a question? If it is not in the man pages or the how-to's this is the place!

Notices


Reply
  Search this Thread
Old 07-11-2013, 07:42 AM   #1
akamel
Member
 
Registered: Dec 2012
Posts: 100

Rep: Reputation: 1
Why SCP time out?


I'm using the following command to transfer huge tar files about 15 GB
The session stopped after 10 hours or something

scp -c 3des filepathname root@destip:/destdir &

Please help ASAP
 
Old 07-11-2013, 07:49 AM   #2
szboardstretcher
Senior Member
 
Registered: Aug 2006
Location: Detroit, MI
Distribution: GNU/Linux systemd
Posts: 4,211

Rep: Reputation: 1612Reputation: 1612Reputation: 1612Reputation: 1612Reputation: 1612Reputation: 1612Reputation: 1612Reputation: 1612Reputation: 1612Reputation: 1612Reputation: 1612
While I can't tell you exactly why it stopped without more information,.. when it comes down to massive files like this, i always break them up into 1Gb chunks, so its more easily recoverable.
 
Old 07-11-2013, 07:51 AM   #3
akamel
Member
 
Registered: Dec 2012
Posts: 100

Original Poster
Rep: Reputation: 1
How?
 
Old 07-11-2013, 08:03 AM   #4
szboardstretcher
Senior Member
 
Registered: Aug 2006
Location: Detroit, MI
Distribution: GNU/Linux systemd
Posts: 4,211

Rep: Reputation: 1612Reputation: 1612Reputation: 1612Reputation: 1612Reputation: 1612Reputation: 1612Reputation: 1612Reputation: 1612Reputation: 1612Reputation: 1612Reputation: 1612
Well, by splitting the large file into chunks.

If you are unable to get to google at the moment to find a guide, or dont have 'man' installed, here is an example:

Code:
split --bytes=1G a_very_large.log chunks_of_large_log
And then cat them back together:

Code:
cat chunks_of_large_log > a_very_large.log
IYI: this is a proper usecase of cat btw, no uuoc award needed
 
1 members found this post helpful.
Old 07-11-2013, 11:16 AM   #5
suicidaleggroll
LQ Guru
 
Registered: Nov 2010
Location: Colorado
Distribution: OpenSUSE, CentOS
Posts: 5,562

Rep: Reputation: 2127Reputation: 2127Reputation: 2127Reputation: 2127Reputation: 2127Reputation: 2127Reputation: 2127Reputation: 2127Reputation: 2127Reputation: 2127Reputation: 2127
10 hours for 15GB? That's half a MB/s, are you trying to send this over the internet (rather than a local network)? If so, there's your problem. Internet connections can be spotty, hesitations and interruptions happen all the time, and will stop an SCP transfer dead in its tracks. You'll need to split up the file to something more manageable in order to reliably transfer it over. You can also try adding compression to speed up the transfer with the -C flag, assuming the data is somewhat compressible.

If, however, this is a transfer over your local network, then it's time to do some serious digging into your speed problems.

Last edited by suicidaleggroll; 07-11-2013 at 11:18 AM.
 
1 members found this post helpful.
Old 07-11-2013, 11:18 AM   #6
szboardstretcher
Senior Member
 
Registered: Aug 2006
Location: Detroit, MI
Distribution: GNU/Linux systemd
Posts: 4,211

Rep: Reputation: 1612Reputation: 1612Reputation: 1612Reputation: 1612Reputation: 1612Reputation: 1612Reputation: 1612Reputation: 1612Reputation: 1612Reputation: 1612Reputation: 1612
Quote:
Originally Posted by suicidaleggroll View Post
10 hours for 15GB? That's half a MB/s, are you trying to send this over the internet (rather than a local network)? If so, there's your problem. Internet connections can be spotty, hesitations and interruptions happen all the time, and will stop an SCP transfer dead in its tracks. You'll need to split up the file to something more manageable in order to reliably transfer it over.

If, however, this is a transfer over your local network, then it's time to do some serious digging into your speed problems.
True, thats about 4mbit right there.
 
Old 07-11-2013, 11:23 AM   #7
akamel
Member
 
Registered: Dec 2012
Posts: 100

Original Poster
Rep: Reputation: 1
Thanks for the replies!
I'm trying to copy the 15 GB through the Internet (2 tar files every one is about 7 GB)
 
Old 07-11-2013, 11:45 AM   #8
TB0ne
LQ Guru
 
Registered: Jul 2003
Location: Birmingham, Alabama
Distribution: SuSE, RedHat, Slack,CentOS
Posts: 19,330

Rep: Reputation: 4473Reputation: 4473Reputation: 4473Reputation: 4473Reputation: 4473Reputation: 4473Reputation: 4473Reputation: 4473Reputation: 4473Reputation: 4473Reputation: 4473
Quote:
Originally Posted by akamel
I'm using the following command to transfer huge tar files about 15 GB The session stopped after 10 hours or something

scp -c 3des filepathname root@destip:/destdir &

Please help ASAP
We volunteer our time here...asking for ASAP help is fairly rude, as has been explained to you before.
Quote:
Originally Posted by akamel View Post
Thanks for the replies!
I'm trying to copy the 15 GB through the Internet (2 tar files every one is about 7 GB)
Right...we understand what you're trying to do. Did you understand what you were told about splitting the files up? Uploading over the internet is ALWAYS going to be slower than downloading. What is your upload connection speed to the internet?
 
Old 07-11-2013, 04:15 PM   #9
akamel
Member
 
Registered: Dec 2012
Posts: 100

Original Poster
Rep: Reputation: 1
I really appreciate what you are doing guys!
How can I know the upload speed as I'm ssh to the server.
the status now is:
File 1 (7 GB) -- > Transferred 1.3 GB
File 2 (8 GB) -- > Transferred 1.1 GB

I hope this will be finished in the coming 48 hours without issues
 
Old 07-11-2013, 04:39 PM   #10
suicidaleggroll
LQ Guru
 
Registered: Nov 2010
Location: Colorado
Distribution: OpenSUSE, CentOS
Posts: 5,562

Rep: Reputation: 2127Reputation: 2127Reputation: 2127Reputation: 2127Reputation: 2127Reputation: 2127Reputation: 2127Reputation: 2127Reputation: 2127Reputation: 2127Reputation: 2127
Well first off, send one file at a time. It'll run twice as fast and have half the risk of stalling during the transfer.. Second, did you not read what was suggested above about splitting the file into multiple parts?
 
Old 07-11-2013, 04:44 PM   #11
akamel
Member
 
Registered: Dec 2012
Posts: 100

Original Poster
Rep: Reputation: 1
Yeah but it will take another time to compress. How I know the upload speed please?
 
Old 07-11-2013, 05:14 PM   #12
TB0ne
LQ Guru
 
Registered: Jul 2003
Location: Birmingham, Alabama
Distribution: SuSE, RedHat, Slack,CentOS
Posts: 19,330

Rep: Reputation: 4473Reputation: 4473Reputation: 4473Reputation: 4473Reputation: 4473Reputation: 4473Reputation: 4473Reputation: 4473Reputation: 4473Reputation: 4473Reputation: 4473
Quote:
Originally Posted by akamel
I really appreciate what you are doing guys!
How can I know the upload speed as I'm ssh to the server.
the status now is:
File 1 (7 GB) -- > Transferred 1.3 GB
File 2 (8 GB) -- > Transferred 1.1 GB
I hope this will be finished in the coming 48 hours without issues
Well, we've given you advice on splitting the file, but you seem unwilling to take it. Yes, it will take more time to compress it, so either you need to transfer more slowly and have it WORK, or transfer quicker, and maybe have it fail. Your choice.
Quote:
Originally Posted by akamel View Post
Yeah but it will take another time to compress. How I know the upload speed please?
You ask your ISP, or run one of the MANY easily-found Internet speed tests, like this:
http://www.speedtest.net/
 
Old 07-11-2013, 05:16 PM   #13
akamel
Member
 
Registered: Dec 2012
Posts: 100

Original Poster
Rep: Reputation: 1
Thanks I meant using CLI.
 
Old 07-11-2013, 05:28 PM   #14
TB0ne
LQ Guru
 
Registered: Jul 2003
Location: Birmingham, Alabama
Distribution: SuSE, RedHat, Slack,CentOS
Posts: 19,330

Rep: Reputation: 4473Reputation: 4473Reputation: 4473Reputation: 4473Reputation: 4473Reputation: 4473Reputation: 4473Reputation: 4473Reputation: 4473Reputation: 4473Reputation: 4473
Quote:
Originally Posted by akamel View Post
Thanks I meant using CLI.
There are several ways...have you bothered to try to look them up??? Putting "linux command line speed test" into Google will (AMAZINGLY) give you answers, like:
http://www.cyberciti.biz/faq/linux-u...-upload-speed/

This goes along with your other thread that you opened, where you said you were in a hurry, and didn't want to look it up yourself. Please, show some effort on your own.
 
  


Reply


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off



Similar Threads
Thread Thread Starter Forum Replies Last Post
Shell scp works perfectly, Net::SCP::Expect calls timing out - reasons? EnderX Programming 1 04-27-2013 02:35 AM
Centos 6.3 host+guest: try scp to guest: scp cmd not found. ssh ok chrism01 Linux - Virtualization and Cloud 3 08-24-2012 02:52 AM
[SOLVED] ssh scp key not working to ssh/scp without password rjo98 Linux - Newbie 9 08-22-2011 05:28 PM
how to understand user time, sys time, wait time, idle time of CPU guixingyi Linux - Server 1 08-24-2010 11:10 AM
scp does not work and gives the following error message: scp: FATAL: Executing ssh1 i akay Linux - Networking 16 09-29-2008 12:41 AM

LinuxQuestions.org > Forums > Linux Forums > Linux - Newbie

All times are GMT -5. The time now is 05:46 PM.

Main Menu
Advertisement
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
Facebook: linuxquestions Google+: linuxquestions
Open Source Consulting | Domain Registration