LinuxQuestions.org
Welcome to the most active Linux Forum on the web.
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - Software
User Name
Password
Linux - Software This forum is for Software issues.
Having a problem installing a new program? Want to know which application is best for the job? Post your question in this forum.

Notices


Reply
  Search this Thread
Old 11-04-2015, 12:37 PM   #1
ilesterg
Member
 
Registered: Jul 2012
Location: München
Distribution: Debian, CentOS/RHEL
Posts: 583

Rep: Reputation: 72
ssh/sftp/scp file transfer of huge files


Hi everyone,

Has anyone tried using ssh/sftp/scp to transfer files as huge as 2Gig without encountering any problems on a daily basis? Could these handle files these big?

Thanks.
 
Old 11-04-2015, 01:02 PM   #2
HMW
Member
 
Registered: Aug 2013
Location: Sweden
Distribution: Debian, Arch, Red Hat, CentOS
Posts: 773
Blog Entries: 3

Rep: Reputation: 369Reputation: 369Reputation: 369Reputation: 369
Quote:
Originally Posted by ilesterg View Post
Hi everyone,

Has anyone tried using ssh/sftp/scp to transfer files as huge as 2Gig without encountering any problems on a daily basis? Could these handle files these big?
Short answer: Yes. I have. No problem whatsoever.

Best regards,
HMW
 
Old 11-04-2015, 01:41 PM   #3
ilesterg
Member
 
Registered: Jul 2012
Location: München
Distribution: Debian, CentOS/RHEL
Posts: 583

Original Poster
Rep: Reputation: 72
Quote:
Originally Posted by HMW View Post
Short answer: Yes. I have. No problem whatsoever.

Best regards,
HMW
Hmmm.that's a quick one. I still wonder why the old folks in my client made this custom app just to transfer files. Ok then, thanks.
 
Old 11-04-2015, 01:48 PM   #4
schneidz
LQ Guru
 
Registered: May 2005
Location: boston, usa
Distribution: fc-15/ fc-20-live-usb/ aix
Posts: 5,272

Rep: Reputation: 914Reputation: 914Reputation: 914Reputation: 914Reputation: 914Reputation: 914Reputation: 914Reputation: 914
Quote:
Originally Posted by ilesterg View Post
Hi everyone,

Has anyone tried using ssh/sftp/scp to transfer files as huge as 2Gig without encountering any problems on a daily basis? Could these handle files these big?

Thanks.
i use sshfs sometimes to do dvd rips. sometimes i create dvd.iso backups created with dd and scp the resultant image to various devices on my network. these files average 10 gb. other than waiting several minutes there is no problem.
Quote:
Originally Posted by ilesterg View Post
Hmmm.that's a quick one. I still wonder why the old folks in my client made this custom app just to transfer files. Ok then, thanks.
status quo ?

Last edited by schneidz; 11-04-2015 at 01:50 PM.
 
Old 11-05-2015, 02:47 AM   #5
chrism01
LQ Guru
 
Registered: Aug 2004
Location: Sydney
Distribution: Centos 6.10, Centos 7.5
Posts: 17,700

Rep: Reputation: 2494Reputation: 2494Reputation: 2494Reputation: 2494Reputation: 2494Reputation: 2494Reputation: 2494Reputation: 2494Reputation: 2494Reputation: 2494Reputation: 2494
Quote:
Short answer: Yes. I have. No problem whatsoever.
me too
 
Old 11-05-2015, 03:23 AM   #6
Emerson
LQ Guru
 
Registered: Nov 2004
Location: Saint Amant, Acadiana
Distribution: Gentoo ~arch
Posts: 6,317

Rep: Reputation: Disabled
I use NFS mounts on LAN and rsync with --bwlimit=xxxx to avoid saturation.
 
Old 11-05-2015, 06:33 AM   #7
wpeckham
Senior Member
 
Registered: Apr 2010
Location: Continental USA
Distribution: Debian, Ubuntu, Fedora, RedHat, DSL, Puppy, CentOS, Knoppix, Mint-DE, Sparky, Vsido, tinycore, Q4OS
Posts: 3,187

Rep: Reputation: 1380Reputation: 1380Reputation: 1380Reputation: 1380Reputation: 1380Reputation: 1380Reputation: 1380Reputation: 1380Reputation: 1380Reputation: 1380
Ask

Perhaps for historical reasons. It was not long ago that nearly all posix software (and software based upon posix libs) would bork at anything nearing the 2G limit. Some things at 1G! I still deal with remote servers with such behavior on a daily basis, though less often every year.
 
Old 11-05-2015, 10:41 AM   #8
ilesterg
Member
 
Registered: Jul 2012
Location: München
Distribution: Debian, CentOS/RHEL
Posts: 583

Original Poster
Rep: Reputation: 72
Background is that, my client no longer wants to add configurations to this old transfer method written in some old Unix code which I can't even comprehend (most of it), but I am quite hesitant to propose using ssh/sftp/scp instead of this app because I don't know the reason why they did not use ssh/sftp/scp during that time.tho again, I can't be sure how long the scripts have been there, living in the dark. So the only possible theory I had is that, the files are quite big even back then. Hence, the question.
 
Old 11-06-2015, 12:26 AM   #9
chrism01
LQ Guru
 
Registered: Aug 2004
Location: Sydney
Distribution: Centos 6.10, Centos 7.5
Posts: 17,700

Rep: Reputation: 2494Reputation: 2494Reputation: 2494Reputation: 2494Reputation: 2494Reputation: 2494Reputation: 2494Reputation: 2494Reputation: 2494Reputation: 2494Reputation: 2494
These days 2 GB is small...
One of the reasons people used to have issues is that 2G = 2^31 (aka the +ve limit if you use signed 32 bit nums http://www.tsm-resources.com/alists/pow2.html)
 
Old 11-06-2015, 05:38 AM   #10
wpeckham
Senior Member
 
Registered: Apr 2010
Location: Continental USA
Distribution: Debian, Ubuntu, Fedora, RedHat, DSL, Puppy, CentOS, Knoppix, Mint-DE, Sparky, Vsido, tinycore, Q4OS
Posts: 3,187

Rep: Reputation: 1380Reputation: 1380Reputation: 1380Reputation: 1380Reputation: 1380Reputation: 1380Reputation: 1380Reputation: 1380Reputation: 1380Reputation: 1380
And again

I would ask. I would also be ready to do a test to prove that sftp would work for that size file. (or, rsync over ssh: you can resume and interrupted or incomplete transfer and much faster -- though only if you are updating text files in-place)

IF the software on these machines is modern, ssh should serve. If you are talking legacy systems, they may well be handling software that will NOT manage a file that large.

It appears that the guys who know are there, not here.
Or, if they are not there either, there may be no one there who remembers WHY and they want only to not break what works, thus the reluctance to make any modifications. That would then become an exercise in either education or politics, neither of which I would want to delve into here.
 
Old 11-06-2015, 07:28 AM   #11
schneidz
LQ Guru
 
Registered: May 2005
Location: boston, usa
Distribution: fc-15/ fc-20-live-usb/ aix
Posts: 5,272

Rep: Reputation: 914Reputation: 914Reputation: 914Reputation: 914Reputation: 914Reputation: 914Reputation: 914Reputation: 914
Quote:
Originally Posted by chrism01 View Post
These days 2 GB is small...
One of the reasons people used to have issues is that 2G = 2^31 (aka the +ve limit if you use signed 32 bit nums http://www.tsm-resources.com/alists/pow2.html)
i thought that was a limitation with windows fat-32 filesystems ?

also its weird that the limit would be 2gb instead of 4gb (is there any such reason why a files size would be negative) ?
 
Old 11-09-2015, 12:15 AM   #12
chrism01
LQ Guru
 
Registered: Aug 2004
Location: Sydney
Distribution: Centos 6.10, Centos 7.5
Posts: 17,700

Rep: Reputation: 2494Reputation: 2494Reputation: 2494Reputation: 2494Reputation: 2494Reputation: 2494Reputation: 2494Reputation: 2494Reputation: 2494Reputation: 2494Reputation: 2494
I think at the time the original coders used signed ints by default. I'm sure I remember seeing that problem come up, but it did vary over time.

It was probably the normal thing that when systems were written for PCs nobody expected to hit the limits that fast - not entirely unlike Y2K and of course once that's in the system it could have knock on effects if you tried to change it because the ecosystem around it expected the same..

It really depends on the actual SW, not always the FS code necessarily.
If you were already wanting files of 2GB, you were likely going for even bigger, so 4GB (32 bit unsigned) was really only a bandaid soln.

See also https://en.wikipedia.org/wiki/Year_2038_problem - the next big related problem.
 
1 members found this post helpful.
  


Reply

Tags
file transfer, scp, sftp, ssh


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off



Similar Threads
Thread Thread Starter Forum Replies Last Post
Slow transfer using SFTP/SCP dbourcier Linux - Software 1 02-17-2012 01:28 PM
configure ssh authentication using password file and sftp/scp authentication using ld cameliab Linux - Software 1 08-29-2011 04:28 AM
File Transfer with scp,ftp and sftp disabled Question Doknik Linux - General 2 12-03-2010 05:09 AM
[SOLVED] SCP SSH automatic file transfer smithy2010 Linux - Newbie 3 05-09-2010 03:07 PM
Text file editors with built-in SFTP/SSH/SCP ToBe Linux - Software 1 11-19-2008 08:45 AM

LinuxQuestions.org > Forums > Linux Forums > Linux - Software

All times are GMT -5. The time now is 12:10 AM.

Main Menu
Advertisement
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
Facebook: linuxquestions Google+: linuxquestions
Open Source Consulting | Domain Registration