LinuxQuestions.org
Share your knowledge at the LQ Wiki.
Home Forums Tutorials Articles Register
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - Networking
User Name
Password
Linux - Networking This forum is for any issue related to networks or networking.
Routing, network cards, OSI, etc. Anything is fair game.

Notices


Reply
  Search this Thread
Old 06-05-2017, 01:16 PM   #1
TheLexx
Member
 
Registered: Apr 2013
Distribution: Gentoo
Posts: 79

Rep: Reputation: Disabled
sftp large file transfer corruption


When I transfer files over my LAN that are greater than 1 GB a few bytes get corrupted. I have tested with files that are under 100 MG and they transfer well. The corruption is so small that I need a program such as md5sum to detect the corruption. This is odd because I thought sftp had built-in error correction. I have used sftp with Finnix and with Gentoo and I get errors regardless. Has anyone else noticed this problem with sftp? has anyone found a workaround.
 
Old 06-05-2017, 01:35 PM   #2
business_kid
LQ Guru
 
Registered: Jan 2006
Location: Ireland
Distribution: Slackware, Slarm64 & Android
Posts: 16,370

Rep: Reputation: 2335Reputation: 2335Reputation: 2335Reputation: 2335Reputation: 2335Reputation: 2335Reputation: 2335Reputation: 2335Reputation: 2335Reputation: 2335Reputation: 2335
To quote someone (was it Ashleigh Brilliant?)

"I don't have a solution for you but I certainly admire the problem."

Can you give us a bit more on the errors? Particularly, the offsets, and also if you're running on a 32bit or 64bit machine, and any other facts you notice as you compare originals with damaged. You will probably have to use some hex comparison program to get offsets, as I think diff is too text based. If you repeat a copy (original --> sftp --> file1, original --> sftp --> file2, diff file1 file2), do file1 & file2 differ?
The 32 or 64 bit matters because of data bus width; there could be a maximum size accidentally built in to sftp somewhere by the coding constructs.
 
Old 06-05-2017, 01:40 PM   #3
Habitual
LQ Veteran
 
Registered: Jan 2011
Location: Abingdon, VA
Distribution: Catalina
Posts: 9,374
Blog Entries: 37

Rep: Reputation: Disabled
same here.
Of all the Ss, I use scp the most for file xfer.
 
Old 06-05-2017, 01:56 PM   #4
Turbocapitalist
LQ Guru
 
Registered: Apr 2005
Distribution: Linux Mint, Devuan, OpenBSD
Posts: 7,331
Blog Entries: 3

Rep: Reputation: 3726Reputation: 3726Reputation: 3726Reputation: 3726Reputation: 3726Reputation: 3726Reputation: 3726Reputation: 3726Reputation: 3726Reputation: 3726Reputation: 3726
I usually use rsync for large transfers. Back when the net was less reliable, large transfers were very difficult and I never gained full trust in what we have now, even if it is better. So with rsync if the transfer works on the first time, it works. If it needs a retry or two, then there is little waste in retransmission.

Edit: and nowadays rsync works over SSH by default.

Last edited by Turbocapitalist; 06-05-2017 at 01:57 PM.
 
Old 06-06-2017, 03:56 AM   #5
business_kid
LQ Guru
 
Registered: Jan 2006
Location: Ireland
Distribution: Slackware, Slarm64 & Android
Posts: 16,370

Rep: Reputation: 2335Reputation: 2335Reputation: 2335Reputation: 2335Reputation: 2335Reputation: 2335Reputation: 2335Reputation: 2335Reputation: 2335Reputation: 2335Reputation: 2335
Having come at this from the hardware end, and having had a career considering component internal capacitances, logic thresholds, speeds, cable capacitance, speed & leakage vs. temperature, etc., it is a continual wonder to me that what we have is as reliable as it is, and can be pushed as fast as they are currently doing. The signal at the receiving end of a long cable (or optic fibre) is a very different looking and inferior signal from what was transmitted.
 
  


Reply



Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off



Similar Threads
Thread Thread Starter Forum Replies Last Post
SFTP throwing "End of file" when doing SFTP transfer in perl exceed1 Programming 5 03-06-2010 01:34 PM
File transfer using SFTP santhoshv Linux - Networking 1 09-18-2008 06:33 AM
sftp problem: unable to transfer file illscience Linux - Networking 2 09-06-2008 11:26 AM
transfer file using expect and sftp tanveer Linux - General 7 05-16-2008 10:29 AM
Linux CIFS large file corruption adz Linux - Kernel 0 11-05-2006 12:19 AM

LinuxQuestions.org > Forums > Linux Forums > Linux - Networking

All times are GMT -5. The time now is 06:46 AM.

Main Menu
Advertisement
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
Open Source Consulting | Domain Registration