LinuxQuestions.org
Download your favorite Linux distribution at LQ ISO.
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - Server
User Name
Password
Linux - Server This forum is for the discussion of Linux Software used in a server related context.

Notices

Reply
 
Search this Thread
Old 11-23-2006, 08:31 AM   #1
deweirdt
Member
 
Registered: Sep 2003
Location: Belgium
Distribution: SUSE10.1
Posts: 40

Rep: Reputation: 15
rsync can't handle large files???


Hi,

I'm trying to make a backup from my Linux machine to a mounted shared drive from a Windows machine.
The machine is using NTFS and the drive is mounted with samba (of course)

Now I'm trying to make the backup with rsync. This with the following command:
Code:
rsync -av --no-o --delete /public/ /backup/public
the error that I get from rsync is:
Code:
rsync: writefd_unbuffered failed to write 4 bytes [sender]: Broken pipe (32)
rsync: write failed on "/backup/public/dev.iso": File too large (27)
rsync error: error in file IO (code 11) at receiver.c(258) [receiver=2.6.9]
rsync: connection unexpectedly closed (724604 bytes received so far) [sender]
rsync error: error in rsync protocol data stream (code 12) at io.c(453) [sender=2.6.9]
The file dev.iso is 3.6GB
When I try to backup other files (smaller) then I don't have this problem.

Can somebody tell me what is going wrong?

Regards Peter-Jan
 
Old 11-23-2006, 09:17 AM   #2
reeseslover531
Member
 
Registered: Nov 2005
Distribution: Fedora Core 5
Posts: 64

Rep: Reputation: 15
that is odd, I know that I have rsync my videos directory which has some 4 gig files in and it worked fine. Maybe the copy of samba is dropping out during such a long transfer. have you tried rsyncing that file over local drives. If that works then it is probably something with samba
 
Old 11-23-2006, 09:41 AM   #3
deweirdt
Member
 
Registered: Sep 2003
Location: Belgium
Distribution: SUSE10.1
Posts: 40

Original Poster
Rep: Reputation: 15
That was also my guess. But I tried it with the command cp and then it was moving correctly.

Then I tried the rsync over to a local directory and this was also working.


Regards
 
Old 11-23-2006, 11:13 AM   #4
reeseslover531
Member
 
Registered: Nov 2005
Distribution: Fedora Core 5
Posts: 64

Rep: Reputation: 15
well since rsync works over the local disks, then it musst be something with samba, are you sure your windows machine is not sleeping or tunring off its hard drives or anything during the copy? That would probably mess up rsync. You could also try setting up ssh on the windows computer and trying to rsync over ssh, that usually works better.
 
Old 11-23-2006, 11:57 AM   #5
deweirdt
Member
 
Registered: Sep 2003
Location: Belgium
Distribution: SUSE10.1
Posts: 40

Original Poster
Rep: Reputation: 15
Well I'm sure that the machine is not into a sleep or standby mode.
How would I be able to debug this more. I tried the steps that samba suggested but there I found "file too large" and this is all.

Any other guess??

Greets
 
Old 11-23-2006, 12:25 PM   #6
deweirdt
Member
 
Registered: Sep 2003
Location: Belgium
Distribution: SUSE10.1
Posts: 40

Original Poster
Rep: Reputation: 15
I retried the command but with now a limited speed
Code:
rsync -av --progress --no-o --delete --bwlimit=2000 /public/ /backup/public
but I got again the error

Code:
rsync: writefd_unbuffered failed to write 4 bytes [sender]: Broken pipe (32)
rsync: write failed on "/backup/public/dev.iso": File too large (27)
rsync error: error in file IO (code 11) at receiver.c(258) [receiver=2.6.9]
rsync: connection unexpectedly closed (76 bytes received so far) [generator]
rsync error: error in rsync protocol data stream (code 12) at io.c(453) [generator=2.6.9]
rsync: connection unexpectedly closed (36 bytes received so far) [sender]
rsync error: error in rsync protocol data stream (code 12) at io.c(453) [sender=2.6.9]
I'm sure now that the connection did not reset.

regards
 
Old 11-23-2006, 07:12 PM   #7
reeseslover531
Member
 
Registered: Nov 2005
Distribution: Fedora Core 5
Posts: 64

Rep: Reputation: 15
what version of rsync are you using, it looks like versions older than 2.6.7 had trouble with multi gig files.
 
Old 11-24-2006, 02:28 AM   #8
deweirdt
Member
 
Registered: Sep 2003
Location: Belgium
Distribution: SUSE10.1
Posts: 40

Original Poster
Rep: Reputation: 15
I'm using the latest version. I recently installed it.

Code:
pjdeweirdt@linuxserver:~> rsync --version
rsync  version 2.6.9  protocol version 29
Copyright (C) 1996-2006 by Andrew Tridgell, Wayne Davison, and others.
<http://rsync.samba.org/>
Capabilities: 64-bit files, socketpairs, hard links, symlinks, batchfiles,
              inplace, IPv6, 64-bit system inums, 64-bit internal inums

rsync comes with ABSOLUTELY NO WARRANTY.  This is free software, and you
are welcome to redistribute it under certain conditions.  See the GNU
General Public Licence for details.

Also I tried it with some other large files and I had the same problem.

Regards PJ
 
Old 11-24-2006, 10:31 AM   #9
reeseslover531
Member
 
Registered: Nov 2005
Distribution: Fedora Core 5
Posts: 64

Rep: Reputation: 15
well I have no idea. This sounds like a bug that you should probably report. Have you seen if anyone else has this problem?
 
Old 11-24-2006, 11:15 AM   #10
deweirdt
Member
 
Registered: Sep 2003
Location: Belgium
Distribution: SUSE10.1
Posts: 40

Original Poster
Rep: Reputation: 15
no not really. How can I report this issue by rsync? Do you have an idea on that?

regards
 
Old 11-25-2006, 08:56 AM   #11
reeseslover531
Member
 
Registered: Nov 2005
Distribution: Fedora Core 5
Posts: 64

Rep: Reputation: 15
well I looked on rsync's bug tracking website. I searched for a file too large error and it looks like there is already a recorded bug. This guy says that he can't sync a file that is about 4 gigs. It has been assigned so they at least do know about it. Hopefully in the 3.0 release, which is the next one they will fix it.
 
Old 03-13-2010, 06:03 PM   #12
whit
LQ Newbie
 
Registered: Jul 2003
Location: Vermont
Distribution: Gentoo
Posts: 3

Rep: Reputation: 0
Quote:
Originally Posted by reeseslover531 View Post
... a recorded bug. This guy says that he can't sync a file that is about 4 gigs. It has been assigned so they at least do know about it. Hopefully in the 3.0 release, which is the next one they will fix it.
Oh joy, I've discovered it can still be a problem in 3.0.7 built from the source from the rsync site. It looks like they may have missed fixing a bug similar to this one in librsync, a separate but related project.
 
Old 03-15-2010, 10:27 AM   #13
RaelOM
Member
 
Registered: Dec 2004
Posts: 110

Rep: Reputation: 16
Have you tried tar copy? This could circumvent the rsync issue and verify where the issue exists if you still encounter the problem.
 
Old 02-10-2011, 06:28 PM   #14
Linuks83
LQ Newbie
 
Registered: Feb 2011
Posts: 2

Rep: Reputation: 0
I had a similar issue today. Setting the protocol to 2.8 via '--protocol=28' did not help. Not even using --partial worked, but it did allow me to see that the transfer was failing at 19% through. What i did was move the offending file out of the way, on the recipient side, by renaming it to {file}.hold and re-running the script. This allowed the file to synchronize.

It appears that the file was corrupted in such a way on the recipient side that --partial was not able to correct the discrepancy. Completely syncing the file again, worked.

BTW, this file was a little over 1 GB in size.

-- cheers

Last edited by Linuks83; 02-10-2011 at 06:35 PM. Reason: further clarification
 
Old 02-10-2011, 09:00 PM   #15
jlinkels
Senior Member
 
Registered: Oct 2003
Location: Bonaire
Distribution: Debian Wheezy/Jessie/Sid, Linux Mint DE
Posts: 4,193

Rep: Reputation: 538Reputation: 538Reputation: 538Reputation: 538Reputation: 538Reputation: 538
Is the Windows file system able to handle > 3.6 GB?

jlinkels
 
  


Reply


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off


Similar Threads
Thread Thread Starter Forum Replies Last Post
Totem can't handle .mp3 files Gins Linux - General 1 11-14-2006 05:39 AM
Rsync: can it handle any and all file formats? hamish Linux - Software 3 12-12-2005 12:18 AM
Program to Handle NZB Files CarnageBox Linux - Software 2 11-02-2005 07:14 PM
How to handle big files > 2Go ??? 242VDM242 Linux - Newbie 9 04-06-2005 12:28 PM
ability to handle MS Word files Maidros Debian 3 03-17-2004 02:58 PM


All times are GMT -5. The time now is 03:33 AM.

Main Menu
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
identi.ca: @linuxquestions
Facebook: linuxquestions Google+: linuxquestions
Open Source Consulting | Domain Registration