rsync can't handle large files???

deweirdt · 11-23-2006, 07:31 AM

Hi,

I'm trying to make a backup from my Linux machine to a mounted shared drive from a Windows machine.
The machine is using NTFS and the drive is mounted with samba (of course)

Now I'm trying to make the backup with rsync. This with the following command:

Code:

rsync -av --no-o --delete /public/ /backup/public

the error that I get from rsync is:

Code:

rsync: writefd_unbuffered failed to write 4 bytes [sender]: Broken pipe (32)
rsync: write failed on "/backup/public/dev.iso": File too large (27)
rsync error: error in file IO (code 11) at receiver.c(258) [receiver=2.6.9]
rsync: connection unexpectedly closed (724604 bytes received so far) [sender]
rsync error: error in rsync protocol data stream (code 12) at io.c(453) [sender=2.6.9]

The file dev.iso is 3.6GB
When I try to backup other files (smaller) then I don't have this problem.

Can somebody tell me what is going wrong?

Regards Peter-Jan

reeseslover531 · 11-23-2006, 08:17 AM

that is odd, I know that I have rsync my videos directory which has some 4 gig files in and it worked fine. Maybe the copy of samba is dropping out during such a long transfer. have you tried rsyncing that file over local drives. If that works then it is probably something with samba

deweirdt · 11-23-2006, 08:41 AM

That was also my guess. But I tried it with the command cp and then it was moving correctly.

Then I tried the rsync over to a local directory and this was also working.

Regards

reeseslover531 · 11-23-2006, 10:13 AM

well since rsync works over the local disks, then it musst be something with samba, are you sure your windows machine is not sleeping or tunring off its hard drives or anything during the copy? That would probably mess up rsync. You could also try setting up ssh on the windows computer and trying to rsync over ssh, that usually works better.

deweirdt · 11-23-2006, 10:57 AM

Well I'm sure that the machine is not into a sleep or standby mode.
How would I be able to debug this more. I tried the steps that samba suggested but there I found "file too large" and this is all.

Any other guess??

Greets

deweirdt · 11-23-2006, 11:25 AM

I retried the command but with now a limited speed

Code:

rsync -av --progress --no-o --delete --bwlimit=2000 /public/ /backup/public

but I got again the error

Code:

rsync: writefd_unbuffered failed to write 4 bytes [sender]: Broken pipe (32)
rsync: write failed on "/backup/public/dev.iso": File too large (27)
rsync error: error in file IO (code 11) at receiver.c(258) [receiver=2.6.9]
rsync: connection unexpectedly closed (76 bytes received so far) [generator]
rsync error: error in rsync protocol data stream (code 12) at io.c(453) [generator=2.6.9]
rsync: connection unexpectedly closed (36 bytes received so far) [sender]
rsync error: error in rsync protocol data stream (code 12) at io.c(453) [sender=2.6.9]

I'm sure now that the connection did not reset.

regards

reeseslover531 · 11-23-2006, 06:12 PM

what version of rsync are you using, it looks like versions older than 2.6.7 had trouble with multi gig files.

deweirdt · 11-24-2006, 01:28 AM

I'm using the latest version. I recently installed it.

Code:

pjdeweirdt@linuxserver:~> rsync --version
rsync  version 2.6.9  protocol version 29
Copyright (C) 1996-2006 by Andrew Tridgell, Wayne Davison, and others.
<http://rsync.samba.org/>
Capabilities: 64-bit files, socketpairs, hard links, symlinks, batchfiles,
              inplace, IPv6, 64-bit system inums, 64-bit internal inums

rsync comes with ABSOLUTELY NO WARRANTY.  This is free software, and you
are welcome to redistribute it under certain conditions.  See the GNU
General Public Licence for details.

Also I tried it with some other large files and I had the same problem.

Regards PJ

reeseslover531 · 11-24-2006, 09:31 AM

well I have no idea. This sounds like a bug that you should probably report. Have you seen if anyone else has this problem?

deweirdt · 11-24-2006, 10:15 AM

no not really. How can I report this issue by rsync? Do you have an idea on that?

regards

reeseslover531 · 11-25-2006, 07:56 AM

well I looked on rsync's bug tracking website. I searched for a file too large error and it looks like there is already a recorded bug. This guy says that he can't sync a file that is about 4 gigs. It has been assigned so they at least do know about it. Hopefully in the 3.0 release, which is the next one they will fix it.

whit · 03-13-2010, 05:03 PM

Quote:

Originally Posted by reeseslover531

... a recorded bug. This guy says that he can't sync a file that is about 4 gigs. It has been assigned so they at least do know about it. Hopefully in the 3.0 release, which is the next one they will fix it.

Oh joy, I've discovered it can still be a problem in 3.0.7 built from the source from the rsync site. It looks like they may have missed fixing a bug similar to this one in librsync, a separate but related project.

RaelOM · 03-15-2010, 09:27 AM

Have you tried tar copy? This could circumvent the rsync issue and verify where the issue exists if you still encounter the problem.

Linuks83 · 02-10-2011, 05:28 PM

I had a similar issue today. Setting the protocol to 2.8 via '--protocol=28' did not help. Not even using --partial worked, but it did allow me to see that the transfer was failing at 19% through. What i did was move the offending file out of the way, on the recipient side, by renaming it to {file}.hold and re-running the script. This allowed the file to synchronize.

It appears that the file was corrupted in such a way on the recipient side that --partial was not able to correct the discrepancy. Completely syncing the file again, worked.

BTW, this file was a little over 1 GB in size.

-- cheers

jlinkels · 02-10-2011, 08:00 PM

Is the Windows file system able to handle > 3.6 GB?

jlinkels