rsync can't handle large files???
Hi,
I'm trying to make a backup from my Linux machine to a mounted shared drive from a Windows machine. The machine is using NTFS and the drive is mounted with samba (of course) Now I'm trying to make the backup with rsync. This with the following command: Code:
rsync -av --no-o --delete /public/ /backup/public Code:
rsync: writefd_unbuffered failed to write 4 bytes [sender]: Broken pipe (32) When I try to backup other files (smaller) then I don't have this problem. Can somebody tell me what is going wrong? Regards Peter-Jan |
that is odd, I know that I have rsync my videos directory which has some 4 gig files in and it worked fine. Maybe the copy of samba is dropping out during such a long transfer. have you tried rsyncing that file over local drives. If that works then it is probably something with samba
|
That was also my guess. But I tried it with the command cp and then it was moving correctly.
Then I tried the rsync over to a local directory and this was also working. Regards |
well since rsync works over the local disks, then it musst be something with samba, are you sure your windows machine is not sleeping or tunring off its hard drives or anything during the copy? That would probably mess up rsync. You could also try setting up ssh on the windows computer and trying to rsync over ssh, that usually works better.
|
Well I'm sure that the machine is not into a sleep or standby mode.
How would I be able to debug this more. I tried the steps that samba suggested but there I found "file too large" and this is all. Any other guess?? Greets |
I retried the command but with now a limited speed
Code:
rsync -av --progress --no-o --delete --bwlimit=2000 /public/ /backup/public Code:
rsync: writefd_unbuffered failed to write 4 bytes [sender]: Broken pipe (32) regards |
what version of rsync are you using, it looks like versions older than 2.6.7 had trouble with multi gig files.
|
I'm using the latest version. I recently installed it.
Code:
pjdeweirdt@linuxserver:~> rsync --version Also I tried it with some other large files and I had the same problem. Regards PJ |
well I have no idea. This sounds like a bug that you should probably report. Have you seen if anyone else has this problem?
|
no not really. How can I report this issue by rsync? Do you have an idea on that?
regards |
well I looked on rsync's bug tracking website. I searched for a file too large error and it looks like there is already a recorded bug. This guy says that he can't sync a file that is about 4 gigs. It has been assigned so they at least do know about it. Hopefully in the 3.0 release, which is the next one they will fix it.
|
Quote:
|
Have you tried tar copy? This could circumvent the rsync issue and verify where the issue exists if you still encounter the problem.
|
I had a similar issue today. Setting the protocol to 2.8 via '--protocol=28' did not help. Not even using --partial worked, but it did allow me to see that the transfer was failing at 19% through. What i did was move the offending file out of the way, on the recipient side, by renaming it to {file}.hold and re-running the script. This allowed the file to synchronize.
It appears that the file was corrupted in such a way on the recipient side that --partial was not able to correct the discrepancy. Completely syncing the file again, worked. BTW, this file was a little over 1 GB in size. -- cheers |
Is the Windows file system able to handle > 3.6 GB?
jlinkels |
All times are GMT -5. The time now is 08:30 AM. |