rsync fork failed because of very large number of files transfer

unclesamcrazy · 11-20-2015, 04:48 AM

I am trying to transfer very large number of files on remote server but rsync process stopped unexpectedly with following error.

Code:

rsync: fork failed in do_recv: Cannot allocate memory (12)
rsync error: error in IPC code (code 14) at main.c(709) [receiver=2.6.9]
rsync: connection unexpectedly closed (8 bytes received so far) [sender]
rsync error: error in IPC code (code 14) at io.c(605) [sender=3.0.9]

I am using following command to transfer files.

Code:

rsync -avzW /source/folder/ --exclude="first-folder" --exclude="2-folder" --exclude="3-folder" --exclude="4-folder" --exclude="5-folder" 192.168.0.200:/Destination/backup

Please help to solve the error, it was working fine a week before but now it is not working with same configuration of source and destination.

Thank you

berndbausch · 11-20-2015, 04:58 AM

A superficial google indicates that this is a problem in the rsync algorithm that goes away with version 3. From your output, it would seem that the receiver is still using version 2.6.9 - perhaps it can be upgraded?

unclesamcrazy · 11-20-2015, 05:15 AM

The receiver is iomega NAS. How should I upgrade, Please help me.

descendant_command · 11-20-2015, 05:19 AM

Quote:

Originally Posted by unclesamcrazy

The receiver is iomega NAS. How should I upgrade, Please help me.

Ask iomega?

rsync is memory intensive with lots of files - maybe break your transfer into smaller sections

unclesamcrazy · 11-23-2015, 12:14 AM

Thanks for help... now I am transferring data in smaller sections...

Code:

find /source/directory -maxdepth 1 -mindepth 1 -type d -exec rsync -av {} --exclude="1folder" --exclude="2folder" --exclude="3folder" --exclude="4folder" --exclude="5folder" 192.168.0.200:/destination/folder/backup \;

Although it is taking time 5 times more than previous time but at least backup process has been started.

malekmustaq · 11-23-2015, 07:20 AM

And you can put the list of excluded items into a single text file listing all the folder names or file-types to be excluded at the source-host, put the excluding list file in the home directory in the host from where rsync is issued, then just define the switch e.g. "--exclude=/my_exluded_list.txt"

unclesamcrazy · 11-23-2015, 11:15 PM

Quote:

Originally Posted by malekmustaq

And you can put the list of excluded items into a single text file listing all the folder names or file-types to be excluded at the source-host, put the excluding list file in the home directory in the host from where rsync is issued, then just define the switch e.g. "--exclude=/my_exluded_list.txt"

It did not make much difference. Initially when I used to take back up of whole source directory, it used to build all files of that directory at once with text "building file list ... done".
But now I am copying backup directory wise, it is building files for each directory and that's why it is taking time more than expected.

There are millions files in source directory, I will paste exact number here once I get the output of

Code:

file source/dir -type f | wc -l

Although the memory exhausted error is not solved. If you have any solution for this error (very very large number of files), please share.
Because I am getting the error while building file. It is not able to build such a large number of files.

Thanks

descendant_command · 11-23-2015, 11:18 PM

Add more memory (or swap, but that will be *slow*).

astrogeek · 11-23-2015, 11:41 PM

With millions of files it is going to be slow, as already noted.

You don't say anything about the target disk size, but in addition to the memory problem you might run into an inode limit on the drive itself, even if the drive space is sufficient. This will vary with the filesystem, also not mentioned so far.

How about a few more details. Target drive partition size, filesystem type, typical file size, length of exclude list, come to mind for starters.

serverpoint.com · 11-24-2015, 12:23 AM

I think its better to upgrade rsync in to version 3 and use the command line option --no-inc-recursive. Have you got a chance to contact iomega for upgrade options?

unclesamcrazy · 11-24-2015, 12:30 AM

No I will not be able to contact support, I need to do it myself and I am not able to do it

The total number of files in source directory : 3845408
The size of the source directory : 250 GB
Operating system source : centos 7.1
operating system destination : iomega NAS
cron job runs on : source operating system centos 7.1
Source hard disk size : 1 TB
/dev/mapper/centos-root 50G 16G 35G 31 /
/dev/mapper/centos-home 879G 253G 627G 29 /home
Source RAM : 1849336 kB
Source SWAP : 2113532 kB
Source processor : Pentium(R) Dual-Core CPU E5700 @ 3.00GHz
siblings : 2
cpu cores : 2