rsync don't works with many files...

ginoboy · 10-23-2010, 10:47 AM

Guys, is this, I maintain some packages my synchronized with my remote server but I have a problem ... is very hard to understand, time it works, time not works. When not working the following happens:

Code:

[root@xxxx repository]# rsync -avzP --delete-after --partial-dir=.partial-dir rsync://xxxx@xxx.xx.xxx.xxx/repositorio 13.1
Password: 
receiving file list ... 
178 files to consider
rsync error: timeout in data send/receive (code 30) at io.c(137) [sender=3.0.7]
rsync: connection unexpectedly closed (6990 bytes received so far) [receiver]
rsync error: error in rsync protocol data stream (code 12) at io.c(600) [receiver=3.0.6]
rsync: connection unexpectedly closed (6990 bytes received so far) [generator]
rsync error: error in rsync protocol data stream (code 12) at io.c(600) [generator=3.0.6]

this happens when you have many files.... strange

I'm using slackware 13.1...

any idea?

escaflown · 10-23-2010, 01:05 PM

178 files is a small number of rsync. You should have a look at your network connections.

ginoboy · 10-23-2010, 03:20 PM

Quote:

Originally Posted by escaflown

178 files is a small number of rsync. You should have a look at your network connections.

Yes, but if a few files (less than 50 for example) works fine.

If I synchronize folders over the local LAN, it works. But it happens on the Internet.

Anyone have any examples of configuration of rsync?

Difficult to understand. On the local network works fine, but if I give this problem on the Internet (port 873 is open on my system).

Thanks.

maxmiorim · 10-23-2010, 03:50 PM

You can try without compression (remove the -z from the command line), use a high value for --timeout (even though it's 0 by default, which means it shouldn't timeout at all) or use rsync over ssh.

Personally, I like to use rsync over ssh because a couple of ISPs that I know tend to prioritize SSH, which helps to avoid such timeouts.

Also, rsync -vvvv in both sides should spit out more details that might help you, like the protocol version used, wether keep-alive packets are being sent or not, what happens right before the timeout and etc.

ginoboy · 10-23-2010, 07:17 PM

I've tried using without compression..

max, you can give me an example of how you use via SSH?

abraço

maxmiorim · 10-23-2010, 08:22 PM

It's pretty much like any ssh tunnel:

1 - Establish the tunnel

Code:

ssh -L 8873:the.remote.host:873 -fNC localhost

The above line will make ssh act like a SOCKS proxy, sending the requests to 127.0.0.1:8873 through localhost to the.remote.host:873.
It will enable compression (-C), go into background (-f) and don't allocate a pseudo-terminal (-N). See ssh(1).

2 - Rsync through it

Code:

rsync -avP --delete-after --partial-dir=.partial-dir rsync://127.0.0.1:8873/repositorio 13.1

It's almost the same thing that you use, except that here compression is disabled (ssh is taking care of it, remember?) and you send the requests to 127.0.0.1:8873 (which goes to the.remote.host:873 through the tunnel).

3 - Kill the tunnel

Code:

pkill -TERM -x "ssh -L 8873:the.remote.host:873 -fNC localhost"

Note: netstat and lsof will give you the PID with a bit of awk/grep, then you can kill it using the ol' good kill if you don't like/want to use pkill.

ginoboy · 10-24-2010, 06:15 AM

Hi Max, see:

Code:

$ssh -L 8873:myip:873 -fNC localhost
$rsync -avP --delete-after --partial-dir=.partial-dir rsync://127.0.0.1:8873/sbo slackbuilds 
receiving file list ... 
19307 files to consider
created directory slackbuilds
./
rsync error: timeout in data send/receive (code 30) at io.c(137) [sender=3.0.7]
rsync: connection unexpectedly closed (445368 bytes received so far) [receiver]
rsync error: error in rsync protocol data stream (code 12) at io.c(600) [receiver=3.0.6]
rsync: connection unexpectedly closed (445365 bytes received so far) [generator]                                                                            
rsync error: error in rsync protocol data stream (code 12) at io.c(600) [generator=3.0.6]

maxmiorim · 10-24-2010, 05:54 PM

Hmm, I'm getting out of ideas. :\

Did you read [1] already?

Also, are you sure it's not a network problem like packet loss or some kind of shaper dropping the packets to throttle the transfer rates? The rsync option --bwlimit may help to work around the latter.

[1] - http://samba.anu.edu.au/rsync/issues.html

plofkat · 06-19-2015, 12:47 AM

I have the same problem with rsync timing out after a day or so.

I am copying server images (almost 7TB)
At the moment, as a work around I am splitting the copy into smaller groups to see if that works.

File copy is on local lan - Gigabit network with 6gbps backbone between switches

The error I get after a few hours:

Code:

rsync: connection unexpectedly closed (85406994 bytes received so far) [sender]
rsync error: error in rsync protocol data stream (code 12) at io.c(601) [sender=3.0.7]

kjhambrick · 06-21-2015, 05:35 PM

ginoboy --

I have had occasional indeterminate issues with remote rsync servers from time-to-time myself. The issues are on machines where I've got no admin rights so I've not bothered to debug.

OTOH ... If there is an ssh server open on the remote server, you can also invoke the rsync -e '<<ssh shell>>' syntax.

Assuming:

Code:

   uuuuuuuu is your remote username, 
   xxx.xxx.xxx.xxx is the Source Machine's IP Addr,
   /remote/path/to/repositorio is the full path to the remote Source directory and ...
   /local/path/to/13.1 is the directory on your local machine containing the 13.1 Target directory.

You'll need to convert the rsync server Source directory ( repositorio ) to a full path on the remote machine and you'll need permissions to reach it !

The following command should work without having to set up a separate ssh tunnel ( note the single-quote pair around the -e arg the -p <<PORT>> Arg is optional if ssh listens on port 22 ):

Code:

rsync -e 'ssh -p 22 -l UUUUUUUU' -Pavz --delete-after --partial-dir=.partial-dir xxx.xx.xxx.xxx:/remote/path/to/repositorio /local/path/to/13.1

Maybe this will help ?

-- kjh

mralk3 · 06-21-2015, 07:20 PM

Quote:

Originally Posted by ginoboy

Guys, is this, I maintain some packages my synchronized with my remote server but I have a problem ... is very hard to understand, time it works, time not works. When not working the following happens:

Code:

[root@xxxx repository]# rsync -avzP --delete-after --partial-dir=.partial-dir rsync://xxxx@xxx.xx.xxx.xxx/repositorio 13.1
Password: 
receiving file list ... 
178 files to consider
rsync error: timeout in data send/receive (code 30) at io.c(137) [sender=3.0.7]
rsync: connection unexpectedly closed (6990 bytes received so far) [receiver]
rsync error: error in rsync protocol data stream (code 12) at io.c(600) [receiver=3.0.6]
rsync: connection unexpectedly closed (6990 bytes received so far) [generator]
rsync error: error in rsync protocol data stream (code 12) at io.c(600) [generator=3.0.6]

this happens when you have many files.... strange

I'm using slackware 13.1...

any idea?

If you had opened your web browser and used google, you would have seen the very first search result provided a serverfault.com thread discussing a very similar question.

http://serverfault.com/questions/370...s-with-timeout

To summarize, run tcpdump that listens on the rsync port and run strace on the rsync process. You will find your answer within the output of those commands.

More than likely you just have intermittent internet connectivity that is causing your rsync session to error out.

bassmadrigal · 06-23-2015, 12:28 PM

Quote:

Originally Posted by mralk3

If you had opened your web browser and used google, you would have seen the very first search result provided a serverfault.com thread discussing a very similar question.

http://serverfault.com/questions/370...s-with-timeout

To summarize, run tcpdump that listens on the rsync port and run strace on the rsync process. You will find your answer within the output of those commands.

More than likely you just have intermittent internet connectivity that is causing your rsync session to error out.

plofkat necro'd the thread. You quoted the original post, which was made back in 2010, where the serverfault question was in 2012.

mralk3 · 06-23-2015, 06:26 PM

Quote:

Originally Posted by bassmadrigal

plofkat necro'd the thread. You quoted the original post, which was made back in 2010, where the serverfault question was in 2012.

I hate necros!

rouvas · 06-24-2015, 11:45 AM

You are seeing this problem, because

The remote server is taking too long to respond to the rsync request, ergo the rsync from client-side drops the connection or thinks that the connection has been somehow been lost.

Monitor the load on the remote server or check for other h/w issues (maybe faulty disks, faulty NICs, etc)