rsync pull on ubuntu 8.10

mrloaf · 06-04-2009, 05:04 PM

I'm setting up a box running Ubuntu 8.10 to rsync my FreeBSD server(s). I want to pull data down to the Ubuntu box, then send it all upstream to an offsite data center. But first I have to get rsync to work.

This works, sort of:

Code:

rsync -halvz /home/administrator/.gvfs/sftp\ for\ root\ on\ myserver /local/backups/ --log-file=/tmp/rlog --exclude-from=/home/administrator/rsync-exclude --delete

It runs for a while, then it errors with:

Code:

rsync: writefd_unbuffered failed to write 461 bytes [sender]: Broken pipe (32)
rsync: connection unexpectedly closed (1258 bytes received so far) [sender]
rsync error: error allocating core memory buffers (code 22) at io.c(600) [sender=3.0.5]

And after that error, it won't run anymore - the error keeps happening. If it were just running out of memory, a restart ought to fix it, but doesn't. The box should be adequate - 2.4 GHz dual P4 with 1 GB ram.

Also: I know there must be a better way to connect to the source, but how? I've been mounting the server via SFTP in the GUI (hence the .gvfs path) which is probably not the fastest way. But ssh thru the shell doesn't seem to work.

rsync daemon is NOT running on the source FreeBSD server.

I'll send whatever add'l info people would like to see - thanks in advance!

irishbitte · 06-04-2009, 06:38 PM

This I'm not getting:

Quote:

...../.gvfs/sftp\ for\ root\ on\ myserver /local/backups/.....

Whats the for\ root\ on\ myserver bit about?

billymayday · 06-04-2009, 06:42 PM

I assume the remote path is "/home/administrator/.gvfs/sftp for root on myserver" and the spaces have been escaped.

However, if you are pulling from a remote server, you don't mention the remote anywhere. Why isn't it something like

rsync -options user@remote:/path /local_path

mrloaf · 06-04-2009, 09:20 PM

Quote:

Originally Posted by billymayday

I assume the remote path is "/home/administrator/.gvfs/sftp for root on myserver" and the spaces have been escaped.

However, if you are pulling from a remote server, you don't mention the remote anywhere. Why isn't it something like

rsync -options user@remote:/path /local_path

I tried user@remote:/path - didn't work. So I resorted to mounting the server in Gnome, and using the local mount point (the .gvfs directory) as the source.

irishbitte · 06-05-2009, 02:24 PM

Quote:

I tried user@remote:/path - didn't work. So I resorted to mounting the server in Gnome, and using the local mount point (the .gvfs directory) as the source.

I don't think that will work very well, tell me, have you an ssh server running on the remote machine, and the keys setup correctly for remote login?

mrloaf · 06-06-2009, 03:49 PM

Quote:

Originally Posted by irishbitte

I don't think that will work very well, tell me, have you an ssh server running on the remote machine, and the keys setup correctly for remote login?

Yes, I can ssh into the remote machine from the command line. And the rsync error doesn't happen until after it asks me for the password (but I'm not getting a password error). I haven't set up keys yet for unattended login in rsync.

irishbitte · 06-07-2009, 08:42 PM

read all of this, you may find it quite enlightening: http://sial.org/howto/rsync/

mrloaf · 06-08-2009, 08:23 PM

Quote:

Originally Posted by irishbitte

read all of this, you may find it quite enlightening: http://sial.org/howto/rsync/

thanks - but still no joy on my rsync server.
Do I need to have rsync running on the client as well?

billymayday · 06-08-2009, 08:32 PM

No you don't

What are the logs on the server telling you?

mrloaf · 06-08-2009, 09:03 PM

Quote:

Originally Posted by billymayday

No you don't

What are the logs on the server telling you?

The logs are just the same error messages I get in the terminal. Not much information there. Unless I'm looking at the wrong log?

mrloaf · 06-10-2009, 01:15 PM

Quote:

Originally Posted by billymayday

No you don't

What are the logs on the server telling you?

Thinking that my partitioning & mountpoint scheme might have been contributing to the problem, I yanked the second hard drive I was using before and re-partitioned using vanilla defaults on one drive.

Now rsync doesn't work at all! Different options give different errors. Here's the log results from several attempts. #2 was the closest, but in no case did anything get copied!

Code:

rsync --log-file=/home/administrator/Desktop/rsync-log root@192.168.1.4:/data /home/administrator/backups/fileserver/

2009/06/10 14:01:41 [18260] rsync: connection unexpectedly closed (0 bytes received so far) [receiver]
2009/06/10 14:01:41 [18260] rsync error: error in rsync protocol data stream (code 12) at io.c(600) [receiver=3.0.5]

Code:

rsync -avz --log-file=/home/administrator/Desktop/rsync-log -e root@192.168.1.4:/data /home/administrator/backups/fileserver/

2009/06/10 14:02:14 [18264] building file list
2009/06/10 14:02:14 [18264] drwxrwx--- 4096 2009/06/08 11:22:41 .
2009/06/10 14:02:15 [18264] sent 57 bytes received 12 bytes 138.00 bytes/sec
2009/06/10 14:02:15 [18264] total size is 0 speedup is 0.00

Code:

rsync -avz --log-file=/home/administrator/Desktop/rsync-log -e 'ssh' root@192.168.1.4:/data /home/administrator/backups/fileserver/

2009/06/10 14:02:35 [18270] rsync: connection unexpectedly closed (0 bytes received so far) [receiver]
2009/06/10 14:02:35 [18270] rsync error: error in rsync protocol data stream (code 12) at io.c(600) [receiver=3.0.5]

irishbitte · 06-10-2009, 03:29 PM

This will only work if you have SSH keys setup between your machine you are sitting at, and the machine with IP 192.168.1.4! You won't have these problems when you setup keys between your machines for unattended rsync. check out the link I sent you earlier!

mrloaf · 06-17-2009, 02:25 PM

I mounted the remote volumes via sshfs, and now it works great.
Thanks for the help everyone!