I'll try not to give too much detail here, but I'm having a nightmare with rsync. I have my current 1.0GHz single core, 512MB RAM SoC server running Debian, which is mechanically dying of old age and needs replacing. I have built a new one on Ubuntu, and now I need to complete the final data sync between the two servers.
All the hardware is Gigabit capable; the switch is a brand new Netgear Nighthawk, network cables are Cat 6 or 7 and brand new.
I have tried removing the drive from the old server and mounting locally to the new one, but despite many hours of trying it cannot read the disc.
I have tried SSHFS - new to old, but I cannot get the old system to accept the mount, so abandoned that idea.
My next thought was rsync, but that has proved a problem, so I am currently using Netcat and tar to transfer 2.2TB of data.
I started using a shell script with a series of lines like this:
#rsync -avz --progress --delete -e "ssh -p xxxxx" yyy.yyy.yyy.yyy:/var/spool/mail/ /var/spool/mail
The problem was, I was getting @1.2MBps data transfers, which is not what I should be getting, but I figured the old server CPU and HDD are the limiting factor. I then did some surfing to find the following:
#rsync -aHAXxv --delete --progress -e "ssh -T -p xxxxx -o Compression=no -x" yyy.yyy.yyy.yyy:/home/storage/extern/disk_0/music/ /home/storage/extern/disk_0/music
This gives me transfers of 8MBps, which is way better, but still not what I should be getting.
The problem now is if I run this a second time to pick up any changes, it is creating a new directory called music? (note the question mark at the end). It is also not copying permissions, users or groups as I would expect. Everything is root:root with chmod 700.
Can anyone spot my error?

It this is your internal network, allow SSH access for root. It makes life much easier regarding retaining permissions. This is the command you want to run on the old server as root:
on the new server you can run as root:
The -u in the options takes care of not re-copying what you already copied. That is, you can interrupt and continue at will.

The --delete option is only needed when you want to delete on the destination what has been deleted on the source. If the source is still being used and files are deleted you might want that.

Files will not be written exclusively as root but retain ownership by name. That is, files owned by foo on the old server will be owned by foo on the new server even if the uid and gid are different. Make sure all users exist on the new server. If not, the numeric uid will be retained and that will create a mess when done copying.

Permissions are retained when you are root on both source and destination.

You don't need the -e option for executing ssh on the remote end. Rsync is transfer over ssh by itself.

I am not sure about the obtainable speed. Recently I copied 1.2 TB from a 1.6 GHz Atom server over a 100 Mbit network. It took 36 hours or so. That sounds like 10 Mbyte/second.

jlinkels

1 members found this post helpful.

Thanks for the reply . Call me paranoid, but I very rarely give root access directly, even on my own servers. I also don't run SSH on a standard port ever, hence issuing a specific port in -e -p etc. Turning compression etc off gives me some improvement in performance.
I can live with the poor network performance for now as I'm hoping to only need to do this once. My main problem is rsync creating a second version of the directory (music and music?) and everything being root. The rsync parameters I've been using for a decade or two are letting me down and I can't see the wood for the trees, hence the shout for another pair of eyes

I am not giving root access to ssh either, except during migration.

OT, but I realized that if someone has guessed my password, he has root access through sudo anyway. For servers accessible from the public internet I have disabled password access, only key authentication. And it seems that it is secure if you use key authentication for root. But I still don't do it. You must have console access if you disable password access over ssh in case something goes wrong.

Changing the port only gives a tiny bit of additional security. A port scan immediately reveal the ssh port.

Back on topic. Rsync should not duplicate directories. If you happen to forget the '/' behind the source directory it will. The command line I wrote down is what I use for cloning home partitions. If I clone complete machines I exclude /proc, /sys and /mnt. But that is it.

You have that right in this line
jlinkels

1 members found this post helpful.

Oh yeah. If you have physical access or a compromised password then you own the box. It is mostly habit as I am involved in pen testing at work occasionally so have a hardened firewall and server is not accessible via the Web. My poor linux and windows servers are my guinea pigs when I'm checking for new vulnerabilities.
I can't figure out why I'm seeing the duplicate directory or permission issue. Glad it isn't something obvious.

I googled "rsync creates directory with question mark" and...

Was your script prepared in Windows? DOS-style newlines are CR-LF, while Unix-style is just LF. The CR character in Linux sometimes shows up in filenames as a question mark.

So get rid of the CR. One way is to change the file save option in your text editor.

1 members found this post helpful.

From man rsync, I see that preserving ownership requires that "the receiving rsync is being run as the super-user."

To send the files of any user, you probably need to operate as super-user also on the sending side.

It is as jlinkels wrote, "Permissions are retained when you are root on both source and destination."

1 members found this post helpful.

Thanks guys. Scripts are written and edited on the Linux box itself using vi, but I'll recheck the cr-lf just in case anyway. Never hurts to take another look. I'll also recheck the putty session settings as both servers are headless.
Script is being run as root, so permissions should carry across anyway.
Bizarrely, if the answer is PEBCAK, I'll be really happy