i have rsync running from cygwin on windows 2008r2 backing up to the correct directories on my centos7 box over the internet

but it is very very slow

i ran the command on 1 directory which is 5.7GB in size
and the list moves up file by file extremely slowly
i have an old copy of the data in the same directory im pointing
rsync to.

i quit after an hour to see if i run the command again if it would run any quicker...but its just as slow

on cygwin i ran rsync --version and it reports:
rsync version 3.1.1 protocol version 31

on my centos7 box i ran the same command and got:
rsync version 3.0.9 protocol version 30

could this have something to do with it?

i ran sudo upgrade rsync but it told me i have the latest version installed.

i have no idea what could be causing this,i googled it and it said if compression options are implemented it could take a long time,but i dont have any compression options and i also read that rsync runs incremental backups by default

i really dont have any idea what else i can try,any help would be greatly appreciated

i upgraded the rsync on my centos7 box using the following commands

wget https://download.samba.org/pub/rsync...c-3.1.1.tar.gz
tar -xf rsync-3.1.1.tar.gz
cd rsync-3.1.1
./configure
make
sudo make install

sudo rm /usr/bin/rsync
sudo ln -s /usr/local/bin/rsync /usr/bin/rsync
rsync --version

rsync now says

rsync version 3.1.1 protocol version 31 (which is the same on my cygwin terminal)

...but its running the same

i added this argument to rsync: --info=progress2
im getting between 115KB/s to 160KB/s is this normal over the internet
been reading that it could be encryption and ssh and that i should be using rsynd to rsyncd instead

any ideas?

a few things

it normally is NOT a good idea to manually install system programs like rsync from source

did you at least add a "EXCLUDE = rsync" to the yum program so that it will NOT replace the version you built with the official version in the OS repos


also i take it you are not familiar with CygWin

everything has to go through the "cygwin1.dll" this is and has been a bottleneck for many years
and slows things down greatly

why not just use normal back up tools on Cent 7.2

ok so i ran the script on one of my smaller directories,it was 2.7GB and it took hours
it completed and when i ran the command again it took seconds
made some changes and the new files copied over in seconds
think the copy i placed in the directory is a bit old...so i will recopy the data and try again

hey John VV im using cygwin cause im pushing from server 2008r2 to centos7...so you suggest i rather pull from the centos 7 box?
how would i go about doing that?

"a few things

it normally is NOT a good idea to manually install system programs like rsync from source

did you at least add a "EXCLUDE = rsync" to the yum program so that it will NOT replace the version you built with the official version in the OS repos "

no i didnt EXCLUDE = rsync...am i going to have problems now with my installation...maybe down the line

so how was i meant to update it?

thanks for the tips

on the only supported version of 7 "CentOS 7.2 "
you use yum as per all the instructions in all of the documentation

installs all the current software updates

You weren't.

Run a yum update, if there's an rsync update available in the CentOS repos it'll pull it in, if not then leave it alone unless you have an absolute driving NEED to install that EXACT version and NOTHING else due to some special feature or bug fix.

Do not go around compiling random pieces of built-in software from scratch and replacing the system default. That's a good way to screw up the system in short order.

As for the speed, you said you're transferring through the internet. The transfer speed is going to be the lesser of the upload speed of your home internet connection or the download speed of whatever connection the remote system has. Chances are your upload is going to be the bottleneck. Contact your ISP if you don't know it, or you can perform one of the numerous bandwidth tests available on the net to test it. You can also enable compression in your transfer with the "-z" flag in rsync, depending on the type of data you're sending (whether or not it's compressible) this could help or hurt.

I don't know anything about the ISPs in Ireland, but the dinkier ones here in the US often have an upload cap of around 1 Mbps, which would be around 120 kBps, which is what you're seeing. Unless you have a balls-to-the-wall internet connection, backing up hundreds of GB to remote systems through the internet is generally a bad idea.

ok john,

so if no update for a program happens through yum update ...its best left like that,as that is the current supported version in the repos

is there a way i can role back to the original version?

thanks again

but i thought rsync only uploaded new/changed files...why is it going through every file,even though i have made a copy of the data in the same directory???

It must think the files have changed. How did you copy the data to the backup drive originally?

Just within windows server I copied the directories to a USB hdd and copied into the directory on the centos box...still my transfer isn't too bad actually I started one of the main directorys Wednesday night and it finished this morning was 125gb...I hope my isp doesn't nail me...even though it's an unlimited connection though the FUP prop doesn't apply to upload.

Never thought about that I know there are tools...I prob should have used robocopy or something that keeps files identical

A click and drag copy using Windows will change all of your time stamps, that may be what's triggering it. I think rsync may have an option to switch to checksum comparison, which would take a lot of overhead to compute, but might help in this scenario.

hey suicidaleggroll

you talking about

-c, --checksum
This forces the sender to checksum all files using a 128-bit MD4 checksum
before transfer. The checksum is then explicitly checked on the receiver and
any files of the same name which already exist and have the same checksum and
size on the receiver are skipped. This option can be quite slow.

but i dont understand where the slow comes in...is it slow on actual transfer of files...or the whole operation...i dont understand