LinuxQuestions.org

LinuxQuestions.org (/questions/)
-   Linux - Server (https://www.linuxquestions.org/questions/linux-server-73/)
-   -   Problems Moving RSYNC Archive - Makes No Sense (https://www.linuxquestions.org/questions/linux-server-73/problems-moving-rsync-archive-makes-no-sense-4175527870/)

MichaelGMorgan 12-10-2014 04:24 AM

Problems Moving RSYNC Archive - Makes No Sense
 
Hi,
I have a cPanel/WHM web server, which hosts a whole bunch of sites.
I then have another dedicated server which is for backups. Every 3 days, the backup server connects to the web server and grabs a copy of all the home dirs.
Here's the basics of the bash script using rsync (I've simplified it here)...

Code:

# Todays date
TODAY=`date -I`

# Previous date
PASTDATE=`date -I -d "3 day ago"`

# RSYNC the new files down from remote
sudo rsync -ach --progress --delete \
        --exclude /somedir \
        --link-dest=/home/serverbackups/WEBSERVER/files/$PASTDATE -e "ssh -i $SSHKEY" --progress \
        root@WEBSERVER:/home \
        /home/serverbackups/WEBSERVER/files/$TODAY

This works... all the date directories are there, each with all the files inside.
On the web server, the combined disk usage for /home is about 450GB.
On the backup server, disk usage is about 600GB - I figure this is about right, showing that this rsync process isn't re-creating files it already has, but is creating symlinks (which is what I want to save disk space). If this wasn't the case, the disk usage would go up by ~450GB every time it runs... which it doesn't.

So I'm switching backup servers, to something with more space (4x 2TB HDDs).
I wrote a bash script using RSYNC which would copy everything from old backup server to the new backup server, and then I ran it.

Three days later, it's still running, with a disk usage of 1.4TB.
That can't be right... disk usage should be the same as it is on the old server.
I assume it's copied the actual files and has followed the symlinks, copying them over again and again, so I cancel this, make some changes to my transfer script and start over.

Before running this script again, I thought I'd double check the source (the old backup server) to make sure all the symlinks are in place.
I "cd" into about 15 different, random directories from the old backup location, and run "ls -l".
From what I can see, there aren't any symlinks... here's an example...

Code:

drwxr-xr-x 13 558 558  4096 Sep 18 18:53 application
drwxr-xr-x  5 558 558  4096 Sep 18 18:53 assets.admin
drwxr-xr-x  6 558 558  4096 Sep 18 18:53 assets.frontend
drwxr-xr-x  2 558 558  4096 Sep  2 19:08 cgi-bin
-rw-r--r-- 14 558 558  8962 Sep 17 15:31 index.php
drwxr-xr-x  8 558 558  4096 Sep 18 18:53 system
-rw-r--r-- 14 558 558 53330 Sep 17 15:31 thumb.php
drwxr-xr-x  3 558 558  4096 Nov  4 14:55 uploads

That's the contents of a websites "public_html" directory. The files are small in this instance, however there are some which are 200MB+ each.

So... a couple of questions...

- What the heck is going on at the old backup server? I can't see any symlinks, and yet disk usage isn't going up by 450GB each time.
- When I transfer everything to the new backup server, what's the best way to do this whilst preserving the symlinks?

Any help would be greatly appreciated.
Thanks!

descendant_command 12-10-2014 05:13 AM

--link-dest creates hardlinks, not symlinks.
You need to preserve hardlinks (-H) on your "new" rsync.

MichaelGMorgan 12-10-2014 08:35 AM

Ahh, that makes sense, hardlinks never even crossed my mind.

Thanks!!


All times are GMT -5. The time now is 05:27 PM.