LinuxQuestions.org
Share your knowledge at the LQ Wiki.
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - Newbie
User Name
Password
Linux - Newbie This Linux forum is for members that are new to Linux.
Just starting out and have a question? If it is not in the man pages or the how-to's this is the place!

Notices


Reply
  Search this Thread
Old 07-31-2017, 08:39 AM   #1
funkytwig
Member
 
Registered: Jun 2016
Posts: 46

Rep: Reputation: Disabled
Odd rsync behaviour (Ubuntu)


I am running 2 rsync commands , one after the other.

The first does a rsync over ssh to sync between a owncloud server and a backup server:

rsync -v -a -H -i -L -e ssh --delete --link-dest=/mnt/backup01/sun/var_www_owncloud/sun_oc_9999999999_current user@domain.org:/var/www/owncloud /mnt/backup01/sun/var_www_owncloud/sun_oc_1501355550_F_Sat_29Jul2017_2012

I then user rsync again to sync between the main backup drive /mnt/backup01/ and the second backup /mnt/backup02/

rsync -aviH --delete /mnt/backup01/ /mnt/backup02/

Thing is when the first one changes verry little

07/29/17_20:12 backup(8517) receiving incremental file list
07/29/17_20:12 backup(8517) created directory /mnt/backup01/sun/var_www_owncloud/sun_oc_1501355550_F_Sat_29Jul2017_2012
07/29/17_20:12 backup(8517) cd..t...... owncloud/database_backup/
07/29/17_20:12 backup(8517) >f+++++++++ owncloud/database_backup/owncloud-sqlbkp_20170729_1955.sql.gz
07/29/17_20:12 backup(8517) >f+++++++++ owncloud/database_backup/owncloud-sqlbkp_20170729_2010.sql.gz
07/29/17_20:12 backup(8517)
07/29/17_20:12 backup(8517) sent 11,815 bytes received 20,381,648 bytes 1,235,967.45 bytes/sec
07/29/17_20:12 backup(8517) total size is 412,002,742,657 speedup is 20,202.69
/mnt/backup01/sun/var_www_owncloud/sun_oc_9999999999_current
07/29/17_20:12 backup(8517)

The second is deleting everything and then adding it back.

Any idea how to fix this?
 
Old 07-31-2017, 09:10 AM   #2
IsaacKuo
Senior Member
 
Registered: Apr 2004
Location: Baton Rouge, Louisiana, USA
Distribution: Debian 9 Stretch
Posts: 2,393
Blog Entries: 8

Rep: Reputation: 411Reputation: 411Reputation: 411Reputation: 411Reputation: 411
What file system is /mnt/backup02/ using? If it's something like NTFS, you'll need to introduce a "modify-window" because it can't save date-time with the precision of ext4 or other *nix file systems.

So, you'd change from this:
Code:
sync -aviH --delete /mnt/backup01/ /mnt/backup02/
to this:
Code:
sync -aviH --delete --modify-window=5 /mnt/backup01/ /mnt/backup02/
Honestly, I don't know exactly what iH flags do. My usual thing if doing a backup to NTFS is this:

Code:
sync -tvlr --delete --modify-window=5 /mnt/backup01/ /mnt/backup02/
These flags are:
-t preserve modification times
-v verbose feedback
-l copy links as symlinks
-r recursive (going through the full directory tree)
--delete delete files that don't exist in the source
--modify-window=5 consider date-time within 5 seconds to be good enough (to compensate for NTFS limitations)

Note that the only file attributes this attempts to replicate are modification times and saving links as symlinks. I think that if the target file system is NTFS, it's more trouble than it's worth to attempt to replicate more than that.
 
Old 07-31-2017, 09:14 AM   #3
funkytwig
Member
 
Registered: Jun 2016
Posts: 46

Original Poster
Rep: Reputation: Disabled
both ext4 (conecxted with usb3).

-H is preserve hard links
-i gives info (thats why I havew '>f+++++++++' above)

Last edited by funkytwig; 07-31-2017 at 09:16 AM.
 
Old 07-31-2017, 09:28 AM   #4
IsaacKuo
Senior Member
 
Registered: Apr 2004
Location: Baton Rouge, Louisiana, USA
Distribution: Debian 9 Stretch
Posts: 2,393
Blog Entries: 8

Rep: Reputation: 411Reputation: 411Reputation: 411Reputation: 411Reputation: 411
Then my guess is that it has something to do with hardlinks, but I'm not familiar with using them and rsync to know what could be going on. I only use symlinks.
 
Old 07-31-2017, 10:02 AM   #5
rknichols
Senior Member
 
Registered: Aug 2009
Distribution: CentOS
Posts: 4,547

Rep: Reputation: 2083Reputation: 2083Reputation: 2083Reputation: 2083Reputation: 2083Reputation: 2083Reputation: 2083Reputation: 2083Reputation: 2083Reputation: 2083Reputation: 2083
Does the "-i" output for that second run not give any clues about what is going on? You've shown the output for just the first (successful) run, which isn't very helpful. Just the output related to one or two of the affected files is sufficient.
 
Old 07-31-2017, 10:21 AM   #6
funkytwig
Member
 
Registered: Jun 2016
Posts: 46

Original Poster
Rep: Reputation: Disabled
Quote:
Originally Posted by rknichols View Post
Does the "-i" output for that second run not give any clues about what is going on? You've shown the output for just the first (successful) run, which isn't very helpful. Just the output related to one or two of the affected files is sufficient.
...
07/30/17_04:05 mirror_drives(10108) rsync -aviH --delete /mnt/backup01/ /mnt/backup02/
07/30/17_04:05 mirror_drives(10108) sending incremental file list
07/30/17_04:05 mirror_drives(10108) *deleting sun/var_www_owncloud/sun_oc_1501345801_H_Sat_29Jul2017_1730/owncloud/updater/vendor/symfony/process/Pipes/WindowsPipes.php
07/30/17_04:05 mirror_drives(10108) *deleting sun/var_www_owncloud/sun_oc_1501345801_H_Sat_29Jul2017_1730/owncloud/updater/vendor/symfony/process/Pipes/UnixPipes.php
07/30/17_04:05 mirror_drives(10108) *deleting sun/var_www_owncloud/sun_oc_1501345801_H_Sat_29Jul2017_1730/owncloud/updater/vendor/symfony/process/Pipes/PipesInterface.php
07/30/17_04:05 mirror_drives(10108) *deleting sun/var_www_owncloud/sun_oc_1501345801_H_Sat_29Jul2017_1730/owncloud/updater/vendor/symfony/process/Pipes/AbstractPipes.php
07/30/17_04:05 mirror_drives(10108) *deleting sun/var_www_owncloud/sun_oc_1501345801_H_Sat_29Jul2017_1730/owncloud/updater/vendor/symfony/process/Pipes/
07/30/17_04:05 mirror_drives(10108) *deleting sun/var_www_owncloud/sun_oc_1501345801_H_Sat_29Jul2017_1730/owncloud/updater/vendor/symfony/process/Exception/RuntimeException.php
07/30/17_04:05 mirror_drives(10108) *deleting sun/var_www_owncloud/sun_oc_1501345801_H_Sat_29Jul2017_1730/owncloud/updater/vendor/symfony/process/Exception/ProcessTimedOutException.php
07/30/17_04:05 mirror_drives(10108) *deleting sun/var_www_owncloud/sun_oc_1501345801_H_Sat_29Jul2017_1730/owncloud/updater/vendor/symfony/process/Exception/ProcessFailedException.php
07/30/17_04:05 mirror_drives(10108) *deleting sun/var_www_owncloud/sun_oc_1501345801_H_Sat_29Jul2017_1730/owncloud/updater/vendor/symfony/process/Exception/LogicException.php
07/30/17_04:05 mirror_drives(10108) *deleting sun/var_www_owncloud/sun_oc_1501345801_H_Sat_29Jul2017_1730/owncloud/updater/vendor/symfony/process/Exception/InvalidArgumentException.php
07/30/17_04:05 mirror_drives(10108) *deleting sun/var_www_owncloud/sun_oc_1501345801_H_Sat_29Jul2017_1730/owncloud/updater/vendor/symfony/process/Exception/ExceptionInterface.php
07/30/17_04:05 mirror_drives(10108) *deleting sun/var_www_owncloud/sun_oc_1501345801_H_Sat_29Jul2017_1730/owncloud/updater/vendor/symfony/process/Exception/
07/30/17_04:05 mirror_drives(10108) *deleting sun/var_www_owncloud/sun_oc_1501345801_H_Sat_29Jul2017_1730/owncloud/updater/vendor/symfony/process/phpunit.xml.dist
07/30/17_04:05 mirror_drives(10108) *deleting sun/var_www_owncloud/sun_oc_1501345801_H_Sat_29Jul2017_1730/owncloud/updater/vendor/symfony/process/composer.json
07/30/17_04:05 mirror_drives(10108) *deleting sun/var_www_owncloud/sun_oc_1501345801_H_Sat_29Jul2017_1730/owncloud/updater/vendor/symfony/process/README.md
07/30/17_04:05 mirror_drives(10108) *deleting sun/var_www_owncloud/sun_oc_1501345801_H_Sat_29Jul2017_1730/owncloud/updater/vendor/symfony/process/ProcessUtils.php
0
...
07/30/17_04:05 mirror_drives(10108) *deleting sun/var_www_owncloud/sun_oc_1501174201_H_Thu_27Jul2017_1750/owncloud/.user.ini
07/30/17_04:05 mirror_drives(10108) *deleting sun/var_www_owncloud/sun_oc_1501174201_H_Thu_27Jul2017_1750/owncloud/.htaccess
07/30/17_04:05 mirror_drives(10108) *deleting sun/var_www_owncloud/sun_oc_1501174201_H_Thu_27Jul2017_1750/owncloud/
07/30/17_04:05 mirror_drives(10108) *deleting sun/var_www_owncloud/sun_oc_1501174201_H_Thu_27Jul2017_1750/
07/30/17_04:05 mirror_drives(10108) .d..t...... sun/var_www_owncloud/
07/30/17_04:05 mirror_drives(10108) cLc.t...... sun/var_www_owncloud/sun_oc_9999999999_current -> /mnt/backup01/sun/var_www_owncloud/sun_oc_1501383908_F_Sun_30Jul2017_0405
07/30/17_04:05 mirror_drives(10108) cd+++++++++ sun/var_www_owncloud/sun_oc_1501345801_D_Sat_29Jul2017_1730/
07/30/17_04:05 mirror_drives(10108) cd+++++++++ sun/var_www_owncloud/sun_oc_1501345801_D_Sat_29Jul2017_1730/owncloud/
07/30/17_04:05 mirror_drives(10108) hf+++++++++ sun/var_www_owncloud/sun_oc_1501345801_D_Sat_29Jul2017_1730/owncloud/.htaccess => sun/var_www_owncloud/sun_oc_1501177501_H_Thu_27Jul2017_1845/owncloud/.htaccess
07/30/17_04:05 mirror_drives(10108) hf+++++++++ sun/var_www_owncloud/sun_oc_1501345801_D_Sat_29Jul2017_1730/owncloud/.user.ini => sun/var_www_owncloud/sun_oc_1501177501_H_Thu_27Jul2017_1845/owncloud/.user.ini
07/30/17_04:05 mirror_drives(10108) hf+++++++++ sun/var_www_owncloud/sun_oc_1501345801_D_Sat_29Jul2017_1730/owncloud/AUTHORS => sun/var_www_owncloud/sun_oc_1501177501_H_Thu_27Jul2017_1845/owncloud/AUTHORS
07/30/17_04:05 mirror_drives(10108) hf+++++++++ sun/var_www_owncloud/sun_oc_1501345801_D_Sat_29Jul2017_1730/owncloud/COPYING-AGPL => sun/var_www_owncloud/sun_oc_1501177501_H_Thu_27Jul2017_1845/owncloud/COPYING-AGPL
07/30/17_04:05 mirror_drives(10108) hf+++++++++ sun/var_www_owncloud/sun_oc_1501345801_D_Sat_29Jul2017_1730/owncloud/console.php => sun/var_www_owncloud/sun_oc_1501177501_H_Thu_27Jul2017_1845/owncloud/console.php
07/30/17_04:05 mirror_drives(10108) hf+++++++++ sun/var_www_owncloud/sun_oc_1501345801_D_Sat_29Jul2017_1730/owncloud/cron.php => sun/var_www_owncloud/sun_oc_1501177501_H_Thu_27Jul2017_1845/owncloud/cron.php
07/30/17_04:05 mirror_drives(10108) hf+++++++++ sun/var_www_owncloud/sun_oc_1501345801_D_Sat_29Jul2017_1730/owncloud/db_structure.xml => sun/var_www_owncloud/sun_oc_1501177501_H_Thu_27Jul2017_1845/owncloud/db_structure.xml
07/30/17_04:05 mirror_drives(10108) hf+++++++++ sun/var_www_owncloud/sun_oc_1501345801_D_Sat_29Jul2017_1730/owncloud/index.html => sun/var_www_owncloud/sun_oc_1501177501_H_Thu_27Jul2017_1845/owncloud/index.html
07/30/17_04:05 mirror_drives(10108) hf+++++++++ sun/var_www_owncloud/sun_oc_1501345801_D_Sat_29Jul2017_1730/owncloud/index.php => sun/var_www_owncloud/sun_oc_1501177501_H_Thu_27Jul2017_1845/owncloud/index.php
07/30/17_04:05 mirror_drives(10108) hf+++++++++ sun/var_www_owncloud/sun_oc_1501345801_D_Sat_29Jul2017_1730/owncloud/occ => sun/var_www_owncloud/sun_oc_1501177501_H_Thu_27Jul2017_1845/ownclo
...
07/30/17_04:05 mirror_drives(10108) cd+++++++++ sun/var_www_owncloud/sun_oc_1501383908_F_Sun_30Jul2017_0405/owncloud/updater/vendor/symfony/process/Pipes/
07/30/17_04:05 mirror_drives(10108) hf+++++++++ sun/var_www_owncloud/sun_oc_1501383908_F_Sun_30Jul2017_0405/owncloud/updater/vendor/symfony/process/Pipes/AbstractPipes.php => sun/var_www_owncloud/sun_oc_1501177501_H_Thu_27Jul2017_1845/owncloud/updater/vendor/symfony/process/Pipes/AbstractPipes.php
07/30/17_04:05 mirror_drives(10108) hf+++++++++ sun/var_www_owncloud/sun_oc_1501383908_F_Sun_30Jul2017_0405/owncloud/updater/vendor/symfony/process/Pipes/PipesInterface.php => sun/var_www_owncloud/sun_oc_1501177501_H_Thu_27Jul2017_1845/owncloud/updater/vendor/symfony/process/Pipes/PipesInterface.php
07/30/17_04:05 mirror_drives(10108) hf+++++++++ sun/var_www_owncloud/sun_oc_1501383908_F_Sun_30Jul2017_0405/owncloud/updater/vendor/symfony/process/Pipes/UnixPipes.php => sun/var_www_owncloud/sun_oc_1501177501_H_Thu_27Jul2017_1845/owncloud/updater/vendor/symfony/process/Pipes/UnixPipes.php
07/30/17_04:05 mirror_drives(10108) hf+++++++++ sun/var_www_owncloud/sun_oc_1501383908_F_Sun_30Jul2017_0405/owncloud/updater/vendor/symfony/process/Pipes/WindowsPipes.php => sun/var_www_owncloud/sun_oc_1501177501_H_Thu_27Jul2017_1845/owncloud/updater/vendor/symfony/process/Pipes/WindowsPipes.php
07/30/17_04:05 mirror_drives(10108)
07/30/17_04:05 mirror_drives(10108) sent 225,125,102 bytes received 49,724,774 bytes 216,673.14 bytes/sec
07/30/17_04:05 mirror_drives(10108) total size is 13,606,976,610,805 speedup is 49,506.94

Ben
 
Old 07-31-2017, 11:13 AM   #7
funkytwig
Member
 
Registered: Jun 2016
Posts: 46

Original Poster
Rep: Reputation: Disabled
It may be to do with this (output from mount)

/dev/sdb1 on /mnt/backup01 type ext4 (rw,relatime,stripe=8191,data=ordered)
/dev/sdc1 on /mnt/backup02 type ext4 (rw,relatime,data=ordered)

Ben
 
Old 07-31-2017, 12:49 PM   #8
rknichols
Senior Member
 
Registered: Aug 2009
Distribution: CentOS
Posts: 4,547

Rep: Reputation: 2083Reputation: 2083Reputation: 2083Reputation: 2083Reputation: 2083Reputation: 2083Reputation: 2083Reputation: 2083Reputation: 2083Reputation: 2083Reputation: 2083
The files being sent are not in the same directory as the ones being deleted. It appears that a new directory "sun/var_www_owncloud/sun_oc_1501345801_D_Sat_29Jul2017_1730/owncloud/" is being created at the destination and then multiple hard links are created in that directory. The files themselves are not being transferred -- the hard links are to existing files in other directories. The same then happens for directory "sun/var_www_owncloud/sun_oc_1501383908_F_Sun_30Jul2017_0405/owncloud/updater/vendor/symfony/process/Pipes/".

None of that is in the directory tree "sun/var_www_owncloud/sun_oc_1501174201_H_Thu_27Jul2017_1750/" that is being deleted.
 
Old 07-31-2017, 05:48 PM   #9
funkytwig
Member
 
Registered: Jun 2016
Posts: 46

Original Poster
Rep: Reputation: Disabled
Thanks, ive gpot to the bottom of it. My rsync comand was not quite right (I had ot add /. to the source. also the devices for the drives swaped when I rebootid, I am now using UUID.
 
  


Reply


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off



Similar Threads
Thread Thread Starter Forum Replies Last Post
[SOLVED] Ubuntu 15.04 odd behaviour at login Mark_667 Ubuntu 1 06-11-2015 04:31 AM
[SOLVED] Odd punctuation behaviour S-14.0 business_kid Slackware 2 11-17-2013 09:05 AM
gmp: odd behaviour. stf92 Slackware 4 11-13-2013 01:47 PM
Random, very odd behaviour crashing rsync switch10 Linux - Desktop 4 04-14-2010 01:19 AM
Very odd CPAN behaviour! bluehz Linux - Software 0 07-08-2002 04:59 PM

LinuxQuestions.org > Forums > Linux Forums > Linux - Newbie

All times are GMT -5. The time now is 07:19 AM.

Main Menu
Advertisement
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
Open Source Consulting | Domain Registration