Help answer threads with 0 replies.
Go Back > Forums > Linux Forums > Linux - Server
User Name
Linux - Server This forum is for the discussion of Linux Software used in a server related context.


  Search this Thread
Old 06-03-2008, 08:53 PM   #1
LQ Newbie
Registered: Dec 2006
Posts: 6

Rep: Reputation: 0
Verifying Rsync Backups of Large Volumes of Files

Hi all, I'm having an issue with backing up my file server and verifying that Rsync is doing its job. I have just starting using Rsync to backup about 150gig of files (audio and visual data mainly) to another server over the gigabit network automatically every night. I was new to using Rsync, so I tried to verify whether or not this was creating a proper backup, and used du to compare sizes, but I'm finding a difference between the sizes. I can't figure out what is causing this. The same exact same number of files exist on each drive - although the backup is about 460mb larger (after last night - earlier this week it was 1gig larger).

I'm using rsync as follows:

rsync -az --delete -e ssh root@***.***.***.***:/source /destination >> $log

I can't find any issues with the command I'm using (I'm using the --delete option, so it can't be a failure of deletes to follow through from source to destination). It's pretty much impossible to compare the two directory listings between source and destination by hand (there are hundreds of thousands of files to compare). So why are the disk usage results different? The only possible explanation I can think of would be a difference between hard drives, but even then they're both using the same FS with the same block size (and they're the same capacity - although different manufacturers).

I was considering writing a script that compared line-by-line the outputs of ls, however sometimes the order of files/directories outputted by ls-lR are different between the two drives (even though the same files are in each dir), which may create false-positives.

Any ideas how I can compare the two and find out why my backups seem to be slightly larger?
Old 06-03-2008, 10:02 PM   #2
Senior Member
Registered: Oct 2007
Location: Brighton, UK
Distribution: Ubuntu Hardy, Ubuntu Jaunty, Eeebuntu, Debian, SME-Server
Posts: 1,213
Blog Entries: 1

Rep: Reputation: 83

interesting one, i have come across it before. It generally happens when the --delete option is not invoked on every backup? at least thats my experience, could be worth SFA!
Old 06-03-2008, 11:30 PM   #3
Registered: Dec 2003
Posts: 96

Rep: Reputation: 35
it may be a matter of file fragmentation or links

I use a snapshot approach - see rsync snapshot backups using cp -al and a bit of renaming. I had to use the modify-window option to handle an CIFS problem.

If you are thinking of writing scripts, I'd consider calculating md5 sums. Remasterys does this with
find . -type f -print0 | xargs -0 md5sum > md5sum.txt
(that's a script to create a bootable DVD backup for an Ubuntu system). You can run this on both systems and then analyze the resulting md5sum lists for errors.


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off

Similar Threads
Thread Thread Starter Forum Replies Last Post
rsync can't handle large files??? deweirdt Linux - Server 19 02-28-2012 02:35 PM
network backups with rsync, keep deleted files somewhere else? Telexen Linux - Server 1 09-06-2007 01:24 PM
File system for large volumes archangel_617b Linux - Enterprise 2 08-30-2007 04:06 PM
help with partitioning large volumes with LVM disorderly Linux - Server 0 08-28-2007 11:08 AM
Rsync backups gabsik Linux - General 3 11-24-2006 07:14 PM > Forums > Linux Forums > Linux - Server

All times are GMT -5. The time now is 04:28 AM.

Main Menu
Write for LQ is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
Facebook: linuxquestions Google+: linuxquestions
Open Source Consulting | Domain Registration