LinuxQuestions.org
Latest LQ Deal: Latest LQ Deals
Home Forums Tutorials Articles Register
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - General
User Name
Password
Linux - General This Linux forum is for general Linux questions and discussion.
If it is Linux Related and doesn't seem to fit in any other forum then this is the place.

Notices


Reply
  Search this Thread
Old 04-15-2010, 01:57 AM   #1
WildDrake!
Member
 
Registered: Dec 2003
Location: Current Location: Colorado
Distribution: Ubuntu 14.10, Mint 17.1 Cinnamon and Mate
Posts: 101

Rep: Reputation: 15
Smile What could add 30 GB to my backup?


I am running Ubuntu 9.10 Gnome

I have twin hard drives in my system. I use rsync to automatically back up my home user directory on the first drive to the second drive everyday. This has been working well for almost a year. My first drive is 85% full, but suddenly, my second drive is 100% full...in fact, rsync errored out having filled the second drive.

I erased the second drive, and using nautilus I copied the section of my first drive that I back up to the second drive (minus a few large files). When it was finished, the second drive was 99% full....again!

As near as I can tell, my backup has about 30 GB of extra information, but I have been unable to determine what the extra data is!!!

I am no longer able to do backups until this gets resolved. Does anyone have any idea what this could be, and how to discover where it is coming from? So that it can be rectified?

I thought maybe some link had somehow gotten created and my backups are following the link and copying the same info twice or something, but I haven't been able to figure out how to list all my links in my file structure on some kind of search.

Thanks!
 
Old 04-15-2010, 02:37 AM   #2
Guttorm
Senior Member
 
Registered: Dec 2003
Location: Trondheim, Norway
Distribution: Debian and Ubuntu
Posts: 1,453

Rep: Reputation: 446Reputation: 446Reputation: 446Reputation: 446Reputation: 446
Hi

Can you post the rsync command you are using?

Also here are some ideas:

- If you have very big files that changes a little bit, the rsync will write a copy of the file, delete the original and then rename the copy. That's safer in case it suddenly errors out some way. You can use the option --inplace with the rsync and it will overwrite directly.

- Add --sparse to the rsync command, otherwise sparse files will use a lot more space on the backup.

- Add -H and -l for it to copy hard and soft links as links.
 
Old 04-15-2010, 03:13 AM   #3
bakdong
Member
 
Registered: Apr 2009
Posts: 214

Rep: Reputation: 44
You can use du --max-depth=1 -h to compare two file trees. You should be able to narrow down where the problem is, assuming it's extra files that you're duplicating.
 
Old 04-15-2010, 11:47 AM   #4
WildDrake!
Member
 
Registered: Dec 2003
Location: Current Location: Colorado
Distribution: Ubuntu 14.10, Mint 17.1 Cinnamon and Mate
Posts: 101

Original Poster
Rep: Reputation: 15
Thanks Guttorm!

But, I am pretty sure it is not rsync doing this, since I erased my backup drive and using the gui of Nautilus, told it to copy my /home/user directory over onto the backup drive and ended up with the same oversized outcome. So, this wouldn't appear to be because of rsync making some temporary files on my backup drive.

All the tests I have tried to discover what is causing this on my primary drive, such as "du -k | sort -n" and an attempt at the compare of the two drives trying to use diff (although I probably didn't use it correctly), didn't disclose to me where that extra 30 gig is coming from. I was looking for some way to do a recursive search of my /home/user directory to list all links, but haven't found such a thing yet.

On the other hand, I can certainly use some help on my rsync program. It is a program I found on the Internet and adapted from script found on the rsync.samba.org written originally by Brian Hone 3/24/2002. I will paste a copy on its own entry just after this message. I still have one bug that I know of in the program, as it ignores my excludes file and copies them to the backup drive anyway. And I am planning on adding your suggestions also. Any suggestions on it would be appreciated.

Program to follow...
 
Old 04-15-2010, 11:50 AM   #5
WildDrake!
Member
 
Registered: Dec 2003
Location: Current Location: Colorado
Distribution: Ubuntu 14.10, Mint 17.1 Cinnamon and Mate
Posts: 101

Original Poster
Rep: Reputation: 15
#!/bin/sh

#########################################################
# Script to do incremental rsync backups - locally
#
# Adapted from script found on the rsync.samba.org
# Brian Hone 3/24/2002
#
# Debugged, Corrected, and modified to keep an exact copy
# of the source drive on a second drive rather than creating
# an additonal directory named "main" or incremental directories, and
# set up to use pmount to auto mount 2nd drive without root access by
# Terry L. Kemmerer 2009/06/30
# This script is freely distributed under the GPL

# NOTE: I have # out some of the functions which I did not want to use
# such as "main" and incremental backup. If you want them back, just
# follow the program flow and remove the #'s where applicable. -Terry
#########################################################

##################################
# Configure These Options
##################################

##################################
# For Safety and Security Purposes, Backup Drive is not mounted
# This portion requires pmount and libpmount0.0 to be installed
# Edit-add to /etc/pmount.allow = /dev/sdb3
# http://ubuntuforums.org/archive/index.php/t-980122.html
# Mount backup drive

pmount /dev/sdb3 /media/disk

###################################
# mail address for status updates
# - This is used to email you a status report
###################################
MAILADDR="wildcard@plains.net"

###################################
# HOSTNAME
# - This is also used for reporting
###################################
HOSTNAME="ispy"

###################################
# directory to backup
# - This is the path to the directory you want to archive
###################################
BACKUPDIR="/home/lucky/"

###################################
# excludes file - contains one wildcard pattern per line of files to exclude
# - This is a rsync exclude file. See the rsync man page and/or the
# example_exclude_file
###################################
# EXCLUDES=example_exclude_file : create this txt file in the same dir as this script

# the below url tells how to set up and use this file
# http://articles.slicehost.com/2007/1...es-and-folders

EXCLUDES="/home/lucky/scripts/excluded.txt"

###################################
# root directory to backup stuff
###################################
ARCHIVEROOT="/media/disk/lucky"

#########################################
# From here on out, you probably don't #
# want to change anything unless you #
# know what you're doing. #
#########################################

# directory which holds our current datastore
CURRENT=main

# directory which we save incremental changes to
INCREMENTDIR=`date +%Y-%m-%d` # This 'date' file will be written to ARCHIVEROOT

# options to pass to rsync
OPTIONS="--force --ignore-errors --delete --delete-excluded \
--exclude-from=$EXCLUDES -av"
# used to be: --exclude-from=$EXCLUDES --backup --backup-dir=$ARCHIVEROOT/$INCREMENTDIR -av"
export PATH=$PATH:/bin:/usr/bin:/usr/local/bin

# make sure our backup tree exists
install -d $ARCHIVEROOT #/$CURRENT

# our actual rsyncing function
do_rsync()
{
rsync $OPTIONS $BACKUPDIR $ARCHIVEROOT # /$CURRENT
}

# our post rsync accounting function
do_accounting()
{
echo "Backup Accounting for Day $INCREMENTDIR on $HOSTNAME:">/tmp/rsync_script_tmpfile
echo >> /tmp/rsync_script_tmpfile
echo "################################################">>/tmp/rsync_script_tmpfile
du -s $ARCHIVEROOT/* >> /tmp/rsync_script_tmpfile
echo "Mail $MAILADDR -s $HOSTNAME Backup Report < /tmp/rsync_script_tmpfile"
mail $MAILADDR -s $HOSTNAME Backup Report < /tmp/rsync_script_tmpfile
echo "rm /tmp/rsync_script_tmpfile"
rm /tmp/rsync_script_tmpfile
}

# some error handling and/or run our backup and accounting
if [ -f $EXCLUDES ]; then
if [ -d $BACKUPDIR ]; then
# now the actual transfer
do_rsync && do_accounting
else
echo "cant find $BACKUPDIR"; exit
fi
else
echo "cant find $EXCLUDES"; exit
fi
##################################
# Un-Mount backup drive

pumount /dev/sdb3

###################################
 
Old 04-16-2010, 12:40 AM   #6
WildDrake!
Member
 
Registered: Dec 2003
Location: Current Location: Colorado
Distribution: Ubuntu 14.10, Mint 17.1 Cinnamon and Mate
Posts: 101

Original Poster
Rep: Reputation: 15
bakdong -- your du --max-depth=1 -h really simplified and helped. This was not a link problem. Instead, it turned out to be a "reserved space" problem. I use ktorrent and often have several things loaded up in it to be downloaded, except I may have turned them off, figuring to do the downloading in the future some time, so they are just setting in the listing. Also, I have turned off ktorrent's ability to "reserve space" for where those items will eventually be downloaded to on the hard drive, so as not to tie up my storage space. BUT FOR SOME REASON, whether I use rsync or simply do a manual copy of my file tree to the backup drive, the reserve space IS BEING ALLOTTED on the backup hard drive.....yet it is not present on the primary drive. Go figure! So I now have a query into ktorrent to find out what is going on.

At any rate, this mystery is solved. Thanks to everyone!
 
Old 04-16-2010, 02:11 PM   #7
WildDrake!
Member
 
Registered: Dec 2003
Location: Current Location: Colorado
Distribution: Ubuntu 14.10, Mint 17.1 Cinnamon and Mate
Posts: 101

Original Poster
Rep: Reputation: 15
The reserve space is what rsync calls "sparse files" and with the option -S or --sparse added, (which Guttorm suggested above) rsync will not back up sparse or reserve areas and waste space on the backup drive. So, that completes the solution.

Thanks!
 
  


Reply



Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off



Similar Threads
Thread Thread Starter Forum Replies Last Post
Newbie trying to write a simple backup script to backup a single folder Nd for school stryker759a Linux - Newbie 2 09-16-2009 08:52 AM
How do you backup Firefox bookmarks history and add ons jgz Linux - Software 11 09-22-2008 09:20 PM
LXer: Automatic backup for sporadically connected clients with Box Backup LXer Syndicated Linux News 0 08-29-2008 08:40 PM
Backup system ala rdiff-backup, but without mirror and with dst encryption dr_dex Linux - Software 0 08-04-2008 03:39 AM
To add additional HDDs for network backup krishna_kawhale Linux - Hardware 1 08-08-2006 09:25 PM

LinuxQuestions.org > Forums > Linux Forums > Linux - General

All times are GMT -5. The time now is 08:58 PM.

Main Menu
Advertisement
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
Open Source Consulting | Domain Registration