backup script help

cbtshare · 11-23-2009, 09:50 PM

I have my webserver with cpanel , and I have a FTP location. I wanna backup to the FTP location directly, because I dont have enough space on the server to back up first then rsync over the data, I have this script I need a little help in getting it to backup directly to the ftp location , or is there a simpler way .

Thanks

Code:

#!/bin/sh
# System + MySQL backup script
# Full backup day - Sun (rest of the day do incremental backup)
# Copyright (c) 2005-2006 nixCraft <http://www.cyberciti.biz/fb/>
# This script is licensed under GNU GPL version 2.0 or above
# Automatically generated by http://bash.cyberciti.biz/backup/wizard-ftp-script.php
# ---------------------------------------------------------------------

### System Setup ###
DIRS="/etc /home"
BACKUP=/tmp/backup.$$
NOW=$(date +"%d-%m-%Y")
INCFILE="/root/tar-inc-backup.dat"
DAY=$(date +"%a")
FULLBACKUP="Sun"

### MySQL Setup ###
MUSER="root"
MPASS="minus"
MHOST="localhost"
MYSQL="$(which mysql)"
MYSQLDUMP="$(which mysqldump)"
GZIP="$(which gzip)"

### FTP server Setup ###
FTPD="/backups/incremental"
FTPU="root"
FTPP="minus"
FTPS="192.168.1.100"
NCFTP="$(which ncftpput)"

### Other stuff ###
EMAILID="you@yourdomain.com"

### Start Backup for file system ###
[ ! -d $BACKUP ] && mkdir -p $BACKUP || :

### See if we want to make a full backup ###
if [ "$DAY" == "$FULLBACKUP" ]; then
  FTPD="/backups/full"
  FILE="fs-full-$NOW.tar.gz"
  tar -zcvf $BACKUP/$FILE $DIRS
else
  i=$(date +"%Hh%Mm%Ss")
  FILE="fs-i-$NOW-$i.tar.gz"
  tar -g $INCFILE -zcvf $BACKUP/$FILE $DIRS
fi

### Start MySQL Backup ###
# Get all databases name
DBS="$($MYSQL -u $MUSER -h $MHOST -p$MPASS -Bse 'show databases')"
for db in $DBS
do
 FILE=$BACKUP/mysql-$db.$NOW-$(date +"%T").gz
 $MYSQLDUMP -u $MUSER -h $MHOST -p$MPASS $db | $GZIP -9 > $FILE
done

### Dump backup using FTP ###
#Start FTP backup using ncftp
ncftp -u"$FTPU" -p"$FTPP" $FTPS<<EOF
mkdir $FTPD
mkdir $FTPD/$NOW
cd $FTPD/$NOW
lcd $BACKUP
mput *
quit
EOF

### Find out if ftp backup failed or not ###
if [ "$?" == "0" ]; then
 rm -f $BACKUP/*
else
 T=/tmp/backup.fail
 echo "Date: $(date)">$T
 echo "Hostname: $(hostname)" >>$T
 echo "Backup failed" >>$T
 mail  -s "BACKUP FAILED" "$EMAILID" <$T
 rm -f $T
fi

ozegoods · 11-23-2009, 10:04 PM

Can't you directly rsync the files to your storage location without any preprocessing? I used to operate an rpm package repository which I backed up daily using rsync. It doesn't take any extra file space on the sending machine to do it that way.

cbtshare · 11-23-2009, 10:52 PM

but I wanna do it incrementally

ozegoods · 11-23-2009, 11:46 PM

Use the --link-dest=DIR to create hardlinks for any files that have not changed. An exact matching file will not be transferred. If a match is not found, a basis file from one of the DIRs will be selected to try to speed up the transfer.

If DIR is a relative path, it is relative to the destination directory.

See also --compare-dest and --copy-dest.

See Incremental backups with rsync.

WhisperiN · 11-23-2009, 11:55 PM

That script seems to be Great..
I'm not geek enough to guide you..

I'll enjoy keeping an eye

cbtshare · 11-24-2009, 12:47 AM

ok so will this work?

rsync -a --delete --link-dest= /home/backup.1 /usr/backup.0/ --progress --partial -e ssh root@201.xx.xx.xx

I also see they have a simplier script, but as I said I dont have enough space, so all I would do is change the BDIR=*the remote destination*****but whats the format?

#!/bin/sh

# This script does personal backups to a rsync backup server. You will end up
# with a 7 day rotating incremental backup. The incrementals will go
# into subdirectories named after the day of the week, and the current
# full backup goes into a directory called "current"
# tridge@linuxcare.com

# directory to backup
BDIR=/home/$USER

# excludes file - this contains a wildcard pattern per line of files to exclude
EXCLUDES=$HOME/cron/excludes

# the name of the backup machine
BSERVER=owl

# your password on the backup server
export RSYNC_PASSWORD=XXXXXX

########################################################################

BACKUPDIR=`date +%A`
OPTS="--force --ignore-errors --delete-excluded --exclude-from=$EXCLUDES
--delete --backup --backup-dir=/$BACKUPDIR -a"

export PATH=$PATH:/bin:/usr/bin:/usr/local/bin

# the following line clears the last weeks incremental directory
[ -d $HOME/emptydir ] || mkdir $HOME/emptydir
rsync --delete -a $HOME/emptydir/ $BSERVER::$USER/$BACKUPDIR/
rmdir $HOME/emptydir

# now the actual transfer
rsync $OPTS $BDIR $BSERVER::$USER/current

cbtshare · 11-24-2009, 10:58 AM

is it normal also that when i do a dump of the database, even with phpadmin, my website locks up?the database is over 4gigs

ozegoods · 11-24-2009, 01:23 PM

Quote:

Originally Posted by cbtshare

ok so will this work?
rsync -a --delete --link-dest= /home/backup.1 /usr/backup.0/ --progress --partial -e ssh root@201.xx.xx.xx

I always put options on the left, file locations on the right, and I authenticated using keys in .ssh directory, not passwords.

Instead of --progress --partial you can just use P. Use z for compression and H if you have hardlinks. Using nice and --bwlimit is an attempt to allow the web server some CPU time and bandwidth during the backup. The ssh is on port 2222 in this example.

Code:

nice -n 19 rsync -azPH --stats --delete --bwlimit=80 -e 'ssh -p 2222' --exclude=pub/somefile --link-dest=../backup_0/ /home/your_directory/public_html/ user_name@your_backup_site.com:backup_1/

Quote:

Originally Posted by cbtshare

I also see they have a simpler script, but as I said I don't have enough space, so all I would do is change the BDIR=*the remote destination*****but whats the format?

Not enough space where?

Quote:

Originally Posted by cbtshare

is it normal also that when i do a dump of the database, even with phpadmin, my website locks up? The database is over 4gigs

NAME: dump - ext2/3 filesystem backup

I have no experience using dump.

The best you can do with rsync is nice the job, and limit the data rate of the backup transfer.

chrism01 · 11-24-2009, 11:19 PM

If you use mysqldump, it has to lock the DB to get a consistent view.
Actually if you have multiple schemas you can mysqldump them individually, so it only locks each schema as it goes.

cbtshare · 11-25-2009, 06:10 AM

ok thanks for the advice ozegoods, i would try nice, but dont have enough space on the server i am backing up, so laswt night i tried a cpanel backup and i chose remote server as the destination instead of home directory and it ran out of space and killed my site.I wanna backup the db directly to a remote location,there isnt space on the serverfew gigs and the db is over 4gigs.Any help there?

ozegoods · 11-25-2009, 10:11 AM

Well, that must me one micro sized server you got there. How can it hold a 4GB data base but not have room to install nice, which is only a 22,000 bytes utility? Are you sure nice is not already there? Every web host I've used has always had nice on the system as part of the basic utility package.

Code:

-rwxr-xr-x 1 root root   21988 2009-10-16 10:43 nice*

I assume you are short on server disk space, not RAM, correct?

You do have SSH access to the server file system, no?

What happens when you try to use an rsync command line to back up the data base to a remote location? It should work, rsync does not create any temporary files on the sending side, and only sometimes on the destination side depending on which options are used.

cbtshare · 11-25-2009, 04:39 PM

Sorry maybe I didnt explain correctly.I can install things, I just cant backup to the server, because the backup would require more space than is free.I have a 300GB disk space on the server and roughly 6GB disk space.The database is over 3 GB and growing daily.

If I use mysqldump the server locks up and causes downtime...(this is unacceptable) and worse it saves to the server directly, so that is out.

I tried using phpadmin , downloading a copy of the database, but the server locks up as well.

I want a command that backs up to the remote location incrementally, then I can just rsync the data itself.

cbtshare · 11-25-2009, 04:43 PM

Quote:

I assume you are short on server disk space, not RAM, correct?

Correct

Quote:

You do have SSH access to the server file system, no?

Yes I do and to the remote location or NAS.

Quote:

What happens when you try to use an rsync command line to back up the data base to a remote location?

I used Rsync to copy over data, I'm not sure maybe I could rsync the database itself.but how would I restore it and I still would need to backup incrementally.

ozegoods · 11-25-2009, 07:42 PM

Quote:

Originally Posted by cbtshare

I used Rsync to copy over data, I'm not sure maybe I could rsync the database itself, but how would I restore it? and I still would need to backup incrementally.

Well, you'd use rsync to backup the database itself, and rsync to restore the backup file, however there may be complications. The backup would have been made on a running data file, I don't know if this would cause some kind of database error when restored later. In any case the rsync backup would not have to transfer the whole data file if there is a basis file in the directory specified by the --link-dest option.

cbtshare · 11-26-2009, 05:24 PM

Ok, I'm trying to just mirror the directory.When I go :

rsync -vaz --rsh="ssh -l root" /home/you 192.168.1.1:/home/bak

it just ask for a password then nothing happens.
any help please?