Hi there!

I am currently using the latest Ubuntu Jaunty V9 and I have to say I love it! Down with Doze! lol

I am currently trying to write a backup script in bash which does the following:

1) Takes a load of files and creates a tar file

2) Splits the tar file into 1.0GB Chunks and stores them in another directory.

3) Arranges the split files into numbered directories where the file size does not exceed the total capacity of a DVD5, i.e 4.3gig.

4) Create ISOs from the numbered directories and then burn them to discs in a loop.

I have figured out how to do steps 1, 2, 4, but i'm stumped on no.3.

What it would need to do is say if we had 7gig worth of data, this would equate to 7 1gig files, the first 4 would need to go into a directory like /backup/1/ and the other 3 would need to go into a directory like /backup/2/

I would need it to be relative to the amount of data being backed up, as say 26gig would be 26 1gig files so this would be 6.5 discs, so 7 discs would be needed. How could I do this within a bash script?

Can anyone help me out with this? Any help would be appreciated. Once I have the complete backup script working I am prepared to share it so others could use it!

Thanks In Advance,

J.

You could use dar instead of tar. Dar will do all the file splitting for you without making the temporary backup directories.

http://sourceforge.net/projects/dar/

Dar is also in the Ubuntu package repository.

Dar bases its capacity calculation on the compressed file sizes and it can write directly to DVD or CD.

jasybee2000,

How did your script turn out?

How about posting it in your blog or on a pastebin & link to it here?

Hi,

I did manage to work it out but it was a total nightmare. I am prepared to share my script with you, but be warned it is a quick and dirty solution as I was under pressure by a friend of mine to provide a quick no frills backup solution, so I had no time in the end to fine tune it lol
I will provide it here and explain how it works with my setup so you should be able to get an idea of what I did and tailor it and improve it for your own needs and requirements. If you do manage to improve it then I would appreciate it if you would share the improvements with myself and others on this forum as i'm sure this is bound to come in very handy for a lot of people.

I have tested this on Ubuntu Server V9 (Jaunty) with the latest updates. This runs under bash and requires the following packages I believe:
cdrecord and associated tools, dvdrw+- tools, and a sound program called beep.

I used this with the following tree setup:
(the ones in caps are directories to store the files at various stages)
the two text files are purely to signal the burnready flag and disccount is for the number of dvds it will burn.

/backup
ARC
burnready.txt
disccount.txt
FILES
ISO
LOG
PART

Here is the script - Be warned it is a bit messy. Use for educational purposes :-

#!/bin/bash

date=`date "+%d%m%y"`

dcntfile="/backup/disccount.txt"
burnready="/backup/burnready.txt"
burnstr=""

findpatt="* *"
arcsource="/home/"
bpath="/backup"
bfile="backup_$date.tar"
outfile="$bpath/ARC/$bfile"
splitf=".part."
outpart="$bpath/PART/$bfile$splitf"
lsource="$bpath/PART/"
ldest="$bpath/FILES/"
cpath="$bpath/"
fisopath="$bpath/ISO/*"

brksize="1024m"
lcmd="ls -lS"

lpath=$lsource
dpath=$ldest

logfile="$bpath/LOG/logfile_$date.log"
out="1>>$logfile 2>>$logfile"

totalsize=0
numfiles=0
discnum=0
currdisc=1

dvdsizelimit=4586993664
blankdiscstr=" Disc status: blank"
chk=""
val=0
val2=0
val3=0
currentdisc=1
exitcond=2
beepfreq=2000
beeppause=.2

burnpid=0;
burnpid1=0
burnpid2=0
burncmd="/usr/bin/growisofs"
wrtspeed="8"
dvdspeed="-speed=$wrtspeed"
dvdopts="-dvd-compat"
ext=".iso"

filepath=$ldest
isopath="$bpath/ISO/"
isoopts="-r -R -J -l -L"
volprefix="_OF_"

cleanup="/backup/ARC/*.tar"
findpath=$lpath
excludes="--exclude=/home/lost+found --exclude=/home/docserver --exclude=/home/global/scratch"

#testing
farray=( "one 1" "two2" "three 3" "four 4" "five 5" )

# write to log
function tolog
{
echo $1 >> $logfile
}

# archive a folder as a tar file
function archive ()
{
tar -cvpf $outfile $arcsource $excludes 1>>$logfile 2>>$logfile
}

# break up the archive
function breakup ()
{
split -b $brksize -d $outfile $outpart 1>>$logfile 2>>$logfile
}

# loops a number of beeps
function beeploop ()
{
# args, 1 = number of beeps, 2 = beep frequency, 3 = delay between beeps (in seconds)
if [ $1 > 0 ]; then
for ((i=1;i<=$1;i+=1)); do
sleep $3
beep -f $2
done
fi
}

# populate array with data and preserve the spaces
function FileArray ()
{
y=$#
for (( x=0; x<y; x++ )); do
LIST[x]="$1"
shift
done

tolog "Array size is ${#LIST[*]}"

for (( A=0; A<${#LIST[*]}; A++ )); do
tolog "$A = ${LIST[$A]}"
done
}

# gets the sizes of the files already listed in the array
function getFileSpecs ()
{
c=0;
cd $lpath
for file in `$lcmd ${file_list}`; do
if [ -f "${file}" ]; then
farray[$c]="$file"
tolog $file
file_size=`$lcmd $file | awk '{print $5}'`
size=`expr ${size} + ${file_size}`
let farray_file_size[$c]=$file_size
let disc_select[$c]=1
let c=$c+1;
fi
done

# total size of files and number of files
let totalsize=size
let numfiles=c
}

# decides on which discs to store the files using indexing information
function arrangefiles ()
{
chksize=0;
arraysize=${#LIST[@]}
for ((i=0; i<$arraysize; i++))
do
let chksize=$chksize+${farray_file_size[$i]}
if [ "$chksize" -gt "$dvdsizelimit" ]; then
let chksize=${farray_file_size[$i]}
let currdisc=$currdisc+1
fi
let disc_select[$i]=$currdisc
done
echo $currdisc > $dcntfile
}

# shows us how the files will be organised
function showfileloc ()
{
for ((i=0; i<${#LIST[@]}; i++))
do
tolog "${LIST[$i]} ,${farray_file_size[$i]}, ${disc_select[$i]}"
done
}

function createdirs ()
{
let cnt=`cat $dcntfile`
let cnt=$cnt+1
for ((i=1; i<$cnt; i++))
do
mkdir -p "$dpath/$i"
done
}

#move files to appropriate location folders
function movefiles ()
{
for ((i=0; i<${#LIST[@]}; i++))
do
mv "$lpath${LIST[$i]}" "$dpath${disc_select[$i]}/${LIST[$i]}"
tolog "$lpath${LIST[$i]} - $dpath${disc_select[$i]}/${LIST[$i]}"
tolog "${LIST[$i]} ,${farray_file_size[$i]}, ${disc_select[$i]}"
done
}

# loop until we have detected 2 blank dvds in the system

function checkdvd ()
{
let val3=0

while [ $val3 -ne $exitcond ]
do
checkdrive dvd
let val1=$val

checkdrive dvd1
let val2=$val

let val3=$val1+$val2;

#echo $val3

if [ $val3 -ne $exitcond ]
then
beeploop $currentdisc $beepfreq $beeppause
fi
done
}

# check to see if disc exists in drive
function checkdrive ()
{
chk=`dvd+rw-mediainfo /dev/$1 2>&1 | grep "$blankdiscstr"`

sleep 1

if [ "$chk" = "$blankdiscstr" ]
then
val=1
else
val=0
fi

#echo $val

sleep 1
}

# debug
function showoutput ()
{
echo
}

# write the ISO file to disc
function burntodvd ()
{
deviceid=$1
isofile=$2

burnstr="$burncmd -dry-run $dvdopts -Z /dev/$deviceid=$isofile $dvdspeed >>$logfile 2>&1"

#echo $burnstr
}

# burn a load of ISO files numbered 1.iso, etc
function burniso ()
{
let numdvds=`cat $dcntfile`
let nums=$numdvds+1

for ((id=1; id<$nums; id++))
do
sleep 1
checkdvd

sleep 1
input="$isopath$id$ext"
tolog "burning disc # $id to dvd unit 0"

#deprecated
#burntodvd "dvd" $input
#$burncmd $dvdopts -Z /dev/dvd=$input $dvdspeed >>$logfile 2>&1 &

cdrecord -v -eject speed=$wrtspeed dev=2,0,0 /backup/ISO/$id.iso >>$logfile 2>&1

sleep 1
tolog "burning disc # $id to dvd unit 1"

#deprecated
#burntodvd "dvd1" $input
#$burncmd $dvdopts -Z /dev/dvd1=$input $dvdspeed >>$logfile 2>&1


cdrecord -v -eject speed=$wrtspeed dev=2,1,0 /backup/ISO/$id.iso >>$logfile 2>&1

sleep 1

let currentdisc=$currentdisc+1
done

tolog "Removing ISO Image Files..."
cleanup $fisopath

echo 0 > $burnready
}

# make ISO from directory

function makeiso ()
{
let numdvds=`cat $dcntfile`
let num=$numdvds+1
dates=`date "+%H%M"`

for ((i=1; i<$num; i++))
do
volname="$i$volprefix$numdvds""-$date-$dates"
output="$isopath/$i$ext"
files="$filepath/$i/"

mkisofs $isoopts -V $volname -o $output $files 1>>$logfile 2>>$logfile
cleanup $files
done

echo 1 > $burnready
}

# creates a number of directories by number index
function makedirs ()
{
num=$1
pth=$2
for ((i=1; i<$num; i++))
do
mkdir -p $pth/$i
done
}

# remove spaces from files
function removespaces ()
{
find $findpath -name $findpatt|while read file
do
mv "$file" "`echo "$file"| nawk ' BEGIN {OFS="_"} $1=$1 '`"
done
}

function cleanup ()
{
rm -rf $1
}

function logtime ()
{
d=`date "+%d%m%y"`
t=`date "+%H%M"`
tolog "$d - $t"
}

# main
touch $logfile
burnmode=`cat $burnready`

if [ $burnmode -eq "1" ]; then
tolog "Burning to DVD(s)..."
burniso
tolog "Backup completed on:"
logtime
tolog "Done..."
else
tolog "Backup started on :"
logtime
tolog "Archiving..."
archive
tolog "Splitting archive into $brksize chunks..."
breakup
tolog "Analysing files to backup"
getFileSpecs
tolog "Organising..."
farg=( "${farray[@]}" )
FileArray "${farg[@]}"
tolog "Arranging..."
arrangefiles
tolog "Locating..."
showfileloc
tolog "Setting up directories"
createdirs
tolog "Moving Files..."
movefiles
tolog "Making ISOs..."
makeiso
tolog "Removing original Archive..."
cleanup $cleanup
tolog "Removing ISO's..."
rm -rf $fisopath
tolog "Backup completed on:"
logtime
tolog "Done..."
fi
---------------------------

Basically what happens is:

It takes files in the home directory and archives them into an uncompressed tar archive, then splits them into 1gb chunks, then arranges them into folders for each dvd ready for burning. It then creates ISO's of the split files for each dvd and deletes the original files and archive. The second time this is run, it goes into burn mode and in a loop asks for two blank dvd's as this script burns 2 copies of each disc one after the other. Because I used this on a remote controlled server it beeps when it wants the blank dvd's. With some clever trickery it can decipher if it really has blank dvd's in it's drives and only continues once this condition is met. It beeps depending on which disc it requires for example. It will beep once with a delay for disc one. Once it gets the blank discs it then burns disc one to dvd unit 1 and 2. It then repeats this for all the dvd's concerned and is totally automatic. Everything is logged to a file in the LOG directory and the filename is dependant on the date of the backup. This is used to ensure there was no errors.

Note you will need a lot of scratch space for this to work successfully because of the large files it creates, which involve a tar file, the split files, and of course iso files of up to 4.3 gig each.

This script is by no means ground breaking in terms of design, but it does work. I am looking into the possibility of modifying this with a type sorting script which organises files into different folders depending on type and then backups up the data without archiving straight to disc. This would be especially useful for transfering p2p type data from a stand alone p2p linux box in an automated fashion without the need to transfer files from one machine to another.

I have searched high and low on the internet for something like this and had no success, and no-one came up with a solution to detect blank dvd's, so this script solves that problem at least. This worked for me and my friend is very happy and this works well for a small business environment. I hope that you can make use of this yourselves in some way.

I forgot to mention that I have run this on the server as cron job and it appears to work successfully. You run it once to create the backup files and then run it again to write the isos to dvds in a loop. Enjoy people! No bad eh? for someone that doesn't do any bash programming usually?