Bash script of doom?

eidolonnight · 03-02-2009, 09:33 AM

I've been running the following Bash script to do hourly backups of our MySQL databases. It was running for awhile, but lately our server has been having issues, and it appears as though it may be this script. Can anyone spot anything possibly wrong? I am by no means an expert Bash programmer, but this seems simple enough that I'm having a hard time figuring out what might be wrong. Any help would be much appreciated.

Code:

#!/bin/sh
# Drupal database backup
# ---------------------------------------------------------------------

#########################
######TO BE MODIFIED#####

### System Setup ###
BACKUP=/home/XXX/XXX

### MySQL Setup ###
MUSER="XXX"
MPASS="XXX"
MHOST="localhost"

### FTP server Setup ###
FTPD="./XXX"
FTPU="XXX"
FTPP="XXX"
FTPS="XXX"

######DO NOT MAKE MODIFICATION BELOW#####
#########################################

### Binaries ###
TAR="$(which tar)"
GZIP="$(which gzip)"
FTP="$(which ftp)"
MYSQL="$(which mysql)"
MYSQLDUMP="$(which mysqldump)"

### Today + hour in 24h format ###
NOW=$(date +"%a%H")

### Create hourly dir ###

mkdir $BACKUP/$NOW

### Create dir for each databases, backup tables in individual files ###
  mkdir $BACKUP/$NOW/XXX
  FILE=$BACKUP/$NOW/XXX/XXX.sql.gz
  $MYSQLDUMP -u $MUSER  -h $MHOST  -p$MPASS XXX | $GZIP -9 > $FILE

  mkdir $BACKUP/$NOW/XXX
  FILE=$BACKUP/$NOW/XXX/XXX.sql.gz
  $MYSQLDUMP -u $MUSER  -h $MHOST  -p$MPASS XXX | $GZIP -9 > $FILE

### Compress all tables in one nice file to upload ###

cd $BACKUP

ARCHIVE=$BACKUP/$NOW.tar.gz
ARCHIVED=$BACKUP/$NOW

$TAR -cvf $ARCHIVE $ARCHIVED

### Dump backup using FTP ###
DUMPFILE=$NOW.tar.gz
$FTP -in $FTPS <<END_SCRIPT
quote USER $FTPU
quote PASS $FTPP
cd $FTPD
put $DUMPFILE
quit
END_SCRIPT

### Delete the backup dir and keep archive ###

rm -rf $ARCHIVED

### Clean up the databases ###

echo "Making things pretty"
mysqlcheck -u $MUSER -h $MHOST -p$MPASS --auto-repair --check --optimize --all-databases

chitambira · 03-03-2009, 10:14 AM

What sort of issues are you having with it? Any error messages, logs? Its hard to look into someone's code whaen you have no idea what you are looking for. cheers

eidolonnight · 03-04-2009, 06:33 AM

Quote:

Originally Posted by chitambira

What sort of issues are you having with it? Any error messages, logs? Its hard to look into someone's code whaen you have no idea what you are looking for. cheers

The Xen server we had the script running on would mysteriously start going into swap after a couple couple days. We saw this in top, but the strange thing is that neither top nor ps showed any processes using all that memory (1.5Gb).

On the Parallels server memory wasn't running out, but after a couple days the server would restart on its own.

This script is one of the few things that the servers have in common, and when we remove the script those issues go away (or at least they seem to have).

robertjinx · 03-04-2009, 06:53 AM

Try to run the script manually like with: sh -x script.sh or whatever is the name of the script.

This way you may find the issue easier... if you dont show any logs/errors something to go on, nobody will be able to give you some hints.

Good luck!

eidolonnight · 03-05-2009, 12:31 AM

I have run this manually. It runs fine with no errors whatsoever. None of the server logs showed any errors either. This is what's so confusing about the whole thing. Everything is fine except for the fact that after a couple days the server is blasted. In Xen we saw memory shoot through the roof (with nothing in ps or top to indicate what was using all the memory) and in parallelz the server just reboots. After removing the script everything seems to be fine. Both servers have run a week with no issues. The same script runs on a third openvz server (but only once a day as opposed to once an hour) just fine.

To give you more background, I've been using Linux for several years, I'm comfortable with BASH, and I configure servers and do this stuff all the time. I wouldn't post if this didn't confuse the heck out of me. I simply don't see how such a simple script could cause these massive failures, yet the fact remains that removing the script solves the issues. So, I'm posting here with the hope that someone with more experience in web server scripts can provide any insight into what may be going on.

You have as much information as I do. Any ideas are appreciated.

eidolonnight · 03-05-2009, 01:01 AM

I have run this manually. It runs fine with no errors whatsoever. None of the server logs showed any errors either. This is what's so confusing about the whole thing. Everything is fine except for the fact that after a couple days the server is blasted. In Xen we saw memory shoot through the roof (with nothing in ps or top to indicate what was using all the memory) and in parallelz the server just reboots. After removing the script everything seems to be fine. Both servers have run a week with no issues. The same script runs on a third openvz server (but only once a day as opposed to once an hour) just fine.

To give you more background, I've been using Linux for several years, I'm comfortable with BASH, and I configure servers and do this stuff all the time. I wouldn't post if this didn't confuse the heck out of me. I simply don't see how such a simple script could cause these massive failures, yet the fact remains that removing the script solves the issues. So, I'm posting here with the hope that someone with more experience in web server scripts can provide any insight into what may be going on.

You have as much information as I do. Any ideas are appreciated.

robertjinx · 03-05-2009, 01:19 AM

Hi, there could be 2 things there not working right... one of them could be this:

echo "Making things pretty"
mysqlcheck -u $MUSER -h $MHOST -p$MPASS --auto-repair --check --optimize --all-databases

mysql could crash the server in some cases and the other one could be the archive part "tar" which could use up the CPU and/or MEM for some weird reason, maybe impossible to backup something or some weird loop. But to be honest I cant test it to tell you for 100% is one of them.

Try more and see maybe somehow u will figure it out.

rweaver · 03-05-2009, 09:19 AM

Quote:

Originally Posted by eidolonnight

I've been running <SNIP>...[/CODE]

Sounds more like a Xen problem than a script problem.

eidolonnight · 03-05-2009, 10:56 AM

Quote:

Originally Posted by rweaver

Sounds more like a Xen problem than a script problem.

So do you think the parallelz thing was a coincidence? I suppose anything is a possibility.

rweaver · 03-05-2009, 10:59 AM

Quote:

Originally Posted by eidolonnight

So do you think the parallelz thing was a coincidence? I suppose anything is a possibility.

Maybe. The only other thing that comes to mind is if the machine becomes over utilized or the disk usage is high enough you might be running into overlapping runs of the script if your db's are fairly large, but even in that case you should be able to see where things are being utilized.

DeathStar_Name · 09-27-2012, 12:22 AM

This is my script that someone made for a small fee to freelance, and apparently someone laid out the network. All variables are familiar. Only I have it now with auto-delete like this

DATE="$(date +%Y%m%d)"

if [ $OS = "FreeBSD" ]; then
LOCAL_OLDDATE="$(date -v-14d +%Y%m%d)"
FTP_OLDDATE="$(date -v-6m +%Y%m%d)"
else
LOCAL_OLDDATE="$(date -d '14 days ago' +%Y%m%d)"
FTP_OLDDATE="$(date -d '4 mouth ago' +%Y%m%d)"
fi

echo "Dumping MySQL databases..."
${MYSQLDUMP} -u${MYSQLU} -p${MYSQLP} --all-databases > ${BACKUP}/mysql.sql

echo "Back up files and dirs..."
${TAR} czf ${BACKUP}/${DATE}.tgz --exclude='*.mp3' --exclude='*.MP3' ${FILE} ${BACKUP}/mysql.sql
echo "Done..."

cd $BACKUP
DUMPFILE="${DATE}.tgz"

cd $BACKUP
DUMPFILE="${DATE}.tgz"
echo "Put backup archive to remote FTP server"
$FTP -n $FTPS <<END_SCRIPT
quote USER $FTPU
quote PASS $FTPP
prompt off
binary
cd $FTPD
mput $DUMPFILE
delete ${FTP_OLDDATE}.tgz
quit
END_SCRIPT

rm ${BACKUP}/mysql.sql
rm ${BACKUP}/${LOCAL_OLDDATE}.tgz

chrism01 · 09-27-2012, 05:48 AM

I'm with rweaver; if the backup runs start overlapping, that's the right sort of symptoms. Worth a look.