Linux - NewbieThis Linux forum is for members that are new to Linux.
Just starting out and have a question?
If it is not in the man pages or the how-to's this is the place!
Notices
Welcome to LinuxQuestions.org, a friendly and active Linux Community.
You are currently viewing LQ as a guest. By joining our community you will have the ability to post topics, receive our newsletter, use the advanced search, subscribe to threads and access many other special features. Registration is quick, simple and absolutely free. Join our community today!
Note that registered members see fewer ads, and ContentLink is completely disabled once you log in.
hello all I am working on a script that will collect the following data to isolate a specific USB device (either a flash drive or an external HDD).
1. gather a list of all sd* in the /dev directory.
2. grep /proc/mounts to find the drive name that contains /boot to remove it from the above list.
3. generate a list of remaining drive names that are left in the sd*.
I have succeeded in the above 3 parts, now I am struggling with the harder parts.
I am attempting to create a while loop, and failing I might add, to take the above list (in my test environment sdb sdb1 sdc sdc1) to do a few things.
1. I would like to skip the sdb and sdc knowing they are NOT partitions therefor can not be mounted as just that.
2. look in /proc/mounts to see if sdb1 and or sdc1 are mounted and if so were are they mounted.
3. if they are not mounted, then to mount them to a specific location (/mnt/backup).
4. so a simple ls -laF (or my if -a you will see in the script i have) to look for the /mnt/backup/.usbbackup file. If it is there to move on with what will be the rest of the scrip. If it is not there to return the drive to what ever state it was in before looking at it, then move to the next device. If no devices have the .usbbackup file to error out and say device not found.
I know this is a bit of a mess of code, please be semi kind ive only been coding for less then 3 weeks.
Code:
#!/bin/bash
#
#
#Defining Functions:
#
FUNCTION1 ()
{
if [ -a /mnt/backup/.usb* ]
then
echo "y"
fi
}
#######################################################################
#
# this is a test to see if we can identify the correct USB device
#
#######################################################################
#
# Remove the devices.txt file before the script populates it with clean data
#
#######################################################################
#
rm -rf /root/devices.txt
#
#######################################################################
#
#
# DISK will generate a list of all /dev/sdx devices in the system.
#
#
#DISK=$( ls -l /dev/sd* 2>/dev/null | awk -F " " '{print $10 }'| cut -d '/' -f 3 >> devices.txt )
#
# A simpler way of getting the list without the use of awk
#
#
cd /dev
ls sd* >> /root/devices.txt
#
# VAR1 will identify the real HDD that contain /boot
#
VAR1=$( cat /proc/mounts | grep /boot | awk -F" " '{print $1}' | awk -F"/" '{print $3}' | cut -b 1-3 )
#
# VAR2 takes the device identified in VAR1 and strips it from devices.txt
# use $VAR2 to mount/umount $DEV and to look for /mnt/backup/.usb*
#
VAR2=$( grep -v $VAR1 /root/devices.txt )
#
# testing to see if i get what I want.
#
echo $VAR2
#
# This works great. I now have the sdx for the mount script to test if the device has the .usb* file on it.
#
# setting up the mount portion of the script now.
#
#
# Setting up while loop to test for the .usb* on the device.
#
#
RC=1
while [ $RC -eq 1 ]
do
WC=$( echo $VAR2 | cut -b 1-3 | wc -m )
if [ $WC -ne 4 ]
then
RC=1
else
cat /proc/mounts | grep $VAR2
if [ $? -eq 0 ]
then
FUNCTION1
if [ $? -eq y ]
then
break
else
mount /dev/$VAR2 /mnt/backup
FUNCTION1
if [ $? -eq y ]
then
break
else
RC=1
fi
fi
fi
fi
done
thank you in advance.
I know already I have messed up my $WC variable as it is not given me what I want there. I'm a bit confused on exactly how the while loop is suppose to grab the data out of the $VAR2. see below when I run these commands from the # prompt (yeah i know root is not the safest place to do this. ill worry about setting up visudo later to give user permissions to mount umount)
Meaningful variable and function names would be easier to understand
Good to run unalias -a to ensure the likes of ls is what you want it to be
An array is easier to loop over
Applying some of that:
Code:
unalias -a # For predictability
ls /dev/sd* >> /root/devices.txt # Working with /dev/sd* will be easier
#
# VAR1 will identify the real HDD that contain /boot
#
#boot_dev_sdX=$( cat /proc/mounts | grep /boot | awk -F" " '{print $1}' | awk -F"/" '{print $3}' | cut -b 1-3 )
boot_dev_sdX=$( cat /proc/mounts | grep /boot | cut -b 1-8 )
#
# VAR2 takes the device identified in VAR1 and strips it from devices.txt
# use $VAR2 to mount/umount $DEV and to look for /mnt/backup/.usb*
#
remaining_dev_sdX=( $( grep -v $boot_dev_sdX /root/devices.txt ) ) # Now an array
#
# testing to see if i get what I want.
#
# echo $VAR2 # Now is in array can more easily do in loop
#
# This works great. I now have the sdx for the mount script to test if the device has the .usb* file on it.
#
# setting up the mount portion of the script now.
#
#
# Setting up while loop to test for the .usb* on the device.
#
#
#RC=1
#while [ $RC -eq 1 ]
for (( i=0; i<${#remaining_dev_sdX[*]}; i++ )) # Standard loop over an array idiom
do
dev_sdX=${remaining_dev_sdX[i]}
echo "DEBUG: dev_sdX is $dev_sdX"
Sorry -- can't do more; have to do some other work now.
EDIT: ooops -- should be ls -1 (that's a number 1) to get one per line so grep -v to remove the /boot ones works.
wow, ok thanks. on the last line i take it more needs to come and what does the DEBUG: do? i can see the setting of dev_sdX to $dev_sdX. im guessing that is from above with the remaining_dev_sdX[i].
I think you need to think a little more about how you will identify if it is a usb device? ie. Simply having /boot as part of the drive only indicates that one may be a hard drive
but you may have a usb drive with that directory or even more hard drives that do not (I was unable to test this as for myself /boot is not mounted separately so this solution would not work at all).
As for DEBUG, it does nothing as this is a simple echo to give you debug information
I was also concerned about the following line:
Quote:
3. if they are not mounted, then to mount them to a specific location (/mnt/backup)
If we assume anymore than one drive is not mounted, I do not see the advantage of mounting them all to the same place? (I may have missed something here)
I think you need to think a little more about how you will identify if it is a usb device? ie. Simply having /boot as part of the drive only indicates that one may be a hard drive
but you may have a usb drive with that directory or even more hard drives that do not (I was unable to test this as for myself /boot is not mounted separately so this solution would not work at all).
As for DEBUG, it does nothing as this is a simple echo to give you debug information
I was also concerned about the following line:
If we assume anymore than one drive is not mounted, I do not see the advantage of mounting them all to the same place? (I may have missed something here)
yes this is only a small portion of a larger backup script. all of our USB devices are formatted ext3 and then we simply do a touch /mnt/backup/.usbbackup on that drive. after it is mounted we will then just use the if -a command in the FUNCTION1 from the top of my portion of the script to verify it is the drive we are looking for.
Also 100% of our servers in the field have the /boot on their real HDD, by checking for that partition I can with certainty remove that device from the list to check.
Do not capitalize normal vars. Only the env vars are capitalized.
This avoids accidentatly deleting them.
Never parse the output of "ls" and progs like that. (use "find ... -print0...." instead)
They are not reliable. They depend on LOCALE settings and more important they do behave different on different GNU/Linux systems.
Do not use #!/bin/bash as a shebang. It is NOT guaranteed that bash will reside there.
Let the os find out, where it is. The os knows much better to find it.
Use "#!/usr/bin/env bash" (without quotes) to ensure that bash will always found.
If youre not bound to write POSIX shell scripts, don't use "[". Use "[[ ... ]] instead. This is newer, can do more and makes your scripts more readable.
Try something like that:
Code:
# we are going to assign the output of $( find....) to an arary var named "devs"
# var=( value1 value2 ) creates an array "var", where
devs=( $( find /dev -name 'sd*') )
# using parameter expansion of bash gives a list of the devices to the for construct
# the var "devic" will hold one device on each step of the for loop
for devic in ${devs[*]};
do
# to get just the name you can use POSIX compatible "basename /dev/sdbX"
# at the cost of spawning a new process.
# much more faster is again the bash parameterexpansion
# following line gives same as "basename..."
devi=${devic##*/}
# use the modern test [[
# and again parameter expansion
# LengthOfVar=${#var}
if [[ ${#devi} > 3 ]]; then
printf $devi'\n' ;
fi
done
The last bit from uhelp is really well put together, but sadly it leaves the HDD in the list and cuts one of the flash drives. as im not up to speed on the find/printf tools you used im at a bit of a loss. going to man page those in a bit.
The first script from catkin only sees one of the 2 USB devices on the system.
just post the pathnames of these flash devices, though that I can get a glue, what you are really after....
sda (on this system) is the known HDD. this is why in my original script i was looking in /proc/mounts for /boot to drop that entire device from the devices.txt file.
the 2 flash drives on this system that i am using for testing are sdb and sdc. in the field this can be from sdb-sde (i think that is the highest drive letter ive seen) so to be safe originally i was looking for all sd* and just scrubbing out any device with /boot.
the OSs are FC5, FC7, and CentOS 5.4. we also have a few RH7/8 out in the field, but none of them will ever have support, from us, for USB devices.
We run a cron job, typically at 9pm local nightly, on a backup script that grabs some of the system files like /etc/hosts and /etc/cups/printers.conf plus a handful of others, then tars up the entire directory of the database and application with those system files. We then just mv the tarball to the /mnt/backup mount point.
Our problem comes in with user error. if a customer removes the USB device while it is still mounted when they put on back in, the new device will have a different drive letter. We hard code the map in fstab, example:
/dev/sdb1 /mnt/backup auto noauto,users,rw 0 0
is a typical mount point we create. This will fail if the drive letter changes to /dev/sdc1 for example. What I am attempting to do is create a portion of the script that will identify the exact device name, then mount the full path of sudo mount /dev/<device> /mnt/backup so the script will also have the highest chance of running.
I am sorry if I was not clear enough to begin with. Does that make my goals a little less fuzzy?
If there is a better way of doing this lie udev (i know nothing about this, it was just pointed out to me by a friend who atm does not have time to go into details, but should later) then I am open to what ever options are better, faster, more reliable, etc... to accomplishing my task.
The next OS we will be adding to the script is CentOS 6.x
This works to loop over all the /dev/sd<letter><number> devices that are not on the whole disk device that contains the partition containing the file system mounted as /boot (which may not be the best way of tackling the requirement but does illustrat bash scripting techniques)
Code:
#!/bin/bash
# Set up environment
unalias -a # For predictable commands
export LANG=C # For predictable pattern matching
shopt -s extglob # To allow pattern repeats in filename expansion
# Find the device (normally /dev/sd[a-z][number]) that includes the /boot file system
boot_dev_sdX=$( cat /proc/mounts | grep /boot | cut -d ' ' -f1 )
# Adjust to the whole device containing the partition
boot_dev_sdX=${boot_dev_sdX%%*([[:digit:]])}
# Populate array with all the /dev/sd* that do not include the /dev/sd[a-z] that includes the /boot file system
remaining_dev_sdX=( $( ls -1 /dev/sd[a-z]+([[:digit:]]) | grep -v $boot_dev_sdX ) )
# Loop for each candidate partition
for (( i=0; i<${#remaining_dev_sdX[*]}; i++ )) # Standard loop over an array idiom
do
dev_sdX=${remaining_dev_sdX[i]}
echo "DEBUG: dev_sdX is $dev_sdX"
done
The output on my system is:
Code:
DEBUG: dev_sdX is /dev/sdb1
DEBUG: dev_sdX is /dev/sdb2
DEBUG: dev_sdX is /dev/sdb5
DEBUG: dev_sdX is /dev/sdb6
DEBUG: dev_sdX is /dev/sdc1
Generally, uhelp's suggestion not to parse ls output is good but, in the case of very regular path names like /dev/sd* and a simple usage like ls -1, I suggest it's OK.
Do not use #!/bin/bash as a shebang. It is NOT guaranteed that bash will reside there.
Let the os find out, where it is. The os knows much better to find it.
Use "#!/usr/bin/env bash" (without quotes) to ensure that bash will always found.
There are pros and cons. env uses environment variable PATH which may or may not be set correctly. In case PATH is not set correctly and when the location of bash on the target systems is known, it is more robust to use a direct bash shebang line.
Our problem comes in with user error. if a customer removes the USB device while it is still mounted when they put on back in, the new device will have a different drive letter.
...
If there is a better way of doing this lie udev (i know nothing about this, it was just pointed out to me by a friend who atm does not have time to go into details, but should later) then I am open to what ever options are better, faster, more reliable, etc... to accomplishing my task.
if a customer removes the USB device while it is still mounted it could damage the file system. To minimise the danger, the script which writes to the USB device should unmount it when it has finished writing (and it can power it down saving a little electricity).
udev is a much better solution. I am happy to share my udev solution -- a udev rule file and two scripts.
if a customer removes the USB device while it is still mounted it could damage the file system. To minimise the danger, the script which writes to the USB device should unmount it when it has finished writing (and it can power it down saving a little electricity).
udev is a much better solution. I am happy to share my udev solution -- a udev rule file and two scripts.
yes our current backup script does umount as one of the last things that it does.
I would love to see/learn more about the udev. Im playing with some stuff now, but I think I botched it. Thankfully on a test system I can do just that and not worry about screwing anything up.
Code:
[root@rx30 rx30]# bash -xv backup.sh
dtstamp="`date +%Y%m%d%H%M%S `"
date +%Y%m%d%H%M%S
++ date +%Y%m%d%H%M%S
+ dtstamp=20120216094234
logger -p local0.info "[Backup] Attempting backup ${dtstamp} at `date`"
date
++ date
+ logger -p local0.info '[Backup] Attempting backup 20120216094234 at Thu Feb 16 09:42:34 EST 2012'
### SCAN / ON-LINE
pvscan >> /dev/null 2>&1 # Never hurts
+ pvscan
vgscan >> /dev/null 2>&1 # Never hurts
+ vgscan
vgchange -ay vgusb >> /dev/null 2>&1
+ vgchange -ay vgusb
# Fail if the logical volume "backup" is not available
# if [ ! -f "/dev/mapper/vgusb-backup" ] ; then
# logger -p local0.warn "[Backup] USB Backup Disk Not Connected"
# exit 1
# fi
### MOUNT ATTEMPT
mount -t ext3 /dev/mapper/vgusb-backup /mnt/backup >> /dev/null 2>&1
+ mount -t ext3 /dev/mapper/vgusb-backup /mnt/backup
rc=$?
+ rc=0
if [ $rc -ne 0 ]; then
logger -p local0.warn "[Backup] Unable to mount (rc=${rc}) USB Backup Disk"
exit 2
fi
+ '[' 0 -ne 0 ']'
### BACKUP
# (do your backup here)
#
if [ ! -d $HOME/logs ]
then
echo "Logfile Doesn't Exist!!!" >> $logname
echo "Creating Directory" >> $logname
mkdir $HOME/logs
fi
+ '[' '!' -d /root/logs ']'
#####################################################################################################
# Move any old backups that might still be in the ########################################
# /usr/rx30 folder to /tmp and erase any older than 2 days ########################################
#####################################################################################################
find /tmp -type f -atime +2 | xargs rm -f # necessary?
+ find /tmp -type f -atime +2
+ xargs rm -f
mv $HOME/rx{license}*.bz /tmp
+ mv '/root/rx{license}*.bz' /tmp
mv: cannot stat `/root/rx{license}*.bz': No such file or directory
#####################################################################################################
# Tar the rx30 folder into a BZip2 archive #########################################################
#####################################################################################################
echo "Compressing Rx30 contents into ${tarfilename}" >> $logname
+ echo 'Compressing Rx30 contents into '
backup.sh: line 39: $logname: ambiguous redirect
tar -cjvf $tarfilename $HOME/*
+ tar -cjvf /root/anaconda-ks.cfg /root/deviceCheck.sh /root/disk.log /root/fixscripts.log /root/foo.sh /root/fsck.sh /root/install.log /root/install.log.syslog /root/logs /root/minicom.log /root/teamviewer_linux.rpm /root/test.sh /root/test2.sh /root/vpninstl /root/vpninstl.bak
tar: Removing leading `/' from member names
/root/deviceCheck.sh
/root/disk.log
/root/fixscripts.log
/root/foo.sh
/root/fsck.sh
/root/install.log
/root/install.log.syslog
/root/logs/
/root/logs/fsck.log
/root/minicom.log
/root/teamviewer_linux.rpm
/root/test.sh
/root/test2.sh
/root/vpninstl
/root/vpninstl.bak
echo "Tar file created" >> $logname
+ echo 'Tar file created'
backup.sh: line 41: $logname: ambiguous redirect
#####################################################################################################
# Start the copy to Network Share ##################################################################
#####################################################################################################
echo "Backup to Network Share in progress" >> $logname
+ echo 'Backup to Network Share in progress'
backup.sh: line 46: $logname: ambiguous redirect
cp $tarfilename /mnt/backup
+ cp /mnt/backup
cp: missing destination file operand after `/mnt/backup'
Try `cp --help' for more information.
if [ $? -gt 0 ]
then
echo "Copy to Network Share FAILED!!!" >> $logname
mv $tarfilename /tmp
echo "Backup file moved to /tmp - Please Contact TDS Support" >> $logname
# Delete local backup files older than 2 days.
echo "Deleting local backups older than 2 days" >> $logname
find /tmp -type f -atime +2 | xargs rm -f
# kdialog --error "Backup was not successful. Please contact TDS Support"
else
echo "Copy to Network Share Successful" >> $logname
rm $tarfilename
echo "Removing Archive" >> $logname
#kdialog --msgbox "Backup completed successful!"
fi
+ '[' 1 -gt 0 ']'
+ echo 'Copy to Network Share FAILED!!!'
backup.sh: line 50: $logname: ambiguous redirect
+ mv /tmp
mv: missing destination file operand after `/tmp'
Try `mv --help' for more information.
+ echo 'Backup file moved to /tmp - Please Contact TDS Support'
backup.sh: line 52: $logname: ambiguous redirect
+ echo 'Deleting local backups older than 2 days'
backup.sh: line 55: $logname: ambiguous redirect
+ find /tmp -type f -atime +2
+ xargs rm -f
echo "Backup to Network Share complete" >> $logname
+ echo 'Backup to Network Share complete'
backup.sh: line 64: 1: ambiguous redirect
sync ; sync
+ sync
+ sync
### UMOUNT / OFF-LINE
umount /dev/mapper/vgusb-backup >> /dev/null 2>&1
+ umount /dev/mapper/vgusb-backup
vgchange -an vgusb >> /dev/null 2>&1
+ vgchange -an vgusb
logger -p local0.info "[Backup] Completed backup ${dtstamp} at `date`"
date
++ date
+ logger -p local0.info '[Backup] Completed backup 20120216094234 at Thu Feb 16 09:42:40 EST 2012'
exit 0
+ exit 0
[root@rx30 rx30]# df -h
Filesystem Size Used Avail Use% Mounted on
/dev/sda3 8.7G 6.2G 2.2G 75% /
/dev/sda2 8.7G 2.2G 6.1G 27% /usr/rx30
/dev/sda1 99M 12M 82M 13% /boot
tmpfs 501M 0 501M 0% /dev/shm
10.10.0.105:/data1 295G 236G 44G 85% /mnt/data1
10.10.0.105:/data2 295G 236G 44G 85% /mnt/data2
10.10.0.105:/data3 295G 236G 44G 85% /mnt/data3
10.10.0.105:/data4 295G 236G 44G 85% /mnt/data4
[root@rx30 rx30]# mount -t ext3 /dev/mapper/vgusb-backup /mnt/backup
mount: special device /dev/mapper/vgusb-backup does not exist
[root@rx30 rx30]# mount -t ext3 /dev/mapper/vgusb-backup /mnt/backup
mount: special device /dev/mapper/vgusb-backup does not exist
[root@rx30 rx30]# backup.sh
-bash: backup.sh: command not found
[root@rx30 rx30]# cat backup.sh
dtstamp="`date +%Y%m%d%H%M%S `"
logger -p local0.info "[Backup] Attempting backup ${dtstamp} at `date`"
### SCAN / ON-LINE
pvscan >> /dev/null 2>&1 # Never hurts
vgscan >> /dev/null 2>&1 # Never hurts
vgchange -ay vgusb >> /dev/null 2>&1
# Fail if the logical volume "backup" is not available
# if [ ! -f "/dev/mapper/vgusb-backup" ] ; then
# logger -p local0.warn "[Backup] USB Backup Disk Not Connected"
# exit 1
# fi
### MOUNT ATTEMPT
mount -t ext3 /dev/mapper/vgusb-backup /mnt/backup >> /dev/null 2>&1
rc=$?
if [ $rc -ne 0 ]; then
logger -p local0.warn "[Backup] Unable to mount (rc=${rc}) USB Backup Disk"
exit 2
fi
### BACKUP
# (do your backup here)
#
if [ ! -d $HOME/logs ]
then
echo "Logfile Doesn't Exist!!!" >> $logname
echo "Creating Directory" >> $logname
mkdir $HOME/logs
fi
#####################################################################################################
# Move any old backups that might still be in the ########################################
# /usr/rx30 folder to /tmp and erase any older than 2 days ########################################
#####################################################################################################
find /tmp -type f -atime +2 | xargs rm -f # necessary?
mv $HOME/rx{license}*.bz /tmp
#####################################################################################################
# Tar the rx30 folder into a BZip2 archive #########################################################
#####################################################################################################
echo "Compressing Rx30 contents into ${tarfilename}" >> $logname
tar -cjvf $tarfilename $HOME/*
echo "Tar file created" >> $logname
#####################################################################################################
# Start the copy to Network Share ##################################################################
#####################################################################################################
echo "Backup to Network Share in progress" >> $logname
cp $tarfilename /mnt/backup
if [ $? -gt 0 ]
then
echo "Copy to Network Share FAILED!!!" >> $logname
mv $tarfilename /tmp
echo "Backup file moved to /tmp - Please Contact TDS Support" >> $logname
# Delete local backup files older than 2 days.
echo "Deleting local backups older than 2 days" >> $logname
find /tmp -type f -atime +2 | xargs rm -f
# kdialog --error "Backup was not successful. Please contact TDS Support"
else
echo "Copy to Network Share Successful" >> $logname
rm $tarfilename
echo "Removing Archive" >> $logname
#kdialog --msgbox "Backup completed successful!"
fi
echo "Backup to Network Share complete" >> $logname
sync ; sync
### UMOUNT / OFF-LINE
umount /dev/mapper/vgusb-backup >> /dev/null 2>&1
vgchange -an vgusb >> /dev/null 2>&1
logger -p local0.info "[Backup] Completed backup ${dtstamp} at `date`"
exit 0
[root@rx30 rx30]#
[root@rx30 rx30]# mount -t ext3 /dev/mapper/vgusb-backup /mnt/backup
mount: special device /dev/mapper/vgusb-backup does not exist
sorry for the mess, this is both the script and its output with bash -xv. I had to comment out the first fail check as it was exit 1 at that point. I think I should move that fail check below the mount portion, but Im still semi lost here. Getting way over my head. I also noted that now the LV is gone from the system. It no longer sees the device. I guess ill have to remake it.
LinuxQuestions.org is looking for people interested in writing
Editorials, Articles, Reviews, and more. If you'd like to contribute
content, let us know.