Need help in writing a script to move old files...

NYMets91587 · 02-18-2009, 03:58 PM

hey guys i could really use a hand. i'm taking a Unix class at school and I need to write a shell script that will read the devices and when the memory is low will take the oldest files and move them to a new device or server...I'm so lost somebody please help. The script can be scheduled in crontab to execute when the system is not being used.

unSpawn · 02-18-2009, 04:44 PM

Per the LQ Rules, please do not post homework assignments verbatim. We're happy to assist if you have specific questions or have hit a stumbling point, however. Let us know what you've already tried and what references you have used (including class notes, books, and Google searches) and we'll do our best to help. Also, keep in mind that your instructor might also be an LQ member.

unSpawn · 02-18-2009, 04:48 PM

...and in case you don't even have any ideas or (pseudo)code to post and need to finish your assignment by this friday here's some Bash scripting guides: http://www.tldp.org/HOWTO/Bash-Prog-Intro-HOWTO.html, http://www.tldp.org/LDP/Bash-Beginne...tml/index.html and http://www.tldp.org/LDP/abs/html/.

NYMets91587 · 02-18-2009, 05:28 PM

This is what I have so far...from what i understand this will set the alert level to 90 percent and if the storage space is at that mark it will search for files older then 60 days, but now i need to know how to move them to another device so clear space...

#!/bin/sh
#

ALERT=90
df -H | grep -vE '^Filesystem|tmpfs|cdrom' | awk '{ print $5 " " $1 }' |

while read output;
do
#echo $output
usep=$(echo $output | awk '{ print $1}' | cut -d'%' -f1 )

partition=$(echo $output | awk '{ print $2 }' )

if [ $usep -ge $ALERT ]; then

find /var/log -mtime +60 -type f -exec cp

fi done

unSpawn · 02-18-2009, 05:40 PM

Quote:

Originally Posted by NYMets91587

find /var/log -mtime +60 -type f -exec cp

Excellent. You're almost there. If you would have searched LQ for "find exec" you'd have found at least three recent threads on the topic: 'find /var/log -mtime +60 -type f -exec echo mv {} /some/dir \;'. Remove the 'echo' and replace /shome/dir when you've verified this is what you want. And welcome to LQ BTW.

NYMets91587 · 02-18-2009, 05:54 PM

Thank you very much...would this be complete?

#!/bin/sh
#

ALERT=90
df -H | grep -vE '^Filesystem|tmpfs|cdrom' | awk '{ print $5 " " $1 }' |

while read output;
do
#echo $output
usep=$(echo $output | awk '{ print $1}' | cut -d'%' -f1 )

partition=$(echo $output | awk '{ print $2 }' )

if [ $usep -ge $ALERT ]; then

find /var/log -mtime +60 -type f -exec mv -t /oldfiles/dir

fi done

unSpawn · 02-19-2009, 02:21 AM

Code:

#!/bin/sh --
# Enable for debugging purposes
# set -xev
# Naming variables is important for understanding the script.
# While I don't alsways adhere either they should be in caps not lower or camelcase.
THRESHOLD=90
# Awk pick only /dev/hd? and /dev/sd/?
# And you don't want the partition but the mountpoint name, easier.
df -H | awk '/\/dev\/[h,s]d/ { print $5, $NF }' | while read USAGE MOUNTPOINT; do
  # Strip out the percentage char.
  if [ ${USAGE//%/} -ge $THRESHOLD ]; then
   # Moving files can be a bad thing, so logging it helps troubleshooting and 
   # moving them to their own subdir at least gives you a chance to recover them.
   logger -t kern.crit "Moving stale files on $MOUNTPOINT to /oldfiles/dir/$MOUNTPOINT."
   # If the subdir doesn't exist we should create it. If it somehow fails we shouldn't try and continue.
   [ -d /oldfiles/dir/$MOUNTPOINT ] || { mkdir -p /oldfiles/dir/$MOUNTPOINT || exit 127; }
   # If it somehow fails we shouldn't try and continue either.
   find $MOUNTPOINT -mtime +60 -type f -exec mv -f {} -t /oldfiles/dir/$MOUNTPOINT || exit 127
  fi
done
# A script should be terminated with the "all is well that ends well" exit line:
exit 0

* To improve this think of 0) handing off the actual find-and-move action to 'at' (echo 'some_action'|at some_timestamp ) because it'll execute in the background allowing you to process the next partition, 1) check file usage (fuser, lsof) before moving and 2) preserving (extended) attributes.

GazL · 02-19-2009, 05:24 PM

You're better off using 'df -P' (POSIX format output) in scripts. On my system if you use 'df -H' it can split the output over multiple lines which breaks the parsing of the output you were using. e.g.

Code:

bash-3.1$ df -H |cat
Filesystem             Size   Used  Avail Use% Mounted on
/dev/mapper/sysvg-lvroot
                       8.6G   5.5G   3.2G  64% /
/dev/hda1              104M    17M    82M  17% /boot
tmpfs                   64M      0    64M   0% /dev/shm
/dev/mapper/sysvg-lvvar
                       1.1G    52M   1.1G   5% /var
/dev/mapper/sysvg-lvtmp
                       535M   6.2M   529M   2% /tmp
/dev/mapper/sysvg-lvhome
                       2.2G   710M   1.5G  34% /home
/dev/mapper/datavg-lvfiles
                        35G    27G   7.7G  78% /srv/files
/dev/mapper/datavg-lvmusic
                        18G    15G   3.1G  83% /srv/music
/dev/mapper/datavg-lvbackup
                       4.3G   847M   3.5G  20% /srv/backup

Another thing you want to do is make sure to use the -xdev option on the 'find' command to restrict the processing to a single filesystem. Without it, you could end up processing filesystems that you shouldn't be.

I had a play with writing a solution for this last night out of curiosity, here's what I came up with. I don't see any harm in posting it now that unSpawn has already posted one solution.

DISCLAIMER: The following script is an example for educational purposes only. If you run it and it deletes all your files or otherwise screws up your system, don't come crying to me. READ IT CAREFULLY BEFORE USE, AND USE AT YOUR OWN RISK!

Code:

#!/bin/bash

# Filesystems to monitor for migration to archive:
#   Specify Filesystem mountpoints only, not subdirectories.
#   Note: Do NOT include trailing '/'
MONITOR_LIST='/srv/files /srv/music /srv/backup'

# Filesystem Percentage Threshold to trigger migration processing:
#   Note: Do NOT include % sign
THRESHOLD=90


# Location of the Archive filesystem to migrate qualifying files to:
#   Note: Do NOT include trailing '/'
ARCHIVE='/archive'

# Number of days old the file has to be to qualify for migration:
AGE=60


while read fsdev blocks used available capacity mount
do
  capacity=${capacity%\%}   # strip % sign

  if [ "$capacity" -ge "$THRESHOLD" ] ; then
    echo "# Filesystem: ${mount} at ${capacity}, ABOVE THRESHOLD!"

    while read filename
    do
      directory=$(dirname "$filename")

      mkdir -p "${ARCHIVE}${directory}" && \
        mv "${filename}" "${ARCHIVE}${filename}"

    done < <(find ${mount} -xdev -type f -mtime +${AGE} -print)

  fi
done < <(df -P ${MONITOR_LIST} | tail +2 )

exit 0

# end

I took a slightly different approach to unSpawn. Hopefully, between us, we've given you a few ideas to use in your own scripts.

Anyway, Hope that was of value. Have fun scripting...

Gaz.

unSpawn · 02-19-2009, 06:14 PM

Quote:

Originally Posted by GazL

You're better off using 'df -P' (POSIX format output) in scripts. On my system if you use 'df -H' it can split the output over multiple lines which breaks the parsing of the output you were using. e.g.

Is that screen representation or has 'df' some cut-off at which it really adds a newline?

Code:

]$ df -H | grep bindfs | while read s; do echo "${#s}"; done
105
]$ df -H | grep bindfs 
bindfs                  21G   5.1G    15G  26% /var
/tmp/some/ridiculously/no/ludicrously/weird/mountpo
int
]$

The above looks crooked but really is one line.

Regardless you're absolutely right about POSIX format output.

GazL · 02-20-2009, 05:43 AM

Quote:

Originally Posted by unSpawn

Is that screen representation or has 'df' some cut-off at which it really adds a newline?

It appears that without the (-P) option, if the filesystem device is over a certain length, df adds a \n after the filesystem device and splits the statistics onto a second line. I only discovered this myself when I was playing with the OP's grep|awk|while|read line.

evidence:

Code:

bash-3.1$ df
Filesystem           1K-blocks      Used Available Use% Mounted on
/dev/mapper/sysvg-lvroot
                       8355380   5292652   3062728  64% /
/dev/hda1               101086     15969     79898  17% /boot
tmpfs                    62424         0     62424   0% /dev/shm
/dev/mapper/sysvg-lvvar
                       1044244     50556    993688   5% /var
/dev/mapper/sysvg-lvtmp
                        522020      6028    515992   2% /tmp
/dev/mapper/sysvg-lvhome
                       2088692    665684   1423008  32% /home
/dev/mapper/datavg-lvfiles
                      33520436  26060884   7459552  78% /srv/files
/dev/mapper/datavg-lvmusic
                      16743732  13796796   2946936  83% /srv/music
/dev/mapper/datavg-lvbackup
                       4177588    826440   3351148  20% /srv/backup
bash-3.1$ df -H | grep lvvar
/dev/mapper/sysvg-lvvar
bash-3.1$ df -P | grep lvvar
/dev/mapper/sysvg-lvvar   1044244     50556    993688       5% /var
bash-3.1$

I suspect that unless you're using something like LVM, you're not likely to encounter this problem as the device names are usually short enough to not trigger this behaviour.