LinuxQuestions.org
Download your favorite Linux distribution at LQ ISO.
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - Newbie
User Name
Password
Linux - Newbie This Linux forum is for members that are new to Linux.
Just starting out and have a question? If it is not in the man pages or the how-to's this is the place!

Notices


Reply
  Search this Thread
Old 02-18-2009, 04:58 PM   #1
NYMets91587
LQ Newbie
 
Registered: Feb 2009
Posts: 3

Rep: Reputation: 0
Unhappy Need help in writing a script to move old files...


hey guys i could really use a hand. i'm taking a Unix class at school and I need to write a shell script that will read the devices and when the memory is low will take the oldest files and move them to a new device or server...I'm so lost somebody please help. The script can be scheduled in crontab to execute when the system is not being used.
 
Old 02-18-2009, 05:44 PM   #2
unSpawn
Moderator
 
Registered: May 2001
Posts: 29,331
Blog Entries: 55

Rep: Reputation: 3530Reputation: 3530Reputation: 3530Reputation: 3530Reputation: 3530Reputation: 3530Reputation: 3530Reputation: 3530Reputation: 3530Reputation: 3530Reputation: 3530
Per the LQ Rules, please do not post homework assignments verbatim. We're happy to assist if you have specific questions or have hit a stumbling point, however. Let us know what you've already tried and what references you have used (including class notes, books, and Google searches) and we'll do our best to help. Also, keep in mind that your instructor might also be an LQ member.
 
Old 02-18-2009, 05:48 PM   #3
unSpawn
Moderator
 
Registered: May 2001
Posts: 29,331
Blog Entries: 55

Rep: Reputation: 3530Reputation: 3530Reputation: 3530Reputation: 3530Reputation: 3530Reputation: 3530Reputation: 3530Reputation: 3530Reputation: 3530Reputation: 3530Reputation: 3530
...and in case you don't even have any ideas or (pseudo)code to post and need to finish your assignment by this friday here's some Bash scripting guides: http://www.tldp.org/HOWTO/Bash-Prog-Intro-HOWTO.html, http://www.tldp.org/LDP/Bash-Beginne...tml/index.html and http://www.tldp.org/LDP/abs/html/.
 
Old 02-18-2009, 06:28 PM   #4
NYMets91587
LQ Newbie
 
Registered: Feb 2009
Posts: 3

Original Poster
Rep: Reputation: 0
This is what I have so far...from what i understand this will set the alert level to 90 percent and if the storage space is at that mark it will search for files older then 60 days, but now i need to know how to move them to another device so clear space...

#!/bin/sh
#

ALERT=90
df -H | grep -vE '^Filesystem|tmpfs|cdrom' | awk '{ print $5 " " $1 }' |

while read output;
do
#echo $output
usep=$(echo $output | awk '{ print $1}' | cut -d'%' -f1 )


partition=$(echo $output | awk '{ print $2 }' )


if [ $usep -ge $ALERT ]; then

find /var/log -mtime +60 -type f -exec cp

fi done
 
Old 02-18-2009, 06:40 PM   #5
unSpawn
Moderator
 
Registered: May 2001
Posts: 29,331
Blog Entries: 55

Rep: Reputation: 3530Reputation: 3530Reputation: 3530Reputation: 3530Reputation: 3530Reputation: 3530Reputation: 3530Reputation: 3530Reputation: 3530Reputation: 3530Reputation: 3530
Quote:
Originally Posted by NYMets91587 View Post
find /var/log -mtime +60 -type f -exec cp
Excellent. You're almost there. If you would have searched LQ for "find exec" you'd have found at least three recent threads on the topic: 'find /var/log -mtime +60 -type f -exec echo mv {} /some/dir \;'. Remove the 'echo' and replace /shome/dir when you've verified this is what you want. And welcome to LQ BTW.

Last edited by unSpawn; 02-18-2009 at 06:42 PM.
 
Old 02-18-2009, 06:54 PM   #6
NYMets91587
LQ Newbie
 
Registered: Feb 2009
Posts: 3

Original Poster
Rep: Reputation: 0
Thank you very much...would this be complete?

#!/bin/sh
#

ALERT=90
df -H | grep -vE '^Filesystem|tmpfs|cdrom' | awk '{ print $5 " " $1 }' |

while read output;
do
#echo $output
usep=$(echo $output | awk '{ print $1}' | cut -d'%' -f1 )


partition=$(echo $output | awk '{ print $2 }' )


if [ $usep -ge $ALERT ]; then

find /var/log -mtime +60 -type f -exec mv -t /oldfiles/dir



fi done
 
Old 02-19-2009, 03:21 AM   #7
unSpawn
Moderator
 
Registered: May 2001
Posts: 29,331
Blog Entries: 55

Rep: Reputation: 3530Reputation: 3530Reputation: 3530Reputation: 3530Reputation: 3530Reputation: 3530Reputation: 3530Reputation: 3530Reputation: 3530Reputation: 3530Reputation: 3530
Code:
#!/bin/sh --
# Enable for debugging purposes
# set -xev
# Naming variables is important for understanding the script.
# While I don't alsways adhere either they should be in caps not lower or camelcase.
THRESHOLD=90
# Awk pick only /dev/hd? and /dev/sd/?
# And you don't want the partition but the mountpoint name, easier.
df -H | awk '/\/dev\/[h,s]d/ { print $5, $NF }' | while read USAGE MOUNTPOINT; do
  # Strip out the percentage char.
  if [ ${USAGE//%/} -ge $THRESHOLD ]; then
   # Moving files can be a bad thing, so logging it helps troubleshooting and 
   # moving them to their own subdir at least gives you a chance to recover them.
   logger -t kern.crit "Moving stale files on $MOUNTPOINT to /oldfiles/dir/$MOUNTPOINT."
   # If the subdir doesn't exist we should create it. If it somehow fails we shouldn't try and continue.
   [ -d /oldfiles/dir/$MOUNTPOINT ] || { mkdir -p /oldfiles/dir/$MOUNTPOINT || exit 127; }
   # If it somehow fails we shouldn't try and continue either.
   find $MOUNTPOINT -mtime +60 -type f -exec mv -f {} -t /oldfiles/dir/$MOUNTPOINT || exit 127
  fi
done
# A script should be terminated with the "all is well that ends well" exit line:
exit 0
* To improve this think of 0) handing off the actual find-and-move action to 'at' (echo 'some_action'|at some_timestamp ) because it'll execute in the background allowing you to process the next partition, 1) check file usage (fuser, lsof) before moving and 2) preserving (extended) attributes.
 
Old 02-19-2009, 06:24 PM   #8
GazL
Senior Member
 
Registered: May 2008
Posts: 4,083
Blog Entries: 1

Rep: Reputation: 1529Reputation: 1529Reputation: 1529Reputation: 1529Reputation: 1529Reputation: 1529Reputation: 1529Reputation: 1529Reputation: 1529Reputation: 1529Reputation: 1529
You're better off using 'df -P' (POSIX format output) in scripts. On my system if you use 'df -H' it can split the output over multiple lines which breaks the parsing of the output you were using. e.g.

Code:
bash-3.1$ df -H |cat
Filesystem             Size   Used  Avail Use% Mounted on
/dev/mapper/sysvg-lvroot
                       8.6G   5.5G   3.2G  64% /
/dev/hda1              104M    17M    82M  17% /boot
tmpfs                   64M      0    64M   0% /dev/shm
/dev/mapper/sysvg-lvvar
                       1.1G    52M   1.1G   5% /var
/dev/mapper/sysvg-lvtmp
                       535M   6.2M   529M   2% /tmp
/dev/mapper/sysvg-lvhome
                       2.2G   710M   1.5G  34% /home
/dev/mapper/datavg-lvfiles
                        35G    27G   7.7G  78% /srv/files
/dev/mapper/datavg-lvmusic
                        18G    15G   3.1G  83% /srv/music
/dev/mapper/datavg-lvbackup
                       4.3G   847M   3.5G  20% /srv/backup
Another thing you want to do is make sure to use the -xdev option on the 'find' command to restrict the processing to a single filesystem. Without it, you could end up processing filesystems that you shouldn't be.

I had a play with writing a solution for this last night out of curiosity, here's what I came up with. I don't see any harm in posting it now that unSpawn has already posted one solution.

DISCLAIMER: The following script is an example for educational purposes only. If you run it and it deletes all your files or otherwise screws up your system, don't come crying to me. READ IT CAREFULLY BEFORE USE, AND USE AT YOUR OWN RISK!

Code:
#!/bin/bash

# Filesystems to monitor for migration to archive:
#   Specify Filesystem mountpoints only, not subdirectories.
#   Note: Do NOT include trailing '/'
MONITOR_LIST='/srv/files /srv/music /srv/backup'

# Filesystem Percentage Threshold to trigger migration processing:
#   Note: Do NOT include % sign
THRESHOLD=90


# Location of the Archive filesystem to migrate qualifying files to:
#   Note: Do NOT include trailing '/'
ARCHIVE='/archive'

# Number of days old the file has to be to qualify for migration:
AGE=60


while read fsdev blocks used available capacity mount
do
  capacity=${capacity%\%}   # strip % sign

  if [ "$capacity" -ge "$THRESHOLD" ] ; then
    echo "# Filesystem: ${mount} at ${capacity}, ABOVE THRESHOLD!"

    while read filename
    do
      directory=$(dirname "$filename")

      mkdir -p "${ARCHIVE}${directory}" && \
        mv "${filename}" "${ARCHIVE}${filename}"

    done < <(find ${mount} -xdev -type f -mtime +${AGE} -print)

  fi
done < <(df -P ${MONITOR_LIST} | tail +2 )

exit 0

# end
I took a slightly different approach to unSpawn. Hopefully, between us, we've given you a few ideas to use in your own scripts.

Anyway, Hope that was of value. Have fun scripting...

Gaz.
 
Old 02-19-2009, 07:14 PM   #9
unSpawn
Moderator
 
Registered: May 2001
Posts: 29,331
Blog Entries: 55

Rep: Reputation: 3530Reputation: 3530Reputation: 3530Reputation: 3530Reputation: 3530Reputation: 3530Reputation: 3530Reputation: 3530Reputation: 3530Reputation: 3530Reputation: 3530
Quote:
Originally Posted by GazL View Post
You're better off using 'df -P' (POSIX format output) in scripts. On my system if you use 'df -H' it can split the output over multiple lines which breaks the parsing of the output you were using. e.g.
Is that screen representation or has 'df' some cut-off at which it really adds a newline?
Code:
]$ df -H | grep bindfs | while read s; do echo "${#s}"; done
105
]$ df -H | grep bindfs 
bindfs                  21G   5.1G    15G  26% /var
/tmp/some/ridiculously/no/ludicrously/weird/mountpo
int
]$
The above looks crooked but really is one line.

Regardless you're absolutely right about POSIX format output.
 
Old 02-20-2009, 06:43 AM   #10
GazL
Senior Member
 
Registered: May 2008
Posts: 4,083
Blog Entries: 1

Rep: Reputation: 1529Reputation: 1529Reputation: 1529Reputation: 1529Reputation: 1529Reputation: 1529Reputation: 1529Reputation: 1529Reputation: 1529Reputation: 1529Reputation: 1529
Quote:
Originally Posted by unSpawn View Post
Is that screen representation or has 'df' some cut-off at which it really adds a newline?
It appears that without the (-P) option, if the filesystem device is over a certain length, df adds a \n after the filesystem device and splits the statistics onto a second line. I only discovered this myself when I was playing with the OP's grep|awk|while|read line.

evidence:
Code:
bash-3.1$ df
Filesystem           1K-blocks      Used Available Use% Mounted on
/dev/mapper/sysvg-lvroot
                       8355380   5292652   3062728  64% /
/dev/hda1               101086     15969     79898  17% /boot
tmpfs                    62424         0     62424   0% /dev/shm
/dev/mapper/sysvg-lvvar
                       1044244     50556    993688   5% /var
/dev/mapper/sysvg-lvtmp
                        522020      6028    515992   2% /tmp
/dev/mapper/sysvg-lvhome
                       2088692    665684   1423008  32% /home
/dev/mapper/datavg-lvfiles
                      33520436  26060884   7459552  78% /srv/files
/dev/mapper/datavg-lvmusic
                      16743732  13796796   2946936  83% /srv/music
/dev/mapper/datavg-lvbackup
                       4177588    826440   3351148  20% /srv/backup
bash-3.1$ df -H | grep lvvar
/dev/mapper/sysvg-lvvar
bash-3.1$ df -P | grep lvvar
/dev/mapper/sysvg-lvvar   1044244     50556    993688       5% /var
bash-3.1$
I suspect that unless you're using something like LVM, you're not likely to encounter this problem as the device names are usually short enough to not trigger this behaviour.
 
  


Reply


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off



Similar Threads
Thread Thread Starter Forum Replies Last Post
Script to Move and rename files scribbl35 Linux - Newbie 1 07-18-2007 10:14 AM
writing spec file for RPM to move files Help Me Linux - Newbie 2 05-08-2006 01:58 PM
How can I move files from a host to another in a script eantoranz Linux - Networking 7 08-12-2005 03:39 PM
Script to Move files jain_rajesh Linux - Newbie 1 10-15-2004 09:53 AM
Need help with script to move files over a NFS Reggy Programming 1 03-19-2004 07:12 PM


All times are GMT -5. The time now is 02:55 PM.

Main Menu
Advertisement
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
Facebook: linuxquestions Google+: linuxquestions
Open Source Consulting | Domain Registration