LinuxQuestions.org
Review your favorite Linux distribution.
Go Back   LinuxQuestions.org > Forums > Non-*NIX Forums > Programming
User Name
Password
Programming This forum is for all programming questions.
The question does not have to be directly related to Linux and any language is fair game.

Notices


Reply
  Search this Thread
Old 07-22-2009, 09:50 AM   #1
aboleth
LQ Newbie
 
Registered: Jul 2009
Posts: 2

Rep: Reputation: 0
Need help writing a script to tar/delete old files


I have a directory of large log files (250MB each) that date back about a year. I want to write a script to find files older than 30 days, tar them, and then delete the originals. Ideally, actually , I would like the script to put similar log files (different names but created on the same day) all into one tarball... Here is a sample listing of files.

./2008-12-16_235900.log
./2008-12-16_235900.logptr
./2008-12-16_235900.loginitial_ptr
./2008-12-16_235900.logaccount_ptr
./2008-12-17_235900.logptr
./2008-12-17_235900.loginitial_ptr
./2008-12-17_235900.logaccount_ptr
./2008-12-18_235900.log
./2008-12-18_235900.logptr
./2008-12-18_235900.loginitial_ptr
./2008-12-18_235900.logaccount_ptr
./2008-12-19_235900.log
./2008-12-19_235900.logptr
./2008-12-19_235900.loginitial_ptr
./2008-12-19_235900.logaccount_ptr
./2008-12-20_235900.log
./2008-12-20_235900.logptr
./2008-12-20_235900.loginitial_ptr
./2008-12-20_235900.logaccount_ptr
./2008-12-21_235900.log
./2008-12-21_235900.logptr
./2008-12-21_235900.loginitial_ptr
./2008-12-21_235900.logaccount_ptr
./2008-12-22_235900.log
./2008-12-22_235900.logptr
./2008-12-22_235900.loginitial_ptr
./2008-12-22_235900.logaccount_ptr
./2008-12-23_235900.logaccount_ptr
./2009-02-19_110940.vlogptr
./2009-02-19_110940.vloginitial_ptr


You can see that I have several files per date, so balling those up into the same tar would be great. Space is also an issue, so i want it to function in the following manner

1. Find file older than X days
2. tar it
3. delete original
4. find next file older than X days..
5. repeat until done.

Putting the similar files in the same tarball isn't needed, it would just be nice.

Any help with this is much appreciated. I sucks with shell scripting so I don't much know where to start here.

Thanks in advance for any replies!
 
Old 07-22-2009, 03:15 PM   #2
jan61
Member
 
Registered: Jun 2008
Posts: 235

Rep: Reputation: 46
Moin,

Code:
# find files older than 30 days, extract date from filename, sort unique
find /path/to/logfiles -type f -mtime +30 -printf "%f\n" | cut -f1 -d_ | sort -u | \
while read prefix; do
  # create a compressed archive for one date
  tar czf /path/to/archives/${prefix}.tgz /path/to/logfiles/${prefix}*
  # remove the archived files
  rm -f /path/to/logfiles/${prefix}*
done
Not tested!

Jan
 
Old 07-22-2009, 03:17 PM   #3
TB0ne
LQ Guru
 
Registered: Jul 2003
Location: Birmingham, Alabama
Distribution: SuSE, RedHat, Slack,CentOS
Posts: 20,657

Rep: Reputation: 5075Reputation: 5075Reputation: 5075Reputation: 5075Reputation: 5075Reputation: 5075Reputation: 5075Reputation: 5075Reputation: 5075Reputation: 5075Reputation: 5075
Quote:
Originally Posted by aboleth View Post
I have a directory of large log files (250MB each) that date back about a year. I want to write a script to find files older than 30 days, tar them, and then delete the originals. Ideally, actually , I would like the script to put similar log files (different names but created on the same day) all into one tarball... Here is a sample listing of files.

You can see that I have several files per date, so balling those up into the same tar would be great. Space is also an issue, so i want it to function in the following manner

1. Find file older than X days
2. tar it
3. delete original
4. find next file older than X days..
5. repeat until done.

Putting the similar files in the same tarball isn't needed, it would just be nice.

Any help with this is much appreciated. I sucks with shell scripting so I don't much know where to start here.

Thanks in advance for any replies!
We'll be glad to help. but you're going to have to put some effort into it yourself. Post what you have done/tried so far.....

HINTS:

1. Read the man page on "find". Pay attention to the atime, ctime, and mtime parameters.
2. Lots of bash tutorials out there, showing how to do a "for" loop.

Structure it as so:

- Do the file find first. This will give you the whole list of files older than whatever date.
- Take that list, and pipe it into tar, creating the output .tar file
- Read that same list, and do an "rm -f" on each file name.
 
Old 07-23-2009, 02:43 AM   #4
vonbiber
Member
 
Registered: Apr 2009
Distribution: slackware 14.1 64-bit, slackware 14.2 64-bit, SystemRescueCD
Posts: 468

Rep: Reputation: 102Reputation: 102
you might need to store the current day, month, and year
into variables, so that when you retrieve the day, month
and year for each file you can compare the 2 dates
do a
man date
but to save you time:
thismonth=$(date +'%m')
thisday=$(date +'%d')
thisyear=$(date + '%Y')

do an echo to see:
echo $thismonth $thisday $thisyear

Also, you may dispose of all the files that are from
older than the previous year or from the previous year but
older than December of previous year

eg if the current year is 2009

for f in 199[0-9]* 200[0-7]* 2008-0* 2008-11*
do
....
dispose of these files
...
done

so that you'll have less files to process

let us know where you got so far
good luck
 
Old 07-23-2009, 07:16 AM   #5
aboleth
LQ Newbie
 
Registered: Jul 2009
Posts: 2

Original Poster
Rep: Reputation: 0
Ok give me a bit here and I'll post some of what I've done.

Thanks for your suggestions thus far!!!!
 
Old 07-23-2009, 10:15 AM   #6
vonbiber
Member
 
Registered: Apr 2009
Distribution: slackware 14.1 64-bit, slackware 14.2 64-bit, SystemRescueCD
Posts: 468

Rep: Reputation: 102Reputation: 102
A word of advice
When you start executing your scripts
before you do anything as drastic as

rm

do a test version first, that is, instead of

rm -f $filename

write this

echo "rm -f $filename"

and use a lot of echo "..." to check that the way you
set your variables look ok

and yes:

man find
man bash (especially
loop: while do ... done, for do ... done
case $something in ...) ;; ... esac
assigning variables, and extracting certain substrings:
${f##*/} ${f%.*}, etc.
the conditionals
for a string comparison: if [ $str1 == $str2 ]
for a number comparison: if [ $nb1 -lt $nb2 ]
)

Don't try to do everything at once.
Start writing small scripts and test them ...

good luck
 
  


Reply


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off



Similar Threads
Thread Thread Starter Forum Replies Last Post
delete files from tar using wildcards bunsenstraat Linux - Newbie 2 03-28-2009 10:21 AM
Delete files script using Find and Tar Zypher123 Red Hat 1 12-12-2008 09:45 AM
Script help - delete files older than 45 days but exclude the system files jojothedogboy Linux - Software 3 06-13-2008 03:43 PM
Delete old files script simpi Linux - Newbie 11 04-25-2008 02:37 AM
writing awk script files bigmark Linux - Software 1 10-19-2005 10:33 AM

LinuxQuestions.org > Forums > Non-*NIX Forums > Programming

All times are GMT -5. The time now is 05:07 AM.

Main Menu
Advertisement
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
Facebook: linuxquestions Google+: linuxquestions
Open Source Consulting | Domain Registration