LinuxQuestions.org
Latest LQ Deal: Complete CCNA, CCNP & Red Hat Certification Training Bundle
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - Newbie
User Name
Password
Linux - Newbie This Linux forum is for members that are new to Linux.
Just starting out and have a question? If it is not in the man pages or the how-to's this is the place!

Notices


Reply
  Search this Thread
Old 03-20-2012, 11:27 AM   #1
oldyankee
LQ Newbie
 
Registered: Dec 2011
Posts: 4

Rep: Reputation: Disabled
Need Scripting Help To Manage Old Logs


I need to purge old WebSphere log files that are a certain number of days old (or older) and that have the date appended to the end of the filenames, e.g.,
WASAppName.log.2012-03-20
WASApp2_log4j.log.2012-02-15
.
.
I know in a C program I could use something like the IsDate() function to test the last 10 characters of each file to make sure it is a date before deleting it.

I know I can use the find command to filter out files that are older than 10 days:

find /logdir -mtime +10 -exec ls -l {} \;

I have piped the output of this find command to a file which gives me a nice list of all of the log files (including their fully-qualified paths) in the specified directory that are more than 10 days old. Unfortunately that list also contains the current log file that WAS has open and is writing to and I sure don't even want to try to delete that one.

At this point, I am thinking that perhaps awk and/or sed might be used to
(1) parse this file
(2) read each entry
(3) verify that the last 10 characters constitute a valid date
(4) delete the file if (3) is true
(5) go get next entry

I am not a sed or awk knowledgeable person and would appreciate any help or suggestions anyone might have to offer. (Including alternative techniques for purging all archived logs that are older than 10 days.)
 
Old 03-20-2012, 11:40 AM   #2
TB0ne
LQ Guru
 
Registered: Jul 2003
Location: Birmingham, Alabama
Distribution: SuSE, RedHat, Slack,CentOS
Posts: 19,254

Rep: Reputation: 4435Reputation: 4435Reputation: 4435Reputation: 4435Reputation: 4435Reputation: 4435Reputation: 4435Reputation: 4435Reputation: 4435Reputation: 4435Reputation: 4435
Quote:
Originally Posted by oldyankee View Post
I need to purge old WebSphere log files that are a certain number of days old (or older) and that have the date appended to the end of the filenames, e.g.,
WASAppName.log.2012-03-20
WASApp2_log4j.log.2012-02-15
.
.
I know in a C program I could use something like the IsDate() function to test the last 10 characters of each file to make sure it is a date before deleting it. I know I can use the find command to filter out files that are older than 10 days:

find /logdir -mtime +10 -exec ls -l {} \;

I have piped the output of this find command to a file which gives me a nice list of all of the log files (including their fully-qualified paths) in the specified directory that are more than 10 days old. Unfortunately that list also contains the current log file that WAS has open and is writing to and I sure don't even want to try to delete that one.

At this point, I am thinking that perhaps awk and/or sed might be used to
(1) parse this file
(2) read each entry
(3) verify that the last 10 characters constitute a valid date
(4) delete the file if (3) is true
(5) go get next entry

I am not a sed or awk knowledgeable person and would appreciate any help or suggestions anyone might have to offer. (Including alternative techniques for purging all archived logs that are older than 10 days.)
...or you can just use logroate and/or logwatch, and have them do what you want. No need to reinvent the wheel. You don't say what version/distro of Linux you're using, but if those utilities aren't installed already, you can probably install them from your online repositories.
 
Old 03-20-2012, 04:10 PM   #3
oldyankee
LQ Newbie
 
Registered: Dec 2011
Posts: 4

Original Poster
Rep: Reputation: Disabled
Sorry - should have said RHEL AS 6 and SuSe Ent 9.2 (some of both).

I am new here and the folks here from long before I arrived tell me they tried logrotate but ran into all kinds of problems and inconsistencies. Also WAS is doing some of the log rotations through its built-in mechanisms and many logs are created and backed up by multiple apps that simply close the current log and rename it each day and open a new one (yeah - they should purge old ones from the app, but that's a can 'o worms that the new guy does not want to open as yet). Wouldn't "wires be crossed" if I were to simply unleash logrotate on these same log files? I guess I figured writing a purge command and executing it via cron would be the most friendly approach for this environment.
 
Old 03-20-2012, 04:35 PM   #4
TB0ne
LQ Guru
 
Registered: Jul 2003
Location: Birmingham, Alabama
Distribution: SuSE, RedHat, Slack,CentOS
Posts: 19,254

Rep: Reputation: 4435Reputation: 4435Reputation: 4435Reputation: 4435Reputation: 4435Reputation: 4435Reputation: 4435Reputation: 4435Reputation: 4435Reputation: 4435Reputation: 4435
Quote:
Originally Posted by oldyankee View Post
Sorry - should have said RHEL AS 6 and SuSe Ent 9.2 (some of both).

I am new here and the folks here from long before I arrived tell me they tried logrotate but ran into all kinds of problems and inconsistencies. Also WAS is doing some of the log rotations through its built-in mechanisms and many logs are created and backed up by multiple apps that simply close the current log and rename it each day and open a new one (yeah - they should purge old ones from the app, but that's a can 'o worms that the new guy does not want to open as yet). Wouldn't "wires be crossed" if I were to simply unleash logrotate on these same log files? I guess I figured writing a purge command and executing it via cron would be the most friendly approach for this environment.
Well, I'd find out first what "they" tried in the past, and what problems they encountered. Could very well be something wasn't configured correctly, and they put the blame on the software. Logrotate can be simple or VERY complex, but it sounds like it should do exactly what you want.
 
Old 03-20-2012, 06:53 PM   #5
chrism01
LQ Guru
 
Registered: Aug 2004
Location: Sydney
Distribution: Centos 6.9, Centos 7.3
Posts: 17,411

Rep: Reputation: 2397Reputation: 2397Reputation: 2397Reputation: 2397Reputation: 2397Reputation: 2397Reputation: 2397Reputation: 2397Reputation: 2397Reputation: 2397Reputation: 2397
Yeah, I'd go with logrotate; it's a very solid tool and you shouldn't have any 'issues'.
The only thing you need to worry about (and this is true even if you handcode something) is not interfering with another process (even the APP itself) if its already doing something with those logs.
You just need to be clear what (if anything) that is.

FYI; logrotate is part of the default install of RHEL; probably Suse too ...
Easiest thing is too look at current logrotate settings (RHEL)

/etc/logrotate.conf
/etc/logrotate.d

and check the man page here http://linux.die.net/man/8/logrotate

Last edited by chrism01; 03-20-2012 at 06:57 PM.
 
Old 03-20-2012, 07:37 PM   #6
Tinkster
Moderator
 
Registered: Apr 2002
Location: in a fallen world
Distribution: slackware by choice, others too :} ... android.
Posts: 23,067
Blog Entries: 11

Rep: Reputation: 910Reputation: 910Reputation: 910Reputation: 910Reputation: 910Reputation: 910Reputation: 910Reputation: 910
What the others said: logrotate is a great tool, and fairly customizable; and as
mentioned above: the only potential problem are long running processes that don't
release/rotate/chunk their own logs.

You mentioned a CURRENT log that had a mtime of 10+ days (which is kind of weird,
I've never known WAS to not be garrulous), so that may not be a good candidate for
rotation via logrotate.


Cheers,
Tink
 
Old 03-20-2012, 08:01 PM   #7
arashi256
Member
 
Registered: Jan 2008
Location: Brighton, UK
Distribution: Ubuntu 12.04 / CentOS 6.5
Posts: 394

Rep: Reputation: 61
I wrote this. Works for me and for any file location. I don't know if logrotate could do the same thing, but it was more fun to write this Hope it helps!

Code:
#!/bin/bash
DAYS_TO_KEEP=$1
echo -e "--------------------------------------------------\n"
echo "Days of file(s} to keep: $DAYS_TO_KEEP day(s)"
FILE_DIRECTORY=$2
echo "File directory location: $FILE_DIRECTORY"
TEMP_DIRECTORY=$3
echo "Temp directory: $TEMP_DIRECTORY"
touch ${TEMP_DIRECTORY}/OLDER-FILES.tmp
find ${FILE_DIRECTORY} -type f -mtime +${DAYS_TO_KEEP} > ${TEMP_DIRECTORY}/OLDER-FILES.tmp
FILE_COUNT=`wc -l ${TEMP_DIRECTORY}/OLDER-FILES.tmp | awk '{ print $1 }'`
echo "File Count over threshold: ${FILE_COUNT}"
if [ "$FILE_COUNT" -gt "0" ]; then
echo "Old log files to delete: ${FILE_COUNT}"
IFS=$'\n'
for filename in $(cat ${TEMP_DIRECTORY}/OLDER-FILES.tmp)
do
rm -f ${filename}
DELETED_COUNT=`expr $DELETED_COUNT + 1`
echo "Old log file ${DELETED_COUNT}/${FILE_COUNT} ${filename} - deleted."
done
else
echo "No log files older than ${DAYS_TO_KEEP} days. Exiting..."
fi
rm -f ${TEMP_DIRECTORY}/OLDER-FILES.tmp
echo "Done - ${DELETED_COUNT} file(s) were deleted"
echo -e "--------------------------------------------------\n"
To use, it’s very simple. Assuming I’ve saved the above code as “file-purge.sh” and I’m intent on deleting files that are over 93 days old I’d use the following parameters: -

Code:
file-purge.sh 93 [File directory to examine] [a temporary directory to use like $HOME/tmp]
Obviously, if logrotate can do this for any arbitrary location of files, I've probably wasted my time but hey...
 
1 members found this post helpful.
Old 03-22-2012, 06:46 PM   #8
oldyankee
LQ Newbie
 
Registered: Dec 2011
Posts: 4

Original Poster
Rep: Reputation: Disabled
Anytime anyone writes code for whatever reason - they are NOT wasting their time. They are exercising their brain and helping to stave off Alzheimer's in the future!
Kudos.
 
Old 03-22-2012, 08:00 PM   #9
Tinkster
Moderator
 
Registered: Apr 2002
Location: in a fallen world
Distribution: slackware by choice, others too :} ... android.
Posts: 23,067
Blog Entries: 11

Rep: Reputation: 910Reputation: 910Reputation: 910Reputation: 910Reputation: 910Reputation: 910Reputation: 910Reputation: 910
Indeed. But the caveat re the open log file with an mtime +10 remains with
that script as it does w/ logrotate.
 
  


Reply


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off



Similar Threads
Thread Thread Starter Forum Replies Last Post
LXer: Scripting KVM with Python, Part 2: Add a GUI to manage KVM with libvirt and Python LXer Syndicated Linux News 0 01-17-2012 09:11 PM
Collect and manage logs from hosts - free syslog server kkoko Linux - Security 6 09-28-2010 03:43 AM
Question about the logs viewer application (/var/logs) balteo Linux - General 1 11-21-2009 12:13 AM
Finding LDAP Server Logs / Application Logs in Linux arbignay Linux - Newbie 2 03-24-2008 10:54 AM
Firefox logs user out? Where are error logs? case1984 Linux - General 0 10-09-2004 03:22 PM

LinuxQuestions.org > Forums > Linux Forums > Linux - Newbie

All times are GMT -5. The time now is 01:13 PM.

Main Menu
Advertisement
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
Facebook: linuxquestions Google+: linuxquestions
Open Source Consulting | Domain Registration