Files don't move back to /var/log and keep getting renamed.

No_one_knows_me · 03-15-2010, 09:59 AM

I have this script that I use to find log files in the /var/log directory that are 2 days old, move them to /var/log/tmp, rename them to the system date.filename and move them back to /var/log. Everything seems to work as planned, except that the files don't get moved out of temp, and they keep getting rename. This leads to very long filenames such as:

2010-02-22.user_040.2010-03-05.user_040.2010-03-07.user_040.gz

What is it about this script that isn't moving it back to /var/log? Also, is there a better way of doing this than what I'm doing? Basically, I'm just trying to set up an audit trail on some of the files in /var/log, so that at the end of the month I can tar them, and then have our syslog server pick up the one giant monthly log.

Code:

# Create variables.
dir="/var/log"
tmp="$dir/tmp"
hostname="$(uname -n)"

# Create the temporary directory.
mkdir -p "$tmp"

# First, mv most of the files:
for log in kdm kernel cron ksyms messages rpmpkgs vmke vmkw secure; do
find "$dir" -name "$log.*" -maxdepth 1 -type f  -mtime +2 -exec cp "{}" "$tmp" +
done 

# Now mv any files that were not mv'd the first time.
# NOTE: next line will possibly mv files that were already mv'd above!
#find "$dir" -name "*.log*" -maxdepth 1 -type f  -mtime +2 -exec cp "{}" "$tmp" +

# mv all files to new filename, and put back in $dir:
for file in "$tmp"/*; do
mv "$file" "$dir/$(date +%F).$hostname.$file"
#gzip -c -9 "$file" > "$dir/$(date +%F).$(hostname).$(file).gz"
done

# uncomment next line to actually permanently delete $tmp in production
# rm -Rf "$tmp"

Any help you can provide would be greatly appreciated. I look forward to your thoughts and suggestions.

RaelOM · 03-15-2010, 10:06 AM

Have you run this script with the -x flag enabled in your shell parser to use step through so you can see where you logic is failing?

ie: bash -x <script name>

zhjim · 03-15-2010, 10:16 AM

Check out logrotate. It does what you are looking for expect moving things to /tmp. It just rotates the logs inside /var/log directory.
It quite got some power. And if you find it's missing some checkout the postrotate and prerotate option of the config files. There in you can run any shell code to suite your needs.

Cheers Zhjim

No_one_knows_me · 03-15-2010, 10:18 AM

No I haven't, so I'll do that now. Here's the output:

Code:

+ alias 'rm=rm -i'
+ alias 'cp=cp -i'
+ alias 'mv=mv -i'
+ '[' -f /etc/bashrc ']'
+ . /etc/bashrc
+++ id -gn
+++ id -un
+++ id -u
++ '[' root = root -a 0 -gt 99 ']'
++ umask 077
++ '[' '' ']'
+ dir=/var/log
+ tmp=/var/log/tmp
++ uname -n
+ hostname=esx5.trdm
+ mkdir -p /var/log/tmp
+ find /var/log -name 'kdm.*' -maxdepth 1 -type f -mtime +2 -exec cp '{}' /var/log/tmp +
find: missing argument to `-exec'
+ find /var/log -name 'kernel.*' -maxdepth 1 -type f -mtime +2 -exec cp '{}' /var/log/tmp +
find: missing argument to `-exec'
+ find /var/log -name 'cron.*' -maxdepth 1 -type f -mtime +2 -exec cp '{}' /var/log/tmp +
find: missing argument to `-exec'
+ find /var/log -name 'ksyms.*' -maxdepth 1 -type f -mtime +2 -exec cp '{}' /var/log/tmp +
find: missing argument to `-exec'
+ find /var/log -name 'messages.*' -maxdepth 1 -type f -mtime +2 -exec cp '{}' /var/log/tmp +
find: missing argument to `-exec'
+ find /var/log -name 'rpmpkgs.*' -maxdepth 1 -type f -mtime +2 -exec cp '{}' /var/log/tmp +
find: missing argument to `-exec'
+ find /var/log -name 'vmke.*' -maxdepth 1 -type f -mtime +2 -exec cp '{}' /var/log/tmp +
find: missing argument to `-exec'
+ find /var/log -name 'vmkw.*' -maxdepth 1 -type f -mtime +2 -exec cp '{}' /var/log/tmp +
find: missing argument to `-exec'
+ find /var/log -name 'secure.*' -maxdepth 1 -type f -mtime +2 -exec cp '{}' /var/log/tmp +
find: missing argument to `-exec'
++ date +%F
+ mv '/var/log/tmp/*' '/var/log/2010-03-15.esx5.trdm./var/log/tmp/*'
mv: cannot stat `/var/log/tmp/*': No such file or directory

So this looks to me like it is thinking that I am wanting to send the files to /var/log/tmp/*, which is what I'm not trying to do. There also appears to be a missing argument to `-exec' command according to the output, but if you look at the script, there is a copy command.

catkin · 03-15-2010, 10:19 AM

The script comment says # First, mv most of the files but the command run by find is cp not rm ???

Try putting an echo in front of mv "$file" "$dir/$(date +%F).$hostname.$file" and you will see why it doesn't work -- no such directory.

No_one_knows_me · 03-15-2010, 10:21 AM

zhjim,

I am wanting to use logrotate, but since I haven't really figured it out too well, I was trying this approach to make it more tangible to me. Do you have any helpful hints and/or suggestions in configuring and using logrotate?

No_one_knows_me · 03-15-2010, 10:28 AM

Quote:

The script comment says # First, mv most of the files but the command run by find is cp not rm ???

Catkin,

LOL, You're right, I never noticed that before. I guess it should be the 'mv' command, however, and not the 'mv' command as you suggested.

zhjim · 03-15-2010, 10:30 AM

Quote:

Originally Posted by No_one_knows_me

zhjim,

I am wanting to use logrotate, but since I haven't really figured it out too well, I was trying this approach to make it more tangible to me. Do you have any helpful hints and/or suggestions in configuring and using logrotate?

man logrotate

Best way to start would be to check out /etc/logrotate.conf and /etc/logrotate.d/*

logrotate.conf just has overall settings. logrotate.d is a directory which gets included by logrotate and normaly holds application specific files.
Most of the time you should just get the frequence of rotation (daily, weekly, monthly..) and the number of files to keep right.
Also when you want to transfer the files to another machine you can do it with logrotate.
I guess when you play around with it a few hours or a day you'll get the hang off it. Just ask if you're stuck. Just make up some rules and look at the output of logrotate -d your_trial_config_file

Maybe this one suites
http://www.debian-administration.org/articles/117

irmin · 03-15-2010, 10:41 AM

Quote:

# mv all files to new filename, and put back in $dir:
for file in "$tmp"/*; do
mv "$file" "$dir/$(date +%F).$hostname.$file"
#gzip -c -9 "$file" > "$dir/$(date +%F).$(hostname).$(file).gz"
done

The error is in this command: the variable $file in the while loop will contain the full path of the temporary file. Thus the mv command will fail, since the slashes in the full path will be regarded as subdirectories, which do no exist.
You better replace $file with $(basename $file).

But why do you need a temporary directory? You can do something like this:

Code:

find /var/log -name "*.log*" -maxdepth 1 -type f -mtime +2 | while read lfile; do gzip -c <$lfile >$(dirname $lfile)/$(date +%F).$(hostname).$(basename $lfile).gz; done

The errors you mention about find are due to the fact, that -exec ... + only allows {} at the end. So try "-exec cp -t $tmp {} +" instead.