LinuxQuestions.org
Go Job Hunting at the LQ Job Marketplace
Go Back   LinuxQuestions.org > Forums > Non-*NIX Forums > Programming
User Name
Password
Programming This forum is for all programming questions.
The question does not have to be directly related to Linux and any language is fair game.

Notices

Reply
 
Search this Thread
Old 10-09-2007, 12:18 AM   #1
DragonM15
Member
 
Registered: Sep 2003
Location: USA
Distribution: Slackware (Multiple Versions)
Posts: 455

Rep: Reputation: 31
Bash script to put log files into single file and email


I am looking for a bash script that will take my log files, put them into a single file, and email that file to my email account. This is what I have so far, but it doesn't work, and besides I know there is a more effective way to do the same thing.

Code:
#!/bin/bash
# script to send logs to email
LOGDATE="Oct  8"
APACHEACDATE="08/Oct"
MYFILEMAME="$HOME/systats"
GREP="/bin/grep"

echo 'Collecting Logs'
echo '/var/log/secure' > $MYFILENAME
cat /var/log/secure | $GREP $LOGDATE >> $MYFILENAME
wait
echo '/var/log/messages' >> $MYFILENAME
cat /var/log/messages | $GREP $LOGDATE >> $MYFILENAME
wait
echo '/var/log/maillog' >> $MYFILENAME
cat /var/log/maillog | $GREP $LOGDATE >> $MYFILENAME 
wait
echo '/var/log/apache/access_log' >> $MYFILENAME
cat /var/log/apache/access_log | $GREP $APACHEACDATA >> $MYFILENAME
wait
echo '/var/log/apache/error_log' >> $MYFILENAME
cat /var/log/apache/error_log | $GREP $LOGDATE >> $MYFILENAME
wait
echo 'dmesg' >> $MYFILENAME
dmesg >> $MYFILENAME
wait
echo '/var/log/secure' >> $MYFILENAME
cat /var/log/secure | $GREP $LOGDATE >> $MYFILENAME
wait
echo '/var/log/syslog' >> $MYFILENAME
cat /var/log/syslog | $GREP $LOGDATE >> $MYFILENAME
wait
echo '/var/log/vsftpd.log' >> $MYFILENAME
cat /var/log/vsftpd.log | $GREP $LOGDATE >> $MYFILENAME
wait
echo 'Sending Mail'
mail -s "Sys Stats for Today" user < $MYFILENAME
rm $MYFILENAME
If I run this file I get the errors:
Code:
Collecting Logs
systats.sh: line 9: $MYFILENAME: ambiguous redirect
systats.sh: line 10: 1: ambiguous redirect
systats.sh: line 12: 1: ambiguous redirect
systats.sh: line 13: 1: ambiguous redirect
systats.sh: line 15: 1: ambiguous redirect
systats.sh: line 16: 1: ambiguous redirect
systats.sh: line 18: 1: ambiguous redirect
systats.sh: line 19: 1: ambiguous redirect
systats.sh: line 21: 1: ambiguous redirect
systats.sh: line 22: 1: ambiguous redirect
systats.sh: line 24: 1: ambiguous redirect
systats.sh: line 25: $MYFILENAME: ambiguous redirect
systats.sh: line 27: $MYFILENAME: ambiguous redirect
systats.sh: line 28: 1: ambiguous redirect
systats.sh: line 30: 1: ambiguous redirect
systats.sh: line 31: 1: ambiguous redirect
systats.sh: line 33: 1: ambiguous redirect
systats.sh: line 34: 1: ambiguous redirect
Sending Mail
systats.sh: line 37: $MYFILENAME: ambiguous redirect
If somebody knows of a more efficient way to do this (which I'm sure there is) please, do tell.

Thanks,
DragonM15
 
Old 10-09-2007, 12:55 AM   #2
purplerice
LQ Newbie
 
Registered: Nov 2004
Location: SF Bay Area
Distribution: Fedora
Posts: 11

Rep: Reputation: 0
There is a typo in your script. You did not define
MYFILENAME, instead, you defined MYFILEMAME --

MYFILEMAME="$HOME/systats"
!(MAME should be NAME)

Last edited by purplerice; 10-09-2007 at 12:57 AM.
 
Old 10-09-2007, 01:47 AM   #3
DragonM15
Member
 
Registered: Sep 2003
Location: USA
Distribution: Slackware (Multiple Versions)
Posts: 455

Original Poster
Rep: Reputation: 31
Ok, I admit I feel like a complete and utter moron for that one. But now for the next hump:
Code:
Collecting Logs
grep: 8: No such file or directory
grep: 8: No such file or directory
grep: 8: No such file or directory
Usage: grep [OPTION]... PATTERN [FILE]...
Try `grep --help' for more information.
grep: 8: No such file or directory
grep: 8: No such file or directory
grep: 8: No such file or directory
grep: 8: No such file or directory
Sending Mail
If I run with -v (for verbose) I get:
Code:
#!/bin/bash
# script to send logs to email
LOGDATE="Oct  8"
APACHEACDATE="08/Oct"
MYFILENAME="$HOME/systats"
GREP="/usr/bin/grep"

echo 'Collecting Logs'
Collecting Logs
echo '/var/log/secure' > $MYFILENAME
cat /var/log/secure | $GREP $LOGDATE >> $MYFILENAME
grep: 8: No such file or directory
wait
echo '/var/log/messages' >> $MYFILENAME
cat /var/log/messages | $GREP $LOGDATE >> $MYFILENAME
grep: 8: No such file or directory
wait
echo '/var/log/maillog' >> $MYFILENAME
cat /var/log/maillog | $GREP $LOGDATE >> $MYFILENAME 
grep: 8: No such file or directory
wait
echo '/var/log/apache/access_log' >> $MYFILENAME
cat /var/log/apache/access_log | $GREP $APACHEACDATA >> $MYFILENAME
Usage: grep [OPTION]... PATTERN [FILE]...
Try `grep --help' for more information.
wait
echo '/var/log/apache/error_log' >> $MYFILENAME
cat /var/log/apache/error_log | $GREP $LOGDATE >> $MYFILENAME
grep: 8: No such file or directory
wait
echo 'dmesg' >> $MYFILENAME
dmesg >> $MYFILENAME
wait
echo '/var/log/secure' >> $MYFILENAME
cat /var/log/secure | $GREP $LOGDATE >> $MYFILENAME
grep: 8: No such file or directory
wait
echo '/var/log/syslog' >> $MYFILENAME
cat /var/log/syslog | $GREP $LOGDATE >> $MYFILENAME
grep: 8: No such file or directory
wait
echo '/var/log/vsftpd.log' >> $MYFILENAME
cat /var/log/vsftpd.log | $GREP $LOGDATE >> $MYFILENAME
grep: 8: No such file or directory
wait
echo 'Sending Mail'
Sending Mail
mail -s "Sys Stats for Today" user < $MYFILENAME
#rm $MYFILENAME
If that helps in figuring out why it wont work... I know it has to do with the number 8 in the date... but why does it work if you simply put in in quotes outside this script?

Thanks,
DragonM15
 
Old 10-09-2007, 02:04 AM   #4
DragonM15
Member
 
Registered: Sep 2003
Location: USA
Distribution: Slackware (Multiple Versions)
Posts: 455

Original Poster
Rep: Reputation: 31
If you read this post before, I DID NOT fix the problem. I only eliminated the error messages by putting '''s around the $LOGDATE 's. It still wont output the data to the file. Any ideas?

Thanks,
DragonM15

Last edited by DragonM15; 10-09-2007 at 02:09 AM.
 
Old 10-09-2007, 03:31 AM   #5
colucix
Moderator
 
Registered: Sep 2003
Location: Bologna
Distribution: CentOS 6.5 OpenSuSE 12.3
Posts: 10,453

Rep: Reputation: 1941Reputation: 1941Reputation: 1941Reputation: 1941Reputation: 1941Reputation: 1941Reputation: 1941Reputation: 1941Reputation: 1941Reputation: 1941Reputation: 1941
Without changing the structure of the script, a proper construct is
Code:
echo "/var/log/secure" >> $MYFILENAME
$GREP "$LOGDATE" /var/log/secure >> $MYFILENAME
Piping the output from cat to the grep command is unuseful because grep itself is able to process the content of a file (not only the standard input as the one coming from the cat command).
The errors you obtain are related to the interpretation of the LOGDATE and APACHEACDATE variables. As you already discovered the quotes are useful to let the shell interpret a string literally, despite of the presence of special characters or blank spaces, but you should put every variable inside quotes when you use it. In other words, it is not enough to put quotes only in the variable assigments at the beginning of the script. Every time the shell encounter a variable, expands it whitout preserving the surrounding quotes. So, the line
Code:
cat /var/log/secure | $GREP $LOGDATE >> $MYFILENAME
is expanded as
Code:
cat /var/log/secure | /usr/bin/grep Oct  8 >> /home/user/systats
where the grep command tries to look for the pattern "Oct" in the file "8". Instead, in the following line you have another typo
Code:
cat /var/log/apache/access_log | $GREP $APACHEACDATA >> $MYFILENAME
since you defined APACHEACDATE at the beginning. In this case the unassigned APACHEACDATA is expanded as a null string and the resulting command is grep without any pattern to search.
Just a note about debugging: executing the script by "bash -x" will give the execution trace with all the variable expansions performed by the shell. The output can be difficult to read, but it gives many more debugging info. Another hint is to look for differences in the usage of single quotes ' and double quotes ".

Last edited by colucix; 10-09-2007 at 03:33 AM.
 
Old 10-09-2007, 10:28 AM   #6
custangro
Senior Member
 
Registered: Nov 2006
Location: California
Distribution: Fedora , CentOS , Solaris 10, RHEL
Posts: 1,933
Blog Entries: 1

Rep: Reputation: 188Reputation: 188
As stated above you should run the script with bash -x scriptname.sh and you will see a lot of errors (if there is any). I used too think that using the -x option didn't help me much...but now I use it all the time for debugging...


-custangro
ps I like using ksh...just my preferance...
 
Old 10-09-2007, 10:44 AM   #7
druuna
LQ Veteran
 
Registered: Sep 2003
Posts: 10,532
Blog Entries: 7

Rep: Reputation: 2371Reputation: 2371Reputation: 2371Reputation: 2371Reputation: 2371Reputation: 2371Reputation: 2371Reputation: 2371Reputation: 2371Reputation: 2371Reputation: 2371
Hi,

There should be double quotes around the $LOGDATE part.

This: cat /var/log/apache/error_log | $GREP $LOGDATE >> $MYFILENAME

Should be: cat /var/log/apache/error_log | $GREP "$LOGDATE" >> $MYFILENAME

Grep sees this: grep oct 09 infile as grep oct in files 09 and infile, so the oct 09 part should be: grep "oct 09" infile.

Hope this helps.
 
Old 10-09-2007, 09:19 PM   #8
DragonM15
Member
 
Registered: Sep 2003
Location: USA
Distribution: Slackware (Multiple Versions)
Posts: 455

Original Poster
Rep: Reputation: 31
Worked like a charm colucix, thanks for the help...
 
Old 10-10-2007, 08:01 AM   #9
bigearsbilly
Senior Member
 
Registered: Mar 2004
Location: england
Distribution: FreeBSD, Debian, Mint, Puppy
Posts: 3,276

Rep: Reputation: 170Reputation: 170
cat file file2 file3 | mailx -vs "logs" user@blah.com
 
Old 10-10-2007, 08:34 AM   #10
matthewg42
Senior Member
 
Registered: Oct 2003
Location: UK
Distribution: Kubuntu 12.10 (using awesome wm though)
Posts: 3,530

Rep: Reputation: 62
The wait statements in the OP are not necessary. wait will pause until background jobs have completed, but you are not running the cat commands in the background.

i.e. this might make sense:
Code:
command1 &
command2 &
wait
command3
...it will allow command1 and command2 to operate in parallel, and the wait statement will halt the progress of the script until they are both done, and only then proceed with command3.

In the case where you are appending files somewhere, running in parallel is probably not what you want to do, unless you want all the data mixed up.

Something else which may be of interest to you: the more command, when called with multiple input files named as arguments, and re-directed to a file will prepend the content of each file with
Code:
::::::::::::::
filename
::::::::::::::
So, as long as you don't mind the ::::::: bits, you can accomplish much the same thing as the intent of the OP, just like this:
Code:
more /var/log/secure /var/log/messages /var/log/maillog ... > "$MYFILENAME"
The shell also allows for some flexibility in formatting of your code - you can put a backslash (\) as the last character on a line, and the shell will ignore the line break. This can greatly help the readability and maintainability of your scripts. For example, the command above is equivalent to this:
Code:
more /var/log/secure \
     /var/log/messages \
     /var/log/maillog \
     ... \
     > "$MYFILENAME"
Note: for this to work there must be no characters after the \ even spaces or tabs.
 
Old 10-10-2007, 09:56 AM   #11
Quigi
Member
 
Registered: Mar 2003
Location: Cambridge, MA, USA
Distribution: Ubuntu (Dapper and Heron)
Posts: 377

Rep: Reputation: 31
Quote:
Originally Posted by druuna View Post
There should be double quotes around the $LOGDATE part.
Yup, Colucix suggested that before. Plus a preferable invocation (saving the waste of typing and forking 'cat').

DragonM15,
to make the script more generic, you may want to generate the date you're looking for, e.g.,
Code:
LOGDATE=`date -d yesterday '+%b %e'`
This way you shouldn't need to change it daily.

Also beware that when the log is rolled, you might miss a part. E.g., my /var/log/messages gets rolled every Sunday at 4:02; presumably the FC2 default. In that case, some relevant messages may be in /var/log/messages.1. You could add that as another argument to grep , e.g.,
Code:
$GREP -h "^$LOGDATE" /var/log/messages.1 /var/log/messages >> $MYFILENAME
I'd put it as the first file, right after the pattern, to preserve chronological order.

If you're willing to change the structure, you could reduce code duplication by using a loop (and curly braces):
Code:
for log in /var/log/{secure,messages,maillog,apache/error_log,syslog}; do
  echo $log >> $MYFILE
  grep -h "$LOGDATE" $log{.1,} >> $MYFILE
done
The apache access log and dmesg would have to be done separately.


BTW, if you'd like a ready-made solution, look at LogWatch. It's running on my machine, and it does a decent job of summarizing all noteworthy log entries in a daily e-mail.

Last edited by Quigi; 10-10-2007 at 09:59 AM.
 
Old 10-10-2007, 10:33 AM   #12
DragonM15
Member
 
Registered: Sep 2003
Location: USA
Distribution: Slackware (Multiple Versions)
Posts: 455

Original Poster
Rep: Reputation: 31
Quote:
Originally Posted by bigearsbilly View Post
cat file file2 file3 | mailx -vs "logs" user@blah.com
This wouldnt do what I want it to do.... yes it would give me the log files, but it would give me ALL of the log files... I just want it a day at a time. Isn't that an organized way of doing things?

Thanks,
DragonM15
 
Old 11-07-2007, 11:10 AM   #13
archtoad6
Senior Member
 
Registered: Oct 2004
Location: Houston, TX (usa)
Distribution: MEPIS, Debian, Knoppix,
Posts: 4,727
Blog Entries: 15

Rep: Reputation: 230Reputation: 230Reputation: 230
Did you read Quigi's post?

This might work, too:
Code:
F=<your_output_file>
D=<your_chosen_date> 
more /var/log/{secure,messages,maillog,apache/error_log,syslog}{.0,} |egrep "(^/|^:|^$D)" >$F
Notes:
  • I used single letter variables for brevity, you can make yours as long as you like.
  • You cannot put variables inside a brace expansion.
  • On my system the (not-yet-compressed) previous log file file has a '.0' extension.

Last edited by archtoad6; 12-13-2007 at 02:50 PM.
 
Old 11-08-2007, 03:27 AM   #14
bigearsbilly
Senior Member
 
Registered: Mar 2004
Location: england
Distribution: FreeBSD, Debian, Mint, Puppy
Posts: 3,276

Rep: Reputation: 170Reputation: 170
Quote:
Originally Posted by DragonM15 View Post
This wouldnt do what I want it to do.... yes it would give me the log files, but it would give me ALL of the log files... I just want it a day at a time. Isn't that an organized way of doing things?

Thanks,
DragonM15
well thanks for the appreciation.

Awfiully sorry I haven't got time to sit there and work out a perfect solution,
but it was a pointer to an easy way to mail log files.
surely you could work out how to limit it to
today's log file only?

phew
 
  


Reply


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off


Similar Threads
Thread Thread Starter Forum Replies Last Post
Need bash script to list files, drop extension and dump to file talwar_ Programming 10 06-03-2011 09:18 AM
Bash script for server log (namely var/log/messages) tenaciousbob Programming 17 05-24-2007 10:43 AM
Automated build and email log in bash sardaukar_siet Linux - Server 1 03-08-2007 11:59 AM
Bash script to email setuid root files deoren LinuxQuestions.org Member Success Stories 1 01-30-2005 09:56 AM
Bash put script jnusa Programming 3 09-07-2004 02:31 AM


All times are GMT -5. The time now is 10:14 AM.

Main Menu
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
identi.ca: @linuxquestions
Facebook: linuxquestions Google+: linuxquestions
Open Source Consulting | Domain Registration