LinuxQuestions.org
Download your favorite Linux distribution at LQ ISO.
Go Back   LinuxQuestions.org > Forums > Non-*NIX Forums > Programming
User Name
Password
Programming This forum is for all programming questions.
The question does not have to be directly related to Linux and any language is fair game.

Notices


Reply
  Search this Thread
Old 07-02-2021, 02:37 AM   #1
Linux Rookie
LQ Newbie
 
Registered: Jul 2021
Posts: 2

Rep: Reputation: Disabled
Help with Bash script - email new error messages from several log


Hi,

This is completely new for me (linux and bash noob), but i got the task to write a script to monitor errors or anything that failed from several logs and for every log email separately its newly arrived errors/failures to a central administration emailaddress.

For example for the log of nginx webserver.
I got something like:


#!/bin/bash
SEARCH=GET
SOURCE='/var/log/nginx/access.log'

function GetNewLogs() {
tail -n0 -F $SOURCE | grep --line-buffered $SEARCH
}

GetNewLogs | mail -s "nginx errors hostname" someemail@domain.com


I save this as test.sh and run it in a terminal like ./test.sh

This shell script needs to be scheduled and running all the time even after a reboot.

How can i accomplish this? I had tried to let the function write to a file first and then cat the file to pass to the mail command but it just hangs in the prompt. I am creating the script in a PUTTY window connected to Linux machine.

Thanks for any help.
 
Old 07-02-2021, 06:26 AM   #2
boughtonp
Senior Member
 
Registered: Feb 2007
Location: UK
Distribution: Debian
Posts: 3,573

Rep: Reputation: 2534Reputation: 2534Reputation: 2534Reputation: 2534Reputation: 2534Reputation: 2534Reputation: 2534Reputation: 2534Reputation: 2534Reputation: 2534Reputation: 2534

There are a number of ways to approach this, depending on the specifics, so what precisely have you been asked to do, and in what context?

For example, if you're monitoring web applications, having a simple error handler within the application that logs to an API of separate bug logging software may be preferable than scraping logs. In particular it means you don't need to waste effort re-implementing logic of "first time error" or "existing error occurring more frequently" and similar, because it's already been written and tested, and can provide a bunch of other useful features (especially if you have multiple applications/servers).

If you must do it as a Bash/shell script, I'd still look for existing solutions - this sort of task will have a bunch of edge cases (i.e. potential bugs) that an established script will have discovered and solved already.

(If/when you do write Bash scripts, ShellCheck is a useful utility - it can't report all errors, but does highlight some common mistakes.)

 
Old 07-02-2021, 08:09 AM   #3
igadoter
Senior Member
 
Registered: Sep 2006
Location: wroclaw, poland
Distribution: many, primary Slackware
Posts: 2,717
Blog Entries: 1

Rep: Reputation: 625Reputation: 625Reputation: 625Reputation: 625Reputation: 625Reputation: 625
I managed this example. can be helpful - it is one-liner
Code:
$ while : ; do grep "12" <(ping www.google.com) > /dev/null && echo found 12 ; done
all the time it test output of ping command for pattern "12", once pattern is found, you obtain message "found" on terminal.
 
Old 07-02-2021, 08:26 AM   #4
dugan
LQ Guru
 
Registered: Nov 2003
Location: Canada
Distribution: distro hopper
Posts: 11,198

Rep: Reputation: 5307Reputation: 5307Reputation: 5307Reputation: 5307Reputation: 5307Reputation: 5307Reputation: 5307Reputation: 5307Reputation: 5307Reputation: 5307Reputation: 5307
You know there are programs (like the so-called ELK stack) that do this, right?

Anyway, you’d use cron to schedule it, you’d use “mail” or msmtp (depending on whether you need an smtp server) to send email, and you’d need to persist the last line you’d read in the file the previous time (probably by redirecting “wc” output to a file).

Also be aware that you have the ability to monitor files for changes (using inotify and programs built on top of it).

Last edited by dugan; 07-02-2021 at 09:23 AM. Reason: Added the anyway
 
Old 07-05-2021, 07:41 AM   #5
Linux Rookie
LQ Newbie
 
Registered: Jul 2021
Posts: 2

Original Poster
Rep: Reputation: Disabled
Thanks all for helping.

Actually we want to monitor some server/application logs for errors/warnings continuously.
As soon an error/warning occurs in a log, the new error/warning from the log since last email, should be send to a mailbox.
Things should be kept simple with a bash script (maybe per log) that monitor the corresponding log file and attach to a cron job. I am aware of existing tools, but we aren't using that in the company.

As mentioned before i am testing the script in a terminal but what i don't understand is why it keeps hanging at the prompt after following statement (see comments)

tail -n0 -F test.log | grep --line-buffered error >messagebody.txt ## <- why does it hang here??
mail -s "subject of mail" email@adres.com <messagebody.txt


I see it creates the messagebody.txt file with contents but the prompt hangs in the terminal like its waiting for input. Why does it not continue to mail command?
 
Old 07-05-2021, 10:39 AM   #6
boughtonp
Senior Member
 
Registered: Feb 2007
Location: UK
Distribution: Debian
Posts: 3,573

Rep: Reputation: 2534Reputation: 2534Reputation: 2534Reputation: 2534Reputation: 2534Reputation: 2534Reputation: 2534Reputation: 2534Reputation: 2534Reputation: 2534Reputation: 2534
Quote:
Originally Posted by Linux Rookie View Post
Things should be kept simple ...
Simple is using existing, proven and well-tested tools.


Quote:
why does it hang here??
...
I see it creates the messagebody.txt file with contents but the prompt hangs in the terminal like its waiting for input. Why does it not continue to mail command?
Why are you expecting it not to? Read the documentation for tail's follow option.


(p.s. In future, you should use [code] tags for code, not italics.)


Last edited by boughtonp; 07-05-2021 at 10:40 AM.
 
Old 07-18-2021, 07:31 PM   #7
dugan
LQ Guru
 
Registered: Nov 2003
Location: Canada
Distribution: distro hopper
Posts: 11,198

Rep: Reputation: 5307Reputation: 5307Reputation: 5307Reputation: 5307Reputation: 5307Reputation: 5307Reputation: 5307Reputation: 5307Reputation: 5307Reputation: 5307Reputation: 5307
Here's a previous brainstorm...

Best way to follow a log and execute a command when some text appears in the log

Last edited by dugan; 07-18-2021 at 07:33 PM.
 
1 members found this post helpful.
Old 07-18-2021, 08:22 PM   #8
igadoter
Senior Member
 
Registered: Sep 2006
Location: wroclaw, poland
Distribution: many, primary Slackware
Posts: 2,717
Blog Entries: 1

Rep: Reputation: 625Reputation: 625Reputation: 625Reputation: 625Reputation: 625Reputation: 625
On the base of that link I managed such command
Code:
$ grep --line-buffered "38" <(ping www.google.com) | 
while read ; do echo found 38 ; done
well it works. If grep matches 38 in output of ping echo outputs "found 38". In case of e-email notification I guess this may work
Code:
$ grep --line-buffered "error" <(tail -n0 -F syslog.log) | 
while read ; do mail ...... ; done
instead dots ... put correct format of mail command. Problem is how to be sure program is still running? You need to trap such situation. Of course it is good to rethink why it works.

Edit: This more fancy solution. I always was curious about coproc command in bash. Now I have some glimpse how it works
Code:
$ coproc grep  --line-buffered "38"  <(ping www.linuxquestions.org)
[2] 29186
$ while read  -u ${COPROC[0]} foo ; do echo $foo ; done
64 bytes from 104.24.136.8 (104.24.136.8): icmp_seq=38 ttl=52 time=55.7 ms
64 bytes from 104.24.136.8 (104.24.136.8): icmp_seq=81 ttl=52 time=1238 ms
comparing to above leftmost part of pipe is now executed in sub-shell. But there is two-way communication through file descriptors ${COPROC[0]} - for output of grep, ${COPROC[1]} for input - in this case is redirected to <(ping www.linuxquestions.org). Another cool stuff for sure useful. But at this moment I don't have good idea for application.

Last edited by igadoter; 07-19-2021 at 09:13 AM.
 
  


Reply


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off



Similar Threads
Thread Thread Starter Forum Replies Last Post
[SOLVED] iptables troubleshooting icmp and best place to log /var/log/messages or /var/log/iptables JockVSJock Linux - Security 18 02-12-2016 12:31 AM
[SOLVED] Central log server aggregating all messages to /var/log/messages lhiggie1 CentOS 6 01-20-2015 04:44 PM
Bash script for server log (namely var/log/messages) tenaciousbob Programming 17 05-24-2007 10:43 AM
From where am i getting error messages to /var/log/messages? prabhuacsp Programming 3 02-16-2005 08:59 AM
From where am i getting error messages to /var/log/messages? prabhuacsp Linux - Networking 1 02-16-2005 12:34 AM

LinuxQuestions.org > Forums > Non-*NIX Forums > Programming

All times are GMT -5. The time now is 04:57 PM.

Main Menu
Advertisement
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
Open Source Consulting | Domain Registration