Help with Bash script - email new error messages from several log
 

Hi,

This is completely new for me (linux and bash noob), but i got the task to write a script to monitor errors or anything that failed from several logs and for every log email separately its newly arrived errors/failures to a central administration emailaddress.

For example for the log of nginx webserver.
I got something like:


#!/bin/bash
SEARCH=GET
SOURCE='/var/log/nginx/access.log'

function GetNewLogs() {
tail -n0 -F $SOURCE | grep --line-buffered $SEARCH
}

GetNewLogs | mail -s "nginx errors hostname" someemail@domain.com


I save this as test.sh and run it in a terminal like ./test.sh

This shell script needs to be scheduled and running all the time even after a reboot.

How can i accomplish this? I had tried to let the function write to a file first and then cat the file to pass to the mail command but it just hangs in the prompt. I am creating the script in a PUTTY window connected to Linux machine.

Thanks for any help.

 

There are a number of ways to approach this, depending on the specifics, so what precisely have you been asked to do, and in what context?

For example, if you're monitoring web applications, having a simple error handler within the application that logs to an API of separate bug logging software may be preferable than scraping logs. In particular it means you don't need to waste effort re-implementing logic of "first time error" or "existing error occurring more frequently" and similar, because it's already been written and tested, and can provide a bunch of other useful features (especially if you have multiple applications/servers).

If you must do it as a Bash/shell script, I'd still look for existing solutions - this sort of task will have a bunch of edge cases (i.e. potential bugs) that an established script will have discovered and solved already.

(If/when you do write Bash scripts, ShellCheck is a useful utility - it can't report all errors, but does highlight some common mistakes.)

I managed this example. can be helpful - it is one-liner
all the time it test output of ping command for pattern "12", once pattern is found, you obtain message "found" on terminal.

You know there are programs (like the so-called ELK stack) that do this, right?

Anyway, you’d use cron to schedule it, you’d use “mail” or msmtp (depending on whether you need an smtp server) to send email, and you’d need to persist the last line you’d read in the file the previous time (probably by redirecting “wc” output to a file).

Also be aware that you have the ability to monitor files for changes (using inotify and programs built on top of it).

Thanks all for helping.

Actually we want to monitor some server/application logs for errors/warnings continuously.
As soon an error/warning occurs in a log, the new error/warning from the log since last email, should be send to a mailbox.
Things should be kept simple with a bash script (maybe per log) that monitor the corresponding log file and attach to a cron job. I am aware of existing tools, but we aren't using that in the company.

As mentioned before i am testing the script in a terminal but what i don't understand is why it keeps hanging at the prompt after following statement (see comments)

tail -n0 -F test.log | grep --line-buffered error >messagebody.txt ## <- why does it hang here??
mail -s "subject of mail" email@adres.com <messagebody.txt


I see it creates the messagebody.txt file with contents but the prompt hangs in the terminal like its waiting for input. Why does it not continue to mail command?

Simple is using existing, proven and well-tested tools.


Why are you expecting it not to? Read the documentation for tail's follow option.


(p.s. In future, you should use [code] tags for code, not italics.)

Here's a previous brainstorm...

Best way to follow a log and execute a command when some text appears in the log

On the base of that link I managed such command
well it works. If grep matches 38 in output of ping echo outputs "found 38". In case of e-email notification I guess this may work
instead dots ... put correct format of mail command. Problem is how to be sure program is still running? You need to trap such situation. Of course it is good to rethink why it works.

Edit: This more fancy solution. I always was curious about coproc command in bash. Now I have some glimpse how it works
comparing to above leftmost part of pipe is now executed in sub-shell. But there is two-way communication through file descriptors ${COPROC[0]} - for output of grep, ${COPROC[1]} for input - in this case is redirected to <(ping www.linuxquestions.org). Another cool stuff for sure useful. But at this moment I don't have good idea for application.