[SOLVED] Monitoring file and running commands based on content

designator · 02-27-2013, 08:52 PM

I'm trying to tail a log file continuously and run commands based on tailing lines written to the log.

For example:
If the last 3 lines are:
Sample log message 1
Connection from IP: 192.168.1.1
Sample log message 3

Unfortunately, this just sits there without any output:

Code:

tail -f /var/log/messages | grep IP | awk '{print $4}' | xargs echo

But it works without xargs

Code:

tail -f /var/log/messages | grep IP | awk '{print $4}'

Google suggested this has something to do with the IO buffer so I tried a bunch of buffer controls like adding --line-buffered to grep and running with

Code:

stdbuf -o0 tail ...

but nothing works.

What's the trick? Is there no way to grep and tokenize "tail -f" and then run commands?

GlennsPref · 02-27-2013, 10:19 PM

Hi,

my system uses dmesg, rather than messages.

/var/log/dmesg

hth, Glenn

designator · 02-28-2013, 10:23 AM

Quote:

Originally Posted by GlennsPref

my system uses dmesg, rather than messages.

I guess I wasn't clear. The log isn't the problem since the messages correctly show up there. The problem is after being piped through several filters that just retrieve the variable I need, the data gets stuck somewhere (likely the i/o buffer) and nothing at all is displayed. You can repeat the same experiment with any other file and get the same results.

unSpawn · 02-28-2013, 11:54 AM

'tail -f /var/log/messages | grep --line-buffered something' should just work. Try injecting a line ('man logger') and see if that triggers things. If that's not it try a lame loop:

Code:

tailf /var/log/messages|while read LINE; do echo "${LINE}"; done|awk '/IP:/ {print $4}'

or try a lame loop:

Code:

doSomething() { echo "$1"; }
tailf /var/log/messages | while read LINE; do LINE=(${LINE});
 [ "${LINE[2]}" = "IP:" ] && doSomething "${LINE[2]}"
done

or try something with a FIFO.
*Note 'grep IP | awk '{print $4}';' may be short for 'awk '/IP:/ {if($3 = IP:) print $4}';'.
*Also note you're re-inventing the wheel: there's already tools that grep a log for strings and perform actions (Swatch, alerttail, etc, etc. See Sourceforge, Freshmeat, etc, etc).

haertig · 02-28-2013, 11:58 AM

I assume "| xargs echo" is just an example placeholder for another program to be used in real life, since echo'ing something that can be printed out directly from awk is useless.

I would run "man xargs", and look at the -L option and the -n option. xargs can "build up" a command line using multiple inputs, but you have control of that buffering with -L and -n

designator · 02-28-2013, 12:22 PM

Quote:

Originally Posted by unSpawn

'tail -f /var/log/messages | grep --line-buffered something' should just work.

That works, it gets stuck when I add the next pipe.

Quote:

Originally Posted by unSpawn

*Also note you're re-inventing the wheel: there's already tools that grep a log for strings and perform actions (Swatch, alerttail, etc, etc. See Sourceforge, Freshmeat, etc, etc).

I was trying to keep it GNU, but I'll probably just use alerttail since it seems to do exactly what I need.

Quote:

Originally Posted by haertig

I assume "| xargs echo" is just an example placeholder for another program to be used in real life, since echo'ing something that can be printed out directly from awk is useless.

Yes, the command will build iptables rules so echo is just an example.

Quote:

Originally Posted by haertig

I would run "man xargs", and look at the -L option and the -n option. xargs can "build up" a command line using multiple inputs, but you have control of that buffering with -L and -n

I also tried 'xargs -n1 -L1' without success.

haertig · 02-28-2013, 12:44 PM

Quote:

Originally Posted by designator

Yes, the command will build iptables rules so echo is just an example.

Ahhh. Sounds like you might be doing something similar to what I did a few years ago. Based on failed login attempts (by the bad guys), I block their IP's by dynamically configuring against them.

If this is indeed what you're doing, here's a link to a post I did on the script several years ago. Might give you some ideas.

http://www.linuxquestions.org/questi...6/#post2290296

unSpawn · 02-28-2013, 02:09 PM

Quote:

Originally Posted by haertig

Might give you some ideas.

Ideas cool, but there's really no need to use the script as fail2ban can already do all of that including rule management. Since two years before that post was made.

haertig · 02-28-2013, 02:21 PM

Quote:

Originally Posted by unSpawn

Ideas cool, but there's really no need to use the script as fail2ban can already do all of that including rule management. Since two years before that post was made.

And I also wrote that script long before that post was made. Back in that post in 2006 I was posting what I had done in the past. Anyway, I was not aware of fail2ban at that time I posted that back in 2006. And I don't know if fail2ban even existed when I actually wrote the script, which was before that 2006 post. I would certainly have considered using fail2ban as an alternate, had I known about it. Definitely no need to reinvent the wheel !

designator · 02-28-2013, 02:28 PM

Before this thread gets off topic, I'm not building a clone of fail2ban. I'm trying to script a process where a list of specific of hosts are only allowed to connect once so after a successful connection, a firewall rule that blocks them is executed.

unSpawn · 02-28-2013, 02:40 PM

Quote:

Originally Posted by haertig

I would certainly have considered using fail2ban as an alternate, had I known about it. Definitely no need to reinvent the wheel!

Likewise I don't have anything against your script, it's approach or showing off mad scripting skills, and my reply shouldn't be read as somehow implying you have or have had a wheel re-invention fetish ;-p

unSpawn · 02-28-2013, 02:42 PM

Quote:

Originally Posted by designator

I'm trying to script a process where a list of specific of hosts are only allowed to connect once so after a successful connection, a firewall rule that blocks them is executed.

Sounds like a case for Netfilter, maybe something like the "recent" module.

designator · 02-28-2013, 03:20 PM

In my case, it is much more efficient to monitor a log file instead of network traffic. Firewalls are not aware of successful authentication, etc. Firewall rules will only be used to block hosts after the first successful connection.

designator · 02-28-2013, 04:25 PM

Figured it out. awk can run commands without having to pipe them to xargs:

Example log from /var/log/secure:

Code:

Feb 28 12:07:34 Logger sshd[14683]: Accepted password for user from 192.168.1.1 port 56534 ssh2

Example to find a successfully authenticated connection from a specific user and run a command with the user's IP address:

Code:

tail -f /var/log/secure|awk '{if($9 == "user") system("echo " $11)}'

The above will echo the IP address.

Thanks to everyone for help and suggestions!

GlennsPref · 03-01-2013, 02:20 AM

Hi, I responding to the OP's response in #3

Quote:

I guess I wasn't clear. The log isn't the problem since the messages correctly show up there. The problem is after being piped through several filters that just retrieve the variable I need, the data gets stuck somewhere (likely the i/o buffer) and nothing at all is displayed. You can repeat the same experiment with any other file and get the same results.

I got it to grep and awk, but not at the same time, the pipe through "xargs echo" never got me any response.

I had no "IP" to grep so I used sda (harddrive) and awk'ed different collumns.

No prob, before that, I was grepping and awking a file that didnot exist and got no response(errors) from the console,

leading me to think it was woking, just no string matches.

That's what I meant, (on my system) Grepping and awking a non-existent file gave no output.

My mistake, I'll read the rest now.

Glenn

You got me, I'm still learning.

I know there are a few reasons why it wouldn't work, but just for instance, there are numerous greps (egrep, rgrep..) that work well at specific lookup ranges. Many variables, ...