LinuxQuestions.org
Latest LQ Deal: Latest LQ Deals
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - Software
User Name
Password
Linux - Software This forum is for Software issues.
Having a problem installing a new program? Want to know which application is best for the job? Post your question in this forum.

Notices


Reply
  Search this Thread
Old 02-27-2013, 09:52 PM   #1
designator
Member
 
Registered: Jun 2003
Location: California, USA
Distribution: OpenSUSE Tumbleweed
Posts: 219

Rep: Reputation: 37
Question Monitoring file and running commands based on content


I'm trying to tail a log file continuously and run commands based on tailing lines written to the log.

For example:
If the last 3 lines are:
Sample log message 1
Connection from IP: 192.168.1.1
Sample log message 3

Unfortunately, this just sits there without any output:
Code:
tail -f /var/log/messages | grep IP | awk '{print $4}' | xargs echo
But it works without xargs
Code:
tail -f /var/log/messages | grep IP | awk '{print $4}'
Google suggested this has something to do with the IO buffer so I tried a bunch of buffer controls like adding --line-buffered to grep and running with
Code:
stdbuf -o0 tail ...
but nothing works.

What's the trick? Is there no way to grep and tokenize "tail -f" and then run commands?
 
Old 02-27-2013, 11:19 PM   #2
GlennsPref
Senior Member
 
Registered: Apr 2004
Location: Tweed Heads, Australia
Distribution: Devuan
Posts: 3,415
Blog Entries: 33

Rep: Reputation: 222Reputation: 222Reputation: 222
Hi,

my system uses dmesg, rather than messages.

/var/log/dmesg

hth, Glenn
 
Old 02-28-2013, 11:23 AM   #3
designator
Member
 
Registered: Jun 2003
Location: California, USA
Distribution: OpenSUSE Tumbleweed
Posts: 219

Original Poster
Rep: Reputation: 37
Quote:
Originally Posted by GlennsPref View Post
my system uses dmesg, rather than messages.
I guess I wasn't clear. The log isn't the problem since the messages correctly show up there. The problem is after being piped through several filters that just retrieve the variable I need, the data gets stuck somewhere (likely the i/o buffer) and nothing at all is displayed. You can repeat the same experiment with any other file and get the same results.
 
Old 02-28-2013, 12:54 PM   #4
unSpawn
Moderator
 
Registered: May 2001
Posts: 29,415
Blog Entries: 55

Rep: Reputation: 3594Reputation: 3594Reputation: 3594Reputation: 3594Reputation: 3594Reputation: 3594Reputation: 3594Reputation: 3594Reputation: 3594Reputation: 3594Reputation: 3594
'tail -f /var/log/messages | grep --line-buffered something' should just work. Try injecting a line ('man logger') and see if that triggers things. If that's not it try a lame loop:
Code:
tailf /var/log/messages|while read LINE; do echo "${LINE}"; done|awk '/IP:/ {print $4}'
or try a lame loop:
Code:
doSomething() { echo "$1"; }
tailf /var/log/messages | while read LINE; do LINE=(${LINE});
 [ "${LINE[2]}" = "IP:" ] && doSomething "${LINE[2]}"
done
or try something with a FIFO.
*Note 'grep IP | awk '{print $4}';' may be short for 'awk '/IP:/ {if($3 = IP:) print $4}';'.
*Also note you're re-inventing the wheel: there's already tools that grep a log for strings and perform actions (Swatch, alerttail, etc, etc. See Sourceforge, Freshmeat, etc, etc).
 
1 members found this post helpful.
Old 02-28-2013, 12:58 PM   #5
haertig
Senior Member
 
Registered: Nov 2004
Distribution: Debian, Ubuntu, LinuxMint, Slackware, SysrescueCD, Raspbian, Arch
Posts: 2,285

Rep: Reputation: 354Reputation: 354Reputation: 354Reputation: 354
I assume "| xargs echo" is just an example placeholder for another program to be used in real life, since echo'ing something that can be printed out directly from awk is useless.

I would run "man xargs", and look at the -L option and the -n option. xargs can "build up" a command line using multiple inputs, but you have control of that buffering with -L and -n

Last edited by haertig; 02-28-2013 at 01:00 PM.
 
1 members found this post helpful.
Old 02-28-2013, 01:22 PM   #6
designator
Member
 
Registered: Jun 2003
Location: California, USA
Distribution: OpenSUSE Tumbleweed
Posts: 219

Original Poster
Rep: Reputation: 37
Quote:
Originally Posted by unSpawn View Post
'tail -f /var/log/messages | grep --line-buffered something' should just work.
That works, it gets stuck when I add the next pipe.

Quote:
Originally Posted by unSpawn View Post
*Also note you're re-inventing the wheel: there's already tools that grep a log for strings and perform actions (Swatch, alerttail, etc, etc. See Sourceforge, Freshmeat, etc, etc).
I was trying to keep it GNU, but I'll probably just use alerttail since it seems to do exactly what I need.

Quote:
Originally Posted by haertig View Post
I assume "| xargs echo" is just an example placeholder for another program to be used in real life, since echo'ing something that can be printed out directly from awk is useless.
Yes, the command will build iptables rules so echo is just an example.

Quote:
Originally Posted by haertig View Post
I would run "man xargs", and look at the -L option and the -n option. xargs can "build up" a command line using multiple inputs, but you have control of that buffering with -L and -n
I also tried 'xargs -n1 -L1' without success.
 
1 members found this post helpful.
Old 02-28-2013, 01:44 PM   #7
haertig
Senior Member
 
Registered: Nov 2004
Distribution: Debian, Ubuntu, LinuxMint, Slackware, SysrescueCD, Raspbian, Arch
Posts: 2,285

Rep: Reputation: 354Reputation: 354Reputation: 354Reputation: 354
Quote:
Originally Posted by designator View Post
Yes, the command will build iptables rules so echo is just an example.
Ahhh. Sounds like you might be doing something similar to what I did a few years ago. Based on failed login attempts (by the bad guys), I block their IP's by dynamically configuring against them.

If this is indeed what you're doing, here's a link to a post I did on the script several years ago. Might give you some ideas.

http://www.linuxquestions.org/questi...6/#post2290296
 
1 members found this post helpful.
Old 02-28-2013, 03:09 PM   #8
unSpawn
Moderator
 
Registered: May 2001
Posts: 29,415
Blog Entries: 55

Rep: Reputation: 3594Reputation: 3594Reputation: 3594Reputation: 3594Reputation: 3594Reputation: 3594Reputation: 3594Reputation: 3594Reputation: 3594Reputation: 3594Reputation: 3594
Quote:
Originally Posted by haertig View Post
Might give you some ideas.
Ideas cool, but there's really no need to use the script as fail2ban can already do all of that including rule management. Since two years before that post was made.
 
1 members found this post helpful.
Old 02-28-2013, 03:21 PM   #9
haertig
Senior Member
 
Registered: Nov 2004
Distribution: Debian, Ubuntu, LinuxMint, Slackware, SysrescueCD, Raspbian, Arch
Posts: 2,285

Rep: Reputation: 354Reputation: 354Reputation: 354Reputation: 354
Quote:
Originally Posted by unSpawn View Post
Ideas cool, but there's really no need to use the script as fail2ban can already do all of that including rule management. Since two years before that post was made.
And I also wrote that script long before that post was made. Back in that post in 2006 I was posting what I had done in the past. Anyway, I was not aware of fail2ban at that time I posted that back in 2006. And I don't know if fail2ban even existed when I actually wrote the script, which was before that 2006 post. I would certainly have considered using fail2ban as an alternate, had I known about it. Definitely no need to reinvent the wheel !
 
Old 02-28-2013, 03:28 PM   #10
designator
Member
 
Registered: Jun 2003
Location: California, USA
Distribution: OpenSUSE Tumbleweed
Posts: 219

Original Poster
Rep: Reputation: 37
Before this thread gets off topic, I'm not building a clone of fail2ban. I'm trying to script a process where a list of specific of hosts are only allowed to connect once so after a successful connection, a firewall rule that blocks them is executed.
 
Old 02-28-2013, 03:40 PM   #11
unSpawn
Moderator
 
Registered: May 2001
Posts: 29,415
Blog Entries: 55

Rep: Reputation: 3594Reputation: 3594Reputation: 3594Reputation: 3594Reputation: 3594Reputation: 3594Reputation: 3594Reputation: 3594Reputation: 3594Reputation: 3594Reputation: 3594
Quote:
Originally Posted by haertig View Post
I would certainly have considered using fail2ban as an alternate, had I known about it. Definitely no need to reinvent the wheel!
Likewise I don't have anything against your script, it's approach or showing off mad scripting skills, and my reply shouldn't be read as somehow implying you have or have had a wheel re-invention fetish ;-p
 
Old 02-28-2013, 03:42 PM   #12
unSpawn
Moderator
 
Registered: May 2001
Posts: 29,415
Blog Entries: 55

Rep: Reputation: 3594Reputation: 3594Reputation: 3594Reputation: 3594Reputation: 3594Reputation: 3594Reputation: 3594Reputation: 3594Reputation: 3594Reputation: 3594Reputation: 3594
Quote:
Originally Posted by designator View Post
I'm trying to script a process where a list of specific of hosts are only allowed to connect once so after a successful connection, a firewall rule that blocks them is executed.
Sounds like a case for Netfilter, maybe something like the "recent" module.
 
1 members found this post helpful.
Old 02-28-2013, 04:20 PM   #13
designator
Member
 
Registered: Jun 2003
Location: California, USA
Distribution: OpenSUSE Tumbleweed
Posts: 219

Original Poster
Rep: Reputation: 37
In my case, it is much more efficient to monitor a log file instead of network traffic. Firewalls are not aware of successful authentication, etc. Firewall rules will only be used to block hosts after the first successful connection.
 
Old 02-28-2013, 05:25 PM   #14
designator
Member
 
Registered: Jun 2003
Location: California, USA
Distribution: OpenSUSE Tumbleweed
Posts: 219

Original Poster
Rep: Reputation: 37
Figured it out. awk can run commands without having to pipe them to xargs:

Example log from /var/log/secure:
Code:
Feb 28 12:07:34 Logger sshd[14683]: Accepted password for user from 192.168.1.1 port 56534 ssh2
Example to find a successfully authenticated connection from a specific user and run a command with the user's IP address:
Code:
tail -f /var/log/secure|awk '{if($9 == "user") system("echo " $11)}'
The above will echo the IP address.

Thanks to everyone for help and suggestions!
 
Old 03-01-2013, 03:20 AM   #15
GlennsPref
Senior Member
 
Registered: Apr 2004
Location: Tweed Heads, Australia
Distribution: Devuan
Posts: 3,415
Blog Entries: 33

Rep: Reputation: 222Reputation: 222Reputation: 222
Hi, I responding to the OP's response in #3
Quote:
I guess I wasn't clear. The log isn't the problem since the messages correctly show up there. The problem is after being piped through several filters that just retrieve the variable I need, the data gets stuck somewhere (likely the i/o buffer) and nothing at all is displayed. You can repeat the same experiment with any other file and get the same results.
I got it to grep and awk, but not at the same time, the pipe through "xargs echo" never got me any response.

I had no "IP" to grep so I used sda (harddrive) and awk'ed different collumns.

No prob, before that, I was grepping and awking a file that didnot exist and got no response(errors) from the console,

leading me to think it was woking, just no string matches.

That's what I meant, (on my system) Grepping and awking a non-existent file gave no output.

My mistake, I'll read the rest now.

Glenn

You got me, I'm still learning.

I know there are a few reasons why it wouldn't work, but just for instance, there are numerous greps (egrep, rgrep..) that work well at specific lookup ranges. Many variables, ...

Last edited by GlennsPref; 03-01-2013 at 04:20 AM. Reason: Glad you got it to work.
 
  


Reply


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off



Similar Threads
Thread Thread Starter Forum Replies Last Post
Remove many lines based on content of other file anderssk Programming 13 06-23-2011 11:34 AM
Running an exe file on a dedicated virtual server at media temple linux based. abasquare Linux - Newbie 2 01-09-2011 07:20 PM
delete a file based on content and date ElectroLinux Linux - Newbie 7 08-19-2009 08:01 AM
Help to sync files based on content nonoitall Linux - Software 2 01-15-2009 04:48 AM
Routing based on Content?? Roddles Linux - Networking 6 01-09-2007 10:29 PM

LinuxQuestions.org > Forums > Linux Forums > Linux - Software

All times are GMT -5. The time now is 07:05 AM.

Main Menu
Advertisement
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
Open Source Consulting | Domain Registration