Stoopid n00b piping question

jholovacs · 01-12-2009, 12:56 PM

I have a set of firewalls that I am dropping to syslog on a linux box, and I want to watch the events realtime. So I do something like this:

Code:

tail -f /var/log/fw01.log &
tail -f /var/log/fw02.log &

...et voila! All my logged events pop up on my screen in realtime (or a close approximation thereof), and I'm happy.

Well, sort of. Problem is, I can't tell which device the messages are coming from, and I need to be looking at them at the same time.

What I'd like to do is add "fw01:" or "fw02:" to all the stuff that shows up on the screen. Can I do that? It seems to me that should be an easy piping operation but I can't wrap my brain around it.

indienick · 01-12-2009, 01:07 PM

I would suggest using the "watch" command.

It will show you what command you are "watching" as it runs. For example, I use it to watch my /proc/mdstat file for monitoring RAID activity:

Code:

$ watch -n1 cat /proc/mdstat

The "-n1" sets the refresh interval to 1 second.

junpa · 01-12-2009, 01:09 PM

no need to use a pipe.

Code:

tail -f /var/log/fw01.log /var/log/fw02.log

you can add the ampersand as well if that's how you like it

Code:

tail -f /var/log/fw01.log /var/log/fw02.log &

to add fw01: and fw02: to the output take a look at --log-prefix in the iptables man page.

basically:

Code:

iptables -I INPUT 1 -j LOG --log-prefix="fw01: "

beware that you will need to add it to each chain that you want to have the prefix.

e.g. OUTPUT/FORWARD and any custom chains you or your distro may have created (RH-Firewall-1-INPUT)

jholovacs · 01-12-2009, 01:46 PM

OK, thanks for the info, but neither of these is what I'm looking to do here.

indienick, your solution still does not specify which input is coming from which host, which is the point of this question.

junpa, I really don't want to mess around with iptables, and I do not want the logs themselves to record fw01:<whatever> and fw02:<whatever>. I just want the output displayed on the screen with these identifiers.

It seems there *should* be a way to just preface every line coming through tail's stdout with a text identifier for easy viewing. The desired result is something like this:

Code:

fw01: <date/ time> packet(xyz) from IP <abc> to IP <ghj> port <d> session <e> allowed.
fw02: <date/ time> packet(xyz) from IP <abc> to IP <ghj> port <d> session <e> allowed.
fw01: <date/ time> packet(xyz) from IP <abc> to IP <ghj> port <d> session <e> allowed.
fw02: <date/ time> packet(xyz) from IP <abc> to IP <ghj> port <d> session <e> dropped - Rule 21.
fw01: <date/ time> packet(xyz) from IP <abc> to IP <ghj> port <d> session <e> dropped - Rule 4.

This is what I want to see on my screen. I'm already getting the data from both in realtime so I'm almost there, I just want to tell them apart.

Thanks

junpa · 01-12-2009, 02:03 PM

Code:

root@smurf(/var/log):# tail -f fw01.log | awk '{print"fw01: "$0}' &

Code:

root@smurf(/var/log):# tail -f fw02.log | awk '{print"fw02: "$0}' &

jholovacs · 01-12-2009, 02:36 PM

Hmm...

this seemed like a good answer but it's not putting anything to the screen.

junpa · 01-12-2009, 02:46 PM

It works on several test boxes for me.

what exactly did you type?

jholovacs · 01-12-2009, 02:55 PM

Code:

tail -f firewall-dmz.log | awk '{print"dmz: "$0}' &
tail -f firewall-sz.log | awk '{print"sz: "$0}' &

...where firewall-dmz.log is the logfile that's getting for the DMZ and firewall-sz.log is the one that's getting filled for the secure zone.

When I kill the process, I get a dump of all the stuff it was parsing. the format looks correct, it's just not showing it in realtime any more.

junpa · 01-12-2009, 03:52 PM

well it seems like your version of awk buffers stdout.

what distro are you running? maybe the output of:

awk -W usage
awk -W version

try:

Code:

tail -f firewall-dmz.log | awk -W interactive  '{print"dmz: "$0}' &

junpa · 01-12-2009, 04:05 PM

this should work the way you want

Code:

tail -f fw01.log  | sed -u 's/[^\n].*/fw01: &/' &
tail -f fw02.log  | sed -u 's/[^\n].*/fw02: &/' &

jholovacs · 01-12-2009, 04:51 PM

Code:

root@mail:~# awk -W usage
awk: vacuous option: -W Usage
root@mail:~# awk -W version
mawk 1.3.3 Nov 1996, Copyright (C) Michael D. Brennan

compiled limits:
max NF             32767
sprintf buffer      1020

I'm running ubuntu server 8.something.

That last post worked beautifully. Awesome, thanks. Now I have to ge research sed because I've never even heard of it before. Time to RTFM...