Help me regain control of my server logs! (squid logging)

Avatar · 11-01-2005, 10:03 AM

Hi there

This has been frustrating me for a while so I hope someone can help. I have a Mandrake machine kernel 2.4.18

I realise that In order to keep track of what is happening on your machine you need to review dmesg, /var/log/messages and /var/log/syslog.

However I use [edit] iptables [/edit] on this server, and I set the log level pretty high so I can keep tabs on my network and Internet usage.

Unfortunately, [edit] iptables [/edit] uses the system logs for logging, rendering them unusable for system administration. Here's an example of what "tail /var/log/messages" looks like:

Code:

[root@MDKSERV log]# tail /var/log/messages
Nov  1 11:02:13 MDKSERV kernel: passed IN=eth1 OUT= MAC=00:09:6b:63:15:4f:00:11:25:ab:73:f4:08:00 SRC=192.168.1.130 DST=192.168.1.1 LEN=92 TOS=0x00 PREC=0x00 TTL=128 ID=41065 DF PROTO=TCP SPT=2067 DPT=22 WINDOW=17436 RES=0x00 ACK PSH URGP=0
Nov  1 11:02:13 MDKSERV kernel: passed IN= OUT=eth1 SRC=192.168.1.1 DST=192.168.1.130 LEN=124 TOS=0x10 PREC=0x00 TTL=64 ID=61338 PROTO=TCP SPT=22 DPT=2067 WINDOW=8576 RES=0x00 ACK PSH URGP=0
Nov  1 11:02:13 MDKSERV kernel: passed IN=eth1 OUT= MAC=00:09:6b:63:15:4f:00:11:25:ab:73:f4:08:00 SRC=192.168.1.130 DST=192.168.1.1 LEN=92 TOS=0x00 PREC=0x00 TTL=128 ID=41066 DF PROTO=TCP SPT=2067 DPT=22 WINDOW=17352 RES=0x00 ACK PSH URGP=0
Nov  1 11:02:13 MDKSERV kernel: passed IN= OUT=eth1 SRC=192.168.1.1 DST=192.168.1.130 LEN=124 TOS=0x10 PREC=0x00 TTL=64 ID=35697 PROTO=TCP SPT=22 DPT=2067 WINDOW=8576 RES=0x00 ACK PSH URGP=0
Nov  1 11:02:13 MDKSERV kernel: passed IN=eth1 OUT= MAC=00:09:6b:63:15:4f:00:11:25:ab:73:f4:08:00 SRC=192.168.1.130 DST=192.168.1.1 LEN=92 TOS=0x00 PREC=0x00 TTL=128 ID=41067 DF PROTO=TCP SPT=2067 DPT=22 WINDOW=17268 RES=0x00 ACK PSH URGP=0
Nov  1 11:02:13 MDKSERV kernel: passed IN= OUT=eth1 SRC=192.168.1.1 DST=192.168.1.130 LEN=40 TOS=0x10 PREC=0x00 TTL=64 ID=24 PROTO=TCP SPT=22 DPT=2067 WINDOW=8576 RES=0x00 ACK URGP=0
Nov  1 11:02:13 MDKSERV kernel: passed IN=eth1 OUT= MAC=00:09:6b:63:15:4f:00:11:25:ab:73:f4:08:00 SRC=192.168.1.130 DST=192.168.1.1 LEN=92 TOS=0x00 PREC=0x00 TTL=128 ID=41068 DF PROTO=TCP SPT=2067 DPT=22 WINDOW=17268 RES=0x00 ACK PSH URGP=0
Nov  1 11:02:13 MDKSERV kernel: passed IN= OUT=eth1 SRC=192.168.1.1 DST=192.168.1.130 LEN=40 TOS=0x10 PREC=0x00 TTL=64 ID=0 PROTO=TCP SPT=22 DPT=2067 WINDOW=8576 RES=0x00 ACK URGP=0
Nov  1 11:02:13 MDKSERV kernel: passed IN= OUT=eth1 SRC=192.168.1.1 DST=192.168.1.130 LEN=124 TOS=0x10 PREC=0x00 TTL=64 ID=26991 PROTO=TCP SPT=22 DPT=2067 WINDOW=8576 RES=0x00 ACK PSH URGP=0
Nov  1 11:02:13 MDKSERV kernel: passed IN=eth1 OUT= MAC=00:09:6b:63:15:4f:00:11:25:ab:73:f4:08:00 SRC=192.168.1.130 DST=192.168.1.1 LEN=40 TOS=0x00 PREC=0x00 TTL=128 ID=41069 DF PROTO=TCP SPT=2067 DPT=22 WINDOW=17184 RES=0x00 ACK URGP=0

This is also what my dmesg output is. What I'd really like to know about is SSH sessions, boot messages, and other things.

How can I separate these two logs, iptables and kernel???? I want my system messages back.

Thanks!

fouldsy · 11-01-2005, 04:17 PM

Try looking at ulogd which is a separate daemon able to handle the logging of firewall rules separate from syslog. It also lets you manipulate the data a little more, though I've never actually used it myself.

Capt_Caveman · 11-01-2005, 09:21 PM

You may also want to try a log parser like logwatch or sawmill.

Avatar · 11-02-2005, 10:52 AM

Well I have a log parser that allows me to examine my internet usage data, but what I want is to be able to type "dmesg" and NOT have all that stuff come up. There are NO system messages in dmesg unless, you type it IMMEDIATELY and I mean immediately after bootup, before the network traffic gets going.

Unfortunately I'm not a linux "guru." I looked at ulogd but I didn't really understand it's purpose.

Right now my IPTABLES rules have this:

Code:

-j LOG --log-level warning --log-prefix "passed "
-j LOG --log-level info --log-prefix "dropped "

The first line is for ACCEPT and the second for DROP.

Is there a way to tell iptables to LOG somewhere else, but still compress and gzip the logs every day like it's doing now?

Capt_Caveman · 11-02-2005, 04:59 PM

In that case, you'll likely need a specialized logging daemon. The standard syslog doesn't really support complex logging like that and really the only way to do it is to have a script parse the log file and pull out the iptables log mesgs. It usually is kind of an ugly hack, so using a specialized logging utility (like ulogd or syslog-ng) is a better solution. Syslog-ng is basically a robust version of the normal system logger, while ulogd interfaces more directly with iptables by using the iptables ULOG target (instead of LOG) which hands the iptables log mesgs over to a set of userland utilities that process them instead of the normal kernel and system logging facility.

Avatar · 11-03-2005, 11:20 AM

Hi, it sounds like ULOGd is what I'm looking for.

But I didn't really understand its website very much. Would all I have to do is install it, then change my IPTABLES to read ULOG whereever it says LOG?

If it's not that easy, do you know of any user manuals or howtos for it.

Thanks!

Capt_Caveman · 11-03-2005, 12:58 PM

Dunno. Never actually used ulogd either. I'd start by taking a look at the README though

Avatar · 11-03-2005, 02:22 PM

Thanks for your help

Capt_Caveman · 11-03-2005, 07:54 PM

Sure, if you have any specific problems with the installation, let us know.