Firewall crashes

micxz · 07-21-2009, 09:15 PM

Hello'

One of my servers seem to have some kind of hickup I would like to figure out why the firewall crashes sometimes. This has happened couple times this month and would like to know if anyone else has this experience.

Background:
Linux version 2.6.18-128.1.10.el5 (mockbuild@builder16.centos.org) (gcc version 4.1.2 20080704 (Red Hat 4.1.2-44)) #1 SMP Thu May 7 10:39:21 EDT 2009

What happened:
- I get an email:
Subject: Cron <root@host> /sbin/service iptables restart >/dev/null
Body: iptables-restore: line 106 failed
(I think this is my problem but 106 in my iptable save file in sysconfig reads, "COMMIT")
- I login and check the loaded table rules. All that is listed is the dropped hosts from portsentry. None of my normal rules. (see log)
- I restart iptables & we're back to normal

Log reads (in order):

Code:

Jul 21 07:59:57 kaden kernel: ****Dropped TCP: IN=eth0 OUT=

"normal" THEN:

Code:

Jul 21 08:00:02 host kernel: Removing netfilter NETLINK layer.
Jul 21 08:00:02 host kernel: ip_tables: (C) 2000-2006 Netfilter Core Team

WTF' Firewall down (time stamp is off?)

Then portsentry lights up as the packets are getting through:

Code:

Jul 21 06:00:02 host portsentry[30932]: attackalert: TCP SYN/Normal scan from host: 88.244.82.90/88.244.82.90 to TCP port: 445
Jul 21 06:00:02 host portsentry[30932]: attackalert: Host 88.244.82.90 has been blocked via wrappers with string: "ALL: 88.244.82.90"
Jul 21 06:00:02 host portsentry[30932]: attackalert: Host 88.244.82.90 has been blocked via dropped route using command: "/sbin/iptables -I INPUT -s 88.244.8
2.90 -j DROP"
Jul 21 06:00:02 host portsentry[30932]: attackalert: External command run for host: 88.244.82.90 using command: "/bin/mail -s 'Portscan from 88.244.82.90 on 
port 445' root@host.mydomain.net < /dev/null"

The above goes on over and over for lots of different scanning machines. Infected windows probably as it's 445

Then this one I think this is me loading iptables (and the time stamp is off again?):

Code:

Jul 21 08:00:02 host kernel: Netfilter messages via NETLINK v0.30.

win32sux · 07-21-2009, 10:39 PM

Quote:

Originally Posted by micxz

Body: iptables-restore: line 106 failed
(I think this is my problem but 106 in my iptable save file in sysconfig reads, "COMMIT")

Are you sure there are no extra spaces or anything on that line? Those would cause an error.

What happens if you manually restore the saved configuration? Like:

Code:

iptables-restore < /etc/sysconfig/iptables

Hopefully you are able to reproduce the error in that way, making it easy to fix.

micxz · 07-21-2009, 10:50 PM

Quote:

Originally Posted by win32sux

Are you sure there are no extra spaces or anything on that line? Those would cause an error.

I just looked closer and I am sure there is no extra spaces before or after "COMMIT"

There is a comment after:
[root@host portsentry]# tail -2 /etc/sysconfig/iptables
COMMIT
# Completed on Tue Jun 9 23:46:45 2009

Quote:

Originally Posted by win32sux

What happens if you manually restore the saved configuration? Hopefully you are able to reproduce the error in that way, making it easy to fix.

No errors either way. Manual or script based:

Code:

[root@host portsentry]# iptables-restore < /etc/sysconfig/iptables
[root@host portsentry]# /etc/init.d/iptables restart
Flushing firewall rules:                                   [  OK  ]
Setting chains to policy ACCEPT: filter                    [  OK  ]
Unloading iptables modules:                                [  OK  ]
Applying iptables firewall rules:                          [  OK  ]
Loading additional iptables modules: ip_conntrack_ftp      [  OK  ]

The errors are random (well hopefully not literally random as there must be a reason) and I am only aware when cron fails to flush and load.

win32sux · 07-22-2009, 10:50 AM

I'm not familiar with PortSentry (I use psad instead), but could it perhaps be trying to load an iptables configuration from a file other than /etc/sysconfig/iptables? That might explain why when the problem occurs your own rules are absent.

unSpawn · 07-22-2009, 04:31 PM

// BTW Portsentry lacks functionality, is no longer developed, supported or unmaintained. Like my fellow moderator indicates you're not left without supported, maintained and better software.

micxz · 07-22-2009, 08:11 PM

PortSentry addes drop rule via iptables then every six hours flushes them out via cron job "service iptables restart".

I odd thing to me is when I manually run this command it works and should work when cron runs it. I do have the iptabes script to not save on stop or restart. I still would like to fix it but it looks like my best option to to research other means of protection.

Would the recommendation be "psad"? Once again Thanks guys/gals'

win32sux · 07-22-2009, 10:18 PM

Quote:

Originally Posted by micxz

PortSentry addes drop rule via iptables then every six hours flushes them out via cron job "service iptables restart".

That's kinda weird, no? I mean, every six hours you lose your entire blacklist?

Quote:

I odd thing to me is when I manually run this command it works and should work when cron runs it. I do have the iptabes script to not save on stop or restart. I still would like to fix it but it looks like my best option to to research other means of protection.

Well, migrating to something maintained is definitely a good idea. It would still be nice to figure out just what exactly is causing this behavior, though. What does your crontab look like? BTW, what happened to the time stamps?

Quote:

Would the recommendation be "psad"? Once again Thanks guys/gals'

Sure, I can recommend it. I've been a very satisfied user for while.

micxz · 07-22-2009, 10:55 PM

Quote:

Originally Posted by win32sux

That's kinda weird, no? I mean, every six hours you lose your entire blacklist?

Actually I forgot about the blocked.udp and blocked.tcp files it maintains. As well as history, ignore and modes files.
crontab looks like:

Code:

[root@xxx ~]# cat /etc/cron.d/portsentry 
# Flush the entries added by portsentry every 6 hours on the hour
#00 */6 * * *   root    /sbin/service ipchains restart >/dev/null
00 */6 * * *    root    /sbin/service iptables restart >/dev/null

Quote:

BTW, what happened to the time stamps?

This is still a real mystery to me as when I manually restart I see the correct timestamps. When I look back on the day in question it seems as if the first whole minute of 8 oclock was restamped when this crash occured.

And when I look through it's as if some parts of the kernel are logging 2 hours ahead here's a snippit:

Code:

Jul 21 04:21:44 xxx xinetd[2373]: EXIT: smtp status=0 pid=4130 duration=0(sec)
Jul 21 06:21:48 xxx kernel: ****Dropped TCP: IN=eth0 OUT= MAC=00:11:11:19:44:a5:00:1e:13:ca:xx:xx:xx:00 SRC=84.224.30.163 DST=xx.xx.xx.102 LEN=48 TOS=0x
00 PREC=0x00 TTL=118 ID=34333 DF PROTO=TCP SPT=3151 DPT=445 WINDOW=16384 RES=0x00 SYN URGP=0 
Jul 21 06:21:51 xxx kernel: ****Dropped TCP: IN=eth0 OUT= MAC=00:11:11:19:44:a5:00:1e:13:xx:xx:xx:08:00 SRC=84.224.30.163 DST=xx.xx.xx.102 LEN=48 TOS=0x
00 PREC=0x00 TTL=118 ID=34441 DF PROTO=TCP SPT=3151 DPT=445 WINDOW=16384 RES=0x00 SYN URGP=0 
Jul 21 04:21:53 xxx xinetd[2373]: START: smtp pid=4131 from=xx.xx.xx.244

micxz · 07-23-2009, 12:18 AM

OK it's only the packet filter kernel log entries that have the time stamp two hours ahead of everything else. Wierd'