[SOLVED] Fail2ban and Hiawatha

NeilR · 10-26-2010, 10:13 AM

I can't get a failregex to match my Hiawatha webserver logs! The log files use "Common Log Format" where the host IP is before the date, but I can't seem to match this with a failregex which wants the date before the IP. Have I missed something obvious?

A sample log entry is:

192.168.0.86 - - [26/Oct/2010:15:13:00 +0100] "GET / HTTP/1.1" 200 3659

unSpawn · 10-26-2010, 12:23 PM

Not exactly a security question but what is the regex you tried for the Hiawatha web server log and what does 'fail2ban-regex' return when you try it?

NeilR · 10-26-2010, 02:32 PM

Yes, wasn't sure where to ask this! I tried "<HOST>.*GET.*" just to get a match, but get reply "Found a match... but no valid date/time found..." from fail2ban-regex. I also tried various mutations of this regex but get either no match at all, or the found match/no date messages.

unSpawn · 10-27-2010, 11:54 AM

What does 'fail2ban-regex "192.168.0.86 - - [26/Oct/2010:15:13:00 +0100] \"GET / HTTP/1.1\" 200 3659" "(?P<host>\S*) [-/\d,.]+ .*GET";' return (post complete output)?

NeilR · 10-28-2010, 09:34 AM

Output follows...

/usr/share/fail2ban/server/filter.py:442: DeprecationWarning: the md5 module is deprecated; use hashlib instead
import md5

Running tests
=============

Use regex line : (?P<host>\S*) [-/\d,.]+ .*GET
Use single line: 192.168.0.86 - - [26/Oct/2010:15:13:00 +0100] "GET...

Results
=======

Failregex
|- Regular expressions:
| [1] (?P<host>\S*) [-/\d,.]+ .*GET
|
`- Number of matches:
[1] 1 match(es)

Ignoreregex
|- Regular expressions:
|
`- Number of matches:

Summary
=======

Addresses found:
[1]
192.168.0.86 (Tue Oct 26 15:13:00 2010)

Date template hits:
0 hit(s): MONTH Day Hour:Minute:Second
0 hit(s): WEEKDAY MONTH Day Hour:Minute:Second Year
0 hit(s): WEEKDAY MONTH Day Hour:Minute:Second
0 hit(s): Year/Month/Day Hour:Minute:Second
0 hit(s): Day/Month/Year Hour:Minute:Second
2 hit(s): Day/MONTH/Year:Hour:Minute:Second
0 hit(s): Month/Day/Year:Hour:Minute:Second
0 hit(s): Year-Month-Day Hour:Minute:Second
0 hit(s): Day-MONTH-Year Hour:Minute:Second[.Millisecond]
0 hit(s): Day-Month-Year Hour:Minute:Second
0 hit(s): TAI64N
0 hit(s): Epoch
0 hit(s): ISO 8601
0 hit(s): Hour:Minute:Second
0 hit(s): <Month/Day/Year@Hour:Minute:Second>

Success, the total number of match is 1

However, look at the above section 'Running tests' which could contain important information.

...Output ends

I have tested this and it works - thanks for the help, though I will have to study this to understand what is happening. Perhaps you could also point in the right direction for the standard Hiawatha log format?
They look like this:

212.183.140.31|Wed 27 Oct 2010 18:02:55 +0100|200|4136||POST / HTTP/1.1|Accept: */*|Accept-Encoding: gzip, deflate|Accep(blah,blah)

Once again, many thanks!

Neil.

unSpawn · 10-28-2010, 10:29 AM

Quote:

Originally Posted by NeilR

I have tested this and it works

Cool.

Quote:

Originally Posted by NeilR

point in the right direction for the standard Hiawatha log format?

Not a Linux Security question. Please see the Hiawatha manual for "LogFormat" (hiawatha|common|extended).

win32sux · 10-28-2010, 05:30 PM

I've moved the thread to Software.

NeilR · 10-29-2010, 02:20 AM

I thought I had some understanding of regexes but this has me totally stumped. I cannot work out a regex to match the Hiawatha standard log format, probably because I can't understand why the regex supplied by unSpawn works. Can you help - I'm feeling very stupid here.

unSpawn · 11-01-2010, 04:30 PM

I think your best bet is to look into the regexes as they are used in /etc/fail2ban/.*/.* and maybe an "easier" Hiawatha log format. Clearly I'm trying to avoid compiling Hiawatha to find out which log format would be more "default" than your current one but I'd venture it's "LogFormat=extended".

NeilR · 11-02-2010, 09:40 AM

I think I have figured it out!!! The Hiawatha default log date format is Day DD Mon YYYY:hh:mm:ss (or "%a %e %b %Y %H:%M:%S %z") which it seems fail2ban doesn't support.

Thanks to unSpawn for helping me along the path to understanding!

unSpawn · 11-07-2010, 09:29 AM

Quote:

Originally Posted by NeilR

which it seems fail2ban doesn't support.

If you're not going for "LogFormat=extended" then maybe hack fail2ban? Something like this may be a start?

Code:

--- /usr/share/fail2ban/server/datedetector.py  2008-02-27 22:44:55.000000000 +0100
+++ /usr/share/fail2ban/server/datedetect0r.py  2008-02-27 22:45:55.000000000 +0100
@@ -93,6 +93,12 @@
                        template = DateEpoch()
                        template.setName("Epoch")
                        self.__templates.append(template)
+                       # Hiawatha web server default log format
+                       template = DateStrptime()
+                       template.setName("Weekday Day Month Year Hour:Minute:Second TZ")
+                       template.setRegex("\S{3} \s{1,2}\S{3}\d{4} \d{2}:\d{2}:\d{2} \S{5}")
+                       template.setPattern("%a %e %b %Y %H:%M:%S %z")
+                       self.__templates.append(template)
                finally:
                        self.__lock.release()

Not as if I know Python though.