fail2ban filters

r.stiltskin · 04-21-2017, 10:21 PM

[Arch Linux, fail2ban 0.9.6]

The default fail2ban filters do find log entries containing strings like
"not allowed because not listed in AllowUsers"
and ban the offending ip addresses when appropriate, but they're not finding multiple occurrences of entries like

Code:

Connection closed by 76.164.165.233 port 50297 [preauth]

or

Code:

Did not receive identification string from 105.106.130.52 port 52759

or

Code:

Unable to negotiate with 40.86.181.208 port 55996: no matching key exchange method found. Their offer: diffie-hellman-group1-sha1 [preauth]

I find many occurrences of lines like those within the specified findtime of 600 seconds in the output of journalctl -t sshd, but they don't appear in the fail2ban log.

I'm not sure what those entries represent. (Needless to say, they're not from valid users of my server.) Is there a reason why filter rules to catch them are not provided?

ondoho · 04-22-2017, 01:30 AM

Quote:

Originally Posted by r.stiltskin

I find many occurrences of lines like those within the specified findtime of 600 seconds in the output of journalctl -t sshd, but they don't appear in the fail2ban log.

fail2ban is pretty simple - it parses log files (and i don't think it parses the systemd journal!) and looks for entries, and makes decisions based upon how often these entries appear and which IP the action originates from. configurable. has to be explicitely enabled for each application you're using.

so, 2 things to check:
- is a fail2ban configuration for sshd active?
- does sshd actually log to the files that fail2ban is looking for?

r.stiltskin · 04-22-2017, 09:22 AM

Sorry, I guess I didn't state my question clearly. I'll try again.

fail2ban is "working" as it is currently configured. It is monitoring sshd activity and does occasionally ban/unban an ip after a series of unsuccessful login attempts.

But in my logs I find sequences of other types of messages that I find concerning but which are not caught by any of the preconfigured fail2ban filters, such as:

Code:

Apr 21 22:23:07 darth sshd[20601]: Connection closed by 76.164.165.233 port 36709 [preauth]
Apr 21 22:25:22 darth sshd[20610]: Connection closed by 76.164.165.233 port 34574 [preauth]
Apr 21 22:27:36 darth sshd[20617]: Connection closed by 76.164.165.233 port 60747 [preauth]
Apr 21 22:29:51 darth sshd[20632]: Connection closed by 76.164.165.233 port 58626 [preauth]
Apr 21 22:32:06 darth sshd[20640]: Connection closed by 76.164.165.233 port 56484 [preauth]
Apr 21 22:34:21 darth sshd[20678]: Connection closed by 76.164.165.233 port 54383 [preauth]
Apr 21 22:36:36 darth sshd[20688]: Connection closed by 76.164.165.233 port 52297 [preauth]
Apr 21 22:38:51 darth sshd[20726]: Connection closed by 76.164.165.233 port 50297 [preauth]

and

Code:

Apr 21 15:01:45 darth sshd[18901]: Did not receive identification string from 105.106.130.52 port 52747
Apr 21 15:01:46 darth sshd[18902]: Did not receive identification string from 105.106.130.52 port 52754
Apr 21 15:01:48 darth sshd[18903]: Did not receive identification string from 105.106.130.52 port 52759
Apr 21 15:01:52 darth sshd[18906]: Did not receive identification string from 105.106.130.52 port 52775
Apr 21 15:02:00 darth sshd[18907]: Did not receive identification string from 105.106.130.52 port 52801

and

Code:

Apr 18 11:52:04 darth sshd[5571]: Unable to negotiate with 40.86.181.208 port 53698: no matching key exchange method found. Their offer: diffie-hellman-group1-sha1 [preauth]
Apr 18 11:52:05 darth sshd[5573]: Unable to negotiate with 40.86.181.208 port 53753: no matching key exchange method found. Their offer: diffie-hellman-group1-sha1 [preauth]
Apr 18 11:52:07 darth sshd[5575]: Unable to negotiate with 40.86.181.208 port 53802: no matching key exchange method found. Their offer: diffie-hellman-group1-sha1 [preauth]
Apr 18 11:52:08 darth sshd[5577]: Unable to negotiate with 40.86.181.208 port 53853: no matching key exchange method found. Their offer: diffie-hellman-group1-sha1 [preauth]
Apr 18 11:52:09 darth sshd[5579]: Unable to negotiate with 40.86.181.208 port 53900: no matching key exchange method found. Their offer: diffie-hellman-group1-sha1 [preauth]
Apr 18 11:52:10 darth sshd[5581]: Unable to negotiate with 40.86.181.208 port 53960: no matching key exchange method found. Their offer: diffie-hellman-group1-sha1 [preauth]
Apr 18 11:52:11 darth sshd[5583]: Unable to negotiate with 40.86.181.208 port 54009: no matching key exchange method found. Their offer: diffie-hellman-group1-sha1 [preauth]
Apr 18 11:52:12 darth sshd[5585]: Unable to negotiate with 40.86.181.208 port 54063: no matching key exchange method found. Their offer: diffie-hellman-group1-sha1 [preauth]
Apr 18 11:52:13 darth sshd[5587]: Unable to negotiate with 40.86.181.208 port 54117: no matching key exchange method found. Their offer: diffie-hellman-group1-sha1 [preauth]
Apr 18 11:52:14 darth sshd[5589]: Unable to negotiate with 40.86.181.208 port 54181: no matching key exchange method found. Their offer: diffie-hellman-group1-sha1 [preauth]

While they aren't exactly login attempts, they seem to be up to no good, but I don't understand exactly what their purpose is. So I guess I really have two questions:

Exactly what are these guys trying to do?
and
Is it important to write custom fail2ban filters to catch and ban them?

ondoho · 04-22-2017, 12:13 PM

they're probably trying to hack you.
do these IP addresses say anything to you?

Habitual · 04-22-2017, 12:22 PM

ssh used to be enabled by default.
Should double-check it.

Code:

fail2ban-regex /var/log/auth.log /etc/fail2ban/filter.d/ssh.conf

is the command-line method of testing a filter.

ondoho · 04-22-2017, 12:33 PM

Quote:

Originally Posted by Habitual

Code:

fail2ban-regex /var/log/auth.log /etc/fail2ban/filter.d/sshd.conf

for me it was sshd...

Habitual · 04-22-2017, 04:28 PM

Quote:

Originally Posted by ondoho

for me it was sshd...

Yer right. ssh would be the client side, and there's nothing to see there, move along.

d is for daemon and I missed it.

r.stiltskin · 04-22-2017, 05:52 PM

Quote:

Originally Posted by ondoho

they're probably trying to hack you.
do these IP addresses say anything to you?

Nope. As I said in post #1, they aren't legit users of my server. I'm just trying to understand exactly what kind of attacks they represent, assuming that they are attacks.

And also wondering why there are no filter rules to pick up these messages included in the fail2ban release, given that these messages are certainly not unusual.

Habitual · 04-22-2017, 08:09 PM

Quote:

Originally Posted by r.stiltskin

Nope. As I said in post #1, they aren't legit users of my server. I'm just trying to understand exactly what kind of attacks they represent, assuming that they are attacks.

And also wondering why there are no filter rules to pick up these messages included in the fail2ban release, given that these messages are certainly not unusual.

The usual "noise" for sshd, IMO. I consider it like "door bell ditch". It's annoying. Hopefully we can help.

There is one included and tons of examples, but it helps to understand the way it works first.

Code:

fail2ban-regex /var/log/auth.log /etc/fail2ban/filter.d/sshd-ddos.conf

I show /etc/fail2ban/filter.d/sshd-ddos.conf has a good starting point as it has at least one of our target strings.

Code:

grep -e identification -e closed -e exchange /etc/fail2ban/filter.d/*
/etc/fail2ban/filter.d/sshd-ddos.conf:#  "Did not receive identification string".
/etc/fail2ban/filter.d/sshd-ddos.conf:failregex = ^%(__prefix_line)sDid not receive identification string from <HOST>\s*$

Let's kick some tires!

Code:

fail2ban-regex /var/log/auth.log /etc/fail2ban/filter.d/sshd-ddos.conf

You may have to edit /etc/fail2ban/filter.d/sshd-ddos.conf manually and add some regular expressions and test until it hits,
which is all I did here below.
then enable the jail after showing that it works as desired.
It does require intervention, but once you have it "dialed in", you will love it and rely on it as I do.

Now, our work...

I have modified the 0.8.1x version of /etc/fail2ban/filter.d/sshd-ddos.conf here locally and used
your supplied logs.
23 entries. 23 hits using this

Fair Warning
Warning: My RE.fu is near useless but I did manage a "result" here.

Maybe you are "that guy"?

Code:

failregex = ^%(__prefix_line)sDid not receive identification string from <HOST> .*?
            ^%(__prefix_line)sConnection closed by <HOST>.*?
            ^%(__prefix_line)sUnable to negotiate with <HOST>.*?

I see theres some delimiter? missing to match the )s, it was it looks like to this dumb bunny. Anyway...

You can create a new /etc/fail2ban/filter.d/custom.conf (for example)
and populate it using this modified copy of /etc/fail2ban/filter.d/sshd-ddos.conf Save it as /etc/fail2ban/filter.d/custom.conf
once in place, test it.

Code:

fail2ban-regex /var/log/auth.log /etc/fail2ban/filter.d/custom.conf

should now provide you with similar results.

Let us know.
I haz a pdf of some nuggets, if you're more than casually interested.

After successful testing, enable the jail.
I suggest

Code:

cp /etc/fail2ban/jail.conf /etc/fail2ban/jail.local

and stick enabled jails only in jail.local.

This file is safe from fail2ban removal or upgrade, as are all custom_named /etc/fail2ban/action.d/ and /etc/fail2ban/filter.d/ configurations.

Subscribed with interest...

r.stiltskin · 04-23-2017, 02:16 AM

Thanks for your help. It got a little more complicated because I'm working in Arch, which has no /var/log/auth.log. Arch uses the systemd-journal.

Yeah, I realized after I posted earlier that I had a separate sshd-ddos.conf file and I hadn't enabled that jail. I have now enabled it. My sshd-ddos.conf contains just one regex -- similar to the one you found:

Code:

failregex = ^%(__prefix_line)sDid not receive identification string from <HOST>\s*$

Notice the slightly different ending.

But it seems that fail2ban-regex ver 0.9.6 is broken for systemd-journal, so I can't test my version of sshd-ddos.conf against my actual journal. According to the manpage my syntax should be

Code:

sudo fail2ban-regex "systemd-journal" /etc/fail2ban/filter.d/sshd-ddos.conf

(or whatever the filename is that contains the filter regexes)...

but when I run it I get

Code:

Running tests
=============

Use   failregex filter file : sshd-ddos, basedir: /etc/fail2ban
Use    journal match : _SYSTEMD_UNIT=sshd.service + _COMM=sshd

Traceback (most recent call last):
  File "/usr/bin/fail2ban-regex", line 34, in <module>
    exec_command_line()
  File "/usr/lib/python3.6/site-packages/fail2ban/client/fail2banregex.py", line 599, in exec_command_line
    if not fail2banRegex.start(opts, args):
  File "/usr/lib/python3.6/site-packages/fail2ban/client/fail2banregex.py", line 538, in start
    self.process(test_lines)
  File "/usr/lib/python3.6/site-packages/fail2ban/client/fail2banregex.py", line 370, in process
    for line_no, line in enumerate(test_lines):
  File "/usr/lib/python3.6/site-packages/fail2ban/client/fail2banregex.py", line 93, in journal_lines_gen
    yield FilterSystemd.formatJournalEntry(entry)
TypeError: formatJournalEntry() missing 1 required positional argument: 'logentry'

so even though I have the right log argument for my system, it doesn't work. This is a bug that I think was fixed upstream but hasn't gotten into the Arch repo yet & I don't feel like patching my own.

So I set up a dummy auth.log containing entries like I posted earlier for testing. The regex I found in my sshd-ddos.conf file didn't catch anything -- I guess <HOST> doesn't include the port specification so the regex can't end with <HOST>/s*$

So I wrote this sshd-ddos.local file (in /etc/fail2ban/filter.d):

Code:

## sshd-ddos.local
## added regexes to catch some more exploits that I found in the systemd-journal


[INCLUDES]

# Read common prefixes. If any customizations available -- read them from
# common.local
before = common.conf

[Definition]

_daemon = sshd

failregex = ^%(__prefix_line)sDid not receive identification string from <HOST>(?: port \d+)\s*$
            ^%(__prefix_line)sConnection closed by <HOST>(?: port \d+) \[preauth\]\s*$
            ^%(__prefix_line)sUnable to negotiate with <HOST>(?: port \d+): no matching key exchange method found. Their offer: diffie-hellman-group1-sha1 \[preauth\]\s*$

ignoreregex = 

[Init]

journalmatch = _SYSTEMD_UNIT=sshd.service + _COMM=sshd

I tested that against the dummy auth.log and these regexes seem to work; time will tell if they also work against my actual journal.

I saved this file as /etc/fail2ban/jail.d/defaults-arch.conf to enable the jails:

Code:

# this file was copied from ArchWiki https://wiki.archlinux.org/index.php/fail2ban
# I changed maxretry to 3 instead of 5 and eliminated sendmail-whois
# and commented out the action line (restoring default action iptables-multiport defined in jail.conf
[sshd]
enabled  = true
filter   = sshd
#action   = iptables[name=SSH, port=ssh, protocol=tcp]
#           sendmail-whois[name=SSH, dest=your@mail.org, sender=fail2ban@mail.com]
backend  = systemd
maxretry = 3


[sshd-ddos]
enabled  = true
filter   = sshd-ddos
# not sure if backend entry is needed here - backend is defined as sshd_backend in jail.conf
# and sshd_backend is defined as default_backend in paths-common.conf
# and default_backend is defined in paths_common.conf as auto
# but I guess it can't hurt:
backend  = systemd
maxretry = 3

Restarted fail2ban and now I'll have to wait and see what happens.

Thanks again.

ondoho · 04-23-2017, 03:06 AM

^ so fail2ban can actually work with systemd's journal? wow, i didn't know that. good stuff.

Habitual · 04-23-2017, 04:44 AM

Well, seems like your run and mine had the same result.

High volume, I sit on

Code:

tail -f /var/log/fail2ban.log

v10 now with IPv6 support.

Habitual · 06-12-2017, 03:58 PM

How did this work out, I wonder?

r.stiltskin · 06-13-2017, 07:22 AM

It's working well. The regexes I wrote in sshd-ddos.local, along with the built-in ones in sshd.conf find almost all of the login attempts, and fail2ban bans them when they reach my threshold of 3 attempts. The only ones that aren't found are the ones that look like this:

Code:

Jun 11 18:35:55 darth sshd[15293]: Received disconnect from 52.178.134.151 port 40378:11: Bye Bye [preauth]
Jun 11 18:35:55 darth sshd[15293]: Disconnected from 52.178.134.151 port 40378 [preauth]

They occur so infrequently that I decided not to bother adding a regex for them.

It's worth mentioning that I also ban (DROP) most of the world altogether so I only log a handful of bogus login attempts daily. I did that by adding this line

Code:

cat /etc/fail2ban/iptables-blacklist | while read IP; do iptables -I f2b-<name> 1 -s $IP -j DROP; done

to the
actionstart section of iptables-multiport.local.

The file /etc/fail2ban/iptables-blacklist is a plain-text list of ip address ranges that cover regions where no valid logins will be coming from. What this does is to add a Chain f2b-sshd-ddos containing all the listed ip addresses to my iptables each time fail2ban service is started.

Habitual · 06-13-2017, 08:39 AM

Quote:

Originally Posted by r.stiltskin

It's working well. The regexes I wrote in sshd-ddos.local, along with the built-in ones in sshd.conf find almost all of the login attempts, and fail2ban bans them when they reach my threshold of 3 attempts. The only ones that aren't found are the ones that look like this:

Code:

Jun 11 18:35:55 darth sshd[15293]: Received disconnect from 52.178.134.151 port 40378:11: Bye Bye [preauth]
Jun 11 18:35:55 darth sshd[15293]: Disconnected from 52.178.134.151 port 40378 [preauth]

They occur so infrequently that I decided not to bother adding a regex for them.

It's worth mentioning that I also ban (DROP) most of the world altogether so I only log a handful of bogus login attempts daily. I did that by adding this line

Code:

cat /etc/fail2ban/iptables-blacklist | while read IP; do iptables -I f2b-<name> 1 -s $IP -j DROP; done

to the
actionstart section of iptables-multiport.local.

The file /etc/fail2ban/iptables-blacklist is a plain-text list of ip address ranges that cover regions where no valid logins will be coming from. What this does is to add a Chain f2b-sshd-ddos containing all the listed ip addresses to my iptables each time fail2ban service is started.

Thanks for letting us know. I used to utilize that method of persistent "bad.guys" file for restarts.
Not much traffic, so we recently rolled back a few rules. Levelled off at 6k from 9, Irritants are about the same pace.
PreAuth = Doorbell Ditch