Server very high load when wordpress was attacked

Habitual · 10-15-2015, 12:17 PM

fastdns:
Have a read of the first 2 links at http://www.digitalfaq.com/forum/guid...l2ban-csf.html
before making a decision to remove CSF.

Quote:

Originally Posted by astrogeek

It is not sufficient to just install any of these types of "solutions" and expect them to take care of all the problems - you have to understand the environment and configure the tools to address your specific problems.

is a very wise statement.

chobong · 10-23-2015, 04:43 AM

Hi Habitual

When I run this command on my live server

Quote:

fail2ban-regex /path/to/my/access/log /etc/fail2ban/filter.d/myfilter.conf

It showed many line as below and CPU increaed though it is working well on my test server.

Quote:

Matched time template Day/MONTH/Year:Hour:Minute:Second
Matched time template Day/MONTH/Year:Hour:Minute:Second
Matched time template Day/MONTH/Year:Hour:Minute:Second
Matched time template Day/MONTH/Year:Hour:Minute:Second
Matched time template Day/MONTH/Year:Hour:Minute:Second
Matched time template Day/MONTH/Year:Hour:Minute:Second
Matched time template Day/MONTH/Year:Hour:Minute:Second
Matched time template Day/MONTH/Year:Hour:Minute:Second
Matched time template Day/MONTH/Year:Hour:Minute:Second
Matched time template Day/MONTH/Year:Hour:Minute:Second
Matched time template Day/MONTH/Year:Hour:Minute:Second
Matched time template Day/MONTH/Year:Hour:Minute:Second
Matched time template Day/MONTH/Year:Hour:Minute:Second
Matched time template Day/MONTH/Year:Hour:Minute:Second
Matched time template Day/MONTH/Year:Hour:Minute:Second
Matched time template Day/MONTH/Year:Hour:Minute:Second
Matched time template Day/MONTH/Year:Hour:Minute:Second
Matched time template Day/MONTH/Year:Hour:Minute:Second
Matched time template Day/MONTH/Year:Hour:Minute:Second
Matched time template Day/MONTH/Year:Hour:Minute:Second
Matched time template Day/MONTH/Year:Hour:Minute:Second
Matched time template Day/MONTH/Year:Hour:Minute:Second
Matched time template Day/MONTH/Year:Hour:Minute:Second
Matched time template Day/MONTH/Year:Hour:Minute:Second
Matched time template Day/MONTH/Year:Hour:Minute:Second
Matched time template Day/MONTH/Year:Hour:Minute:Second
Matched time template Day/MONTH/Year:Hour:Minute:Second
....

Could you please help to fix this?

Thanks in advance.

chobong · 10-23-2015, 05:09 AM

And here is the output on my test server. It doesn't like as your result "Success, the total number of match is 35" but it's working.

Quote:

Running tests
=============

Use failregex filter file : myfilter, basedir: /etc/fail2ban
Use log file : /mylogfile
Use encoding : UTF-8

Results
=======

Failregex: 5 total
|- #) [# of hits] regular expression
| 4) [5] ^<HOST> .* "GET .*abdullkarem.*" .*$
`-

Ignoreregex: 0 total

Date template hits:
|- [# of hits] date format
| [10] Day(?P<_sep>[-/])MON(?P=_sep)Year[ :]?24hour:Minute:Second(?:\.Microseconds)?(?: Zone offset)?
`-

Lines: 10 lines, 0 ignored, 5 matched, 5 missed [processed in 0.00 sec]
|- Missed line(s):
| xxx.xxx.xxx.xxx - - [21/Oct/2015:11:13:04 +0700] "GET / HTTP/1.1" 200 4 "-" "Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/46.0.2490.71 Safari/537.36"
| xxx.xxx.xxx.xxx - - [21/Oct/2015:11:13:04 +0700] "GET /favicon.ico HTTP/1.1" 404 209 "http://isd.mbizlab.com/" "Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/46.0.2490.71 Safari/537.36"
| xxx.xxx.xxx.xxx - - [21/Oct/2015:11:29:46 +0700] "GET / HTTP/1.1" 200 4 "-" "Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/46.0.2490.71 Safari/537.36"
| xxx.xxx.xxx.xxx - - [21/Oct/2015:11:29:46 +0700] "GET /favicon.ico HTTP/1.1" 404 209 "http://isd.mbizlab.com/" "Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/46.0.2490.71 Safari/537.36"
| xxx.xxx.xxx.xxx - - [21/Oct/2015:11:31:15 +0700] "GET / HTTP/1.1" 304 - "-" "Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/46.0.2490.71 Safari/537.36"

Habitual · 10-23-2015, 08:10 AM

Quote:

Originally Posted by chobong

And here is the output on my test server. It doesn't like as your result "Success, the total number of match is 35" but it's working.

chobong:
I'm confused by your last 2 posts.
On the test server, it works manually, but fails on the live server and drives up the CPU?

I'm not certain that there is not something else messing with your live server.
You'll need to examine the differences in the 2 environments to determine what the differences are.
There are far too many reasons that running fail2ban-regex on your live server seems to drive up the CPU.

No 2 systems are identical. Even if deployed in an identical manner.

Was fail2ban installed when you first reported this attack here on LQ?

It seems as if your CPU on the live server is taxed by something other that some butt-nugget pounding your web server with "abdullkarem" string requests.

Are those requests 404s?

You need to post a complete sample of say 5 to 10 lines where abdullkarem is present in the logs.
Don't need to obfuscate the suspect IP either. (if they are bad guys, they are bad guys).
Here are 5 records from my apache2 access.log.1 (a logrotated copy of access.log):

Code:

www5.mld.nu - - [16/Oct/2015:03:40:32 -0700] "GET /wp-admin/wp-editors.php?450699=1&php4=1&root=1&upl=1&wphp4=1&abdullkarem=1&wp=1&module=1&php=1&php5=1&wphp5=1 HTTP/1.0" 403 412 "-" "-"
5.178.78.182 - - [16/Oct/2015:03:40:32 -0700] "GET /wp-content/plugins/hi.php?450699=1&php4=1&root=1&upl=1&wphp4=1&abdullkarem=1&wp=1&module=1&php=1&php5=1&wphp5=1 HTTP/1.0" 301 652 "-" "-"
5.178.78.182 - - [16/Oct/2015:03:40:33 -0700] "GET /index.php?450699=1&php4=1&root=1&upl=1&wphp4=1&abdullkarem=1&wp=1&module=1&php=1&php5=1&wphp5=1 HTTP/1.0" 301 653 "-" "-"
5.178.78.182 - - [16/Oct/2015:03:40:35 -0700] "GET /wp-content/uploads/_input__test.php5?php4&root&upl&wphp4&abdullkarem& HTTP/1.0" 301 609 "-" "-"
5.178.78.182 - - [16/Oct/2015:03:40:40 -0700] "GET /index.php?php4&root&upl&wphp4&abdullkarem& HTTP/1.0" 301 599 "-" "-"

It takes a lot of work to stay on top of issues like this.
There is no "one size fits all" remedy for these situations.
It takes patience, practice and persistence to stay on top of these aggressors.

Security is a Verb, not a noun.
I wish you luck.

hiepbg · 10-23-2015, 01:56 PM

Quote:

Originally Posted by Habitual

Add this to myfilter.conf

Code:

^<HOST> .* "GET .*abdullkarem.*" .*$

and the result here is:

Code:

Success, the total number of match is 35

Sorry about that.

Original post updated.

Reference:
https://hoopercharles.wordpress.com/...linux-selinux/

Hi,
I have a problem. Your failregex don't work on nginx. When i restart fail2ban, it show an error:

Code:

ERROR  Found no accessible config files for 'filter.d/abdullkarem.conf' under /etc/fail2ban
ERROR  Unable to read the filter
ERROR  Errors in jail 'abdullkarem'. Skipping...

Habitual · 10-23-2015, 02:06 PM

hiepbg:

Yeah, sorry. I'm not fail2ban support.

You'll have to open a new thread for your issue.

Thank you.

hiepbg · 10-23-2015, 02:13 PM

Quote:

Originally Posted by Habitual

hiepbg:

Yeah, sorry. I'm not fail2ban support.

You'll have to open a new thread for your issue.

Thank you.

Thanks,

Habitual · 10-23-2015, 02:19 PM

No worries.

Please see "First things first" link in my signature for what you may wish to include on a new post.

Thank you.

hiepbg · 10-23-2015, 02:40 PM

Thanks,
It's my typo error. Everything work fine

Habitual · 10-23-2015, 02:43 PM

Quote:

Originally Posted by hiepbg

Thanks,
It's my typo error. Everything work fine

Well, that is Good News!

Have a great Weekend.
And welcome to LQ!

chobong · 10-26-2015, 01:54 AM

Thank's lot Habitual for your kind support.

The fail2ban-regex on my live server now is working. I followed your guide to setup fail2ban.

The CPU was high when running the fail2ban-regex is due to the access log is too big.

I just wonder one thing about your configuration

Quote:

docroot = /var/www/html

Do we need this option? I am going to use myfilter.conf for many sites on my server. But the docroot of each site is not same.

Thank you

Habitual · 10-26-2015, 09:41 AM

Quote:

Originally Posted by chobong

But the docroot of each site is not same.

Thank you

Try using

Code:

docroot = /var/www

They are all "under that" no?

chobong · 10-26-2015, 10:33 PM

Hi Habitual

Yes, I will set the docroot to parent folder of my websites.

But I am still wondering we set the logpath in jail.local already, why we need this option?

Thank you so much.

Habitual · 10-27-2015, 03:00 AM

chobong:
That is a very good question!
Unfortunately, one I don't have a ready answer for.

But I shall attempt to find out...

Peace.

Habitual · 11-24-2015, 09:19 AM

chobong:

Update:
I'm not sure docroot is necessary, as /etc/fail2ban/filter.d/apache-badbots.conf
doesn't have this directive and that .conf works.

Peace.