LinuxQuestions.org
Latest LQ Deal: Latest LQ Deals
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - Security
User Name
Password
Linux - Security This forum is for all security related questions.
Questions, tips, system compromises, firewalls, etc. are all included here.

Notices


Reply
  Search this Thread
Old 10-01-2015, 10:30 AM   #1
landysaccount
Member
 
Registered: Sep 2008
Location: Dominican Republic
Distribution: Debian
Posts: 188

Rep: Reputation: 18
Unhappy Apache2 Server attacked all day long and can't block it


Hello.

Lately, our server have been experiencing some sort of attack from google's ip addresses. I suspect this might be a SYN or DDos attack and can't control it is getting out of hand with thousands and thousands requests daily, freezing up the server with CPU usage over 100.

We have a squid "accelerator" on our web server and looking at the log I see a LOT of these:

Quote:
1443712417.300 157 66.249.75.191 TCP_MISS/200 8518 GET <OURDOMAIN> - FIRST_UP_PARENT/myAccel text/html
The attack "originates" from 66.249.75.0 subnet, belonging to google company. Obviously spoofing the ip addresses.

I've tried some "solutions" but, have not work properly:
- Have decreased Apache's Timeout to 10
- MaxKeepAliveRequests 200
- KeepAliveTimeout 6

I've even tried blocking SYN flood to port 80 but, it drops connections from real users:

$iptables -N no-syn-flood
$iptables -A no-syn-flood -m limit --limit 1/s --limit-burst 5 -j RETURN
$iptables -A no-syn-flood -j DROP
$iptables -I INPUT -p tcp -i $EXT_IFACE --dport 80 --syn -m state --state NEW -j no-syn-flood

Temporarily, I've dropped all traffic from that subnet and things are back to normal:
$iptables -A INPUT -s 66.249.75.0/255.255.255.0 -j DROP

What other measure you recommend?

Thanks in advanced for your time and help.

Last edited by landysaccount; 10-01-2015 at 11:08 AM.
 
Old 10-01-2015, 01:07 PM   #2
Sefyir
Member
 
Registered: Mar 2015
Distribution: Linux Mint
Posts: 634

Rep: Reputation: 316Reputation: 316Reputation: 316Reputation: 316
Code:
$iptables -A no-syn-flood -m limit --limit 1/s --limit-burst 5 -j RETURN
That's really low. A single website request will sometimes take 7-9 requests
Have you tried debugging it with something like the below and seeing how easy it is to trigger?
Code:
$ipt -A no-syn-flood -p tcp -j LOG --log-prefix "OVER_USAGE "
I have something like this

Code:
$ipt -A WEBSERVER -p tcp -m limit --limit 25/minute --limit-burst 120 -j ACCEPT
That adds a token slightly less then about 1 every 2 seconds and hasn't affected existing users but when I try to spam it, slows it down a whole lot after the initial 120 burst.

What kind of content are you serving? static? php? I might suggest going to nginx instead of apache. After I switched from apache to nginx + php5-fpm I could handle a lot more traffic for less resources.
 
Old 10-01-2015, 02:17 PM   #3
Habitual
LQ Veteran
 
Registered: Jan 2011
Location: Abingdon, VA
Distribution: Catalina
Posts: 9,374
Blog Entries: 37

Rep: Reputation: Disabled
Quote:
Originally Posted by landysaccount View Post
Obviously spoofing the ip addresses.
How is this 'obvious'?
Code:
host 66.249.75.191
191.75.249.66.in-addr.arpa domain name pointer crawl-66-249-75-191.googlebot.com
You could block 66.249.64.0/19
 
Old 10-01-2015, 03:27 PM   #4
landysaccount
Member
 
Registered: Sep 2008
Location: Dominican Republic
Distribution: Debian
Posts: 188

Original Poster
Rep: Reputation: 18
Thanks for replying.

I have a server hosting 8 websites under Joomla but, only one of these get a lot of traffic.

After blocking:

Quote:
$iptables -A INPUT -s 66.249.75.0/255.255.255.0 -j DROP
The server is behaving a lot better then it was:

Quote:
16:22:56 up 10 days, 6:53, 2 users, load average: 0.73, 0.68, 0.67
Before doing that it was over 50.

I have nginx installed and was thinking of giving it a chance to test it, I guess that would be a good option.
 
Old 10-01-2015, 03:33 PM   #5
Rinndalir
Member
 
Registered: Sep 2015
Posts: 733

Rep: Reputation: Disabled
Google have been, for some time now, able to execute javascript as a client so probably they are interested in your site and they are doing more than just crawling static content. Also I forget how many blocks of IPs google have but it is a lot. So don't be surprised if googlebot comes around again soon.
 
Old 10-01-2015, 04:14 PM   #6
Sefyir
Member
 
Registered: Mar 2015
Distribution: Linux Mint
Posts: 634

Rep: Reputation: 316Reputation: 316Reputation: 316Reputation: 316
This may also be useful to manage crawl rates.

https://support.google.com/webmasters/answer/48620

After reflecting, this is what crawlers do. Otherwise your content will not show up on google.

I grabbed my most recent ones to display
http://pastebin.com/xx63dq9x

This barely touches the performance of my server but it does make quite a few requests

Also, even with thousands and thousands of requests a day, that still only amounts to maybe like 166/hr (at 4k/day) which definitely should not be enough to cause your system to lock up. You should still be able to use your site even with 10-ish concurrent users making connections (albeit slower). Use something like ab to stress test your server?

Just out of curiosity, what is your ram usage like? How much do you have and how much is being used with "higher levels of traffic"?
It's possible you're running out of ram and going into excessive swapping
Just speculation at this point though, too little info to go on.

Last edited by Sefyir; 10-01-2015 at 04:20 PM.
 
Old 10-01-2015, 04:31 PM   #7
Habitual
LQ Veteran
 
Registered: Jan 2011
Location: Abingdon, VA
Distribution: Catalina
Posts: 9,374
Blog Entries: 37

Rep: Reputation: Disabled
Quote:
Originally Posted by Rinndalir View Post
I forget how many blocks of IPs google have but it is a lot.
Code:
nslookup -q=TXT _netblocks.google.com 8.8.8.8
speweth
Code:
ip4:64.18.0.0/20 ip4:64.233.160.0/19 ip4:66.102.0.0/20 ip4:66.249.80.0/20 ip4:72.14.192.0/18 ip4:74.125.0.0/16 ip4:108.177.8.0/21 ip4:173.194.0.0/16 ip4:207.126.144.0/20 ip4:209.85.128.0/17 ip4:216.58.192.0/19 ip4:216.239.32.0/19
Enjoy the goodness.
 
1 members found this post helpful.
Old 10-01-2015, 05:16 PM   #8
landysaccount
Member
 
Registered: Sep 2008
Location: Dominican Republic
Distribution: Debian
Posts: 188

Original Poster
Rep: Reputation: 18
We have 8G of RAM on that server and no theres no swapping going on, the most I've seen the memory usage rate is at 2.5G, just high cpu usage by apache and mysql since, there is a lot of request for the websites index page.

So, blocking those addresses will do more harm because our websites won't be on google?

Last edited by landysaccount; 10-01-2015 at 05:20 PM.
 
Old 10-01-2015, 05:26 PM   #9
Habitual
LQ Veteran
 
Registered: Jan 2011
Location: Abingdon, VA
Distribution: Catalina
Posts: 9,374
Blog Entries: 37

Rep: Reputation: Disabled
Quote:
Originally Posted by landysaccount View Post
So, blocking those addresses will do more harm because our websites won't be on google?
Did any one say "block these also"? No. It is merely information about google's netblocks.

What you do with it is up to you.
 
Old 10-01-2015, 05:47 PM   #10
landysaccount
Member
 
Registered: Sep 2008
Location: Dominican Republic
Distribution: Debian
Posts: 188

Original Poster
Rep: Reputation: 18
Quote:
Originally Posted by Habitual View Post
Did any one say "block these also"? No. It is merely information about google's netblocks.

What you do with it is up to you.
Didn't mean those last ip addresses, I refer to the ones I blocked earlier (see my previous posts)
 
Old 10-01-2015, 06:09 PM   #11
Sefyir
Member
 
Registered: Mar 2015
Distribution: Linux Mint
Posts: 634

Rep: Reputation: 316Reputation: 316Reputation: 316Reputation: 316
If you block google spiders, how does google know what is on the site?
This is your server and you're asking how to block them and suggestions have been made... but my thinking is there is a config issue causing high cpu usage after a certain threshold of traffic that should be warranting your attention and investigation. Since you have high levels of RAM, my next guess would be related to the processes handling the requests (mpm-worker and such)
Or maybe just search it -> https://duckduckgo.com/?q=apache+hig...age&t=lm&ia=qa
 
Old 10-01-2015, 06:53 PM   #12
syg00
LQ Veteran
 
Registered: Aug 2003
Location: Australia
Distribution: Lots ...
Posts: 21,103

Rep: Reputation: 4117Reputation: 4117Reputation: 4117Reputation: 4117Reputation: 4117Reputation: 4117Reputation: 4117Reputation: 4117Reputation: 4117Reputation: 4117Reputation: 4117
Quote:
Originally Posted by Sefyir View Post
... but my thinking is there is a config issue causing high cpu usage after a certain threshold of traffic that should be warranting your attention and investigation.
Absolutely.
In post #1, the OP mentions CPU usage, in #4 we are shown loadavg - so which is it that the OP is worried about ?. These are not the same - especially in Linux, and more especially where Apache is involved.
 
Old 10-01-2015, 07:18 PM   #13
Rinndalir
Member
 
Registered: Sep 2015
Posts: 733

Rep: Reputation: Disabled
8GB of RAM is not a lot for a server running apache/mysql and serving images. In your case more RAM would be money well spent.

What OS are you running and is it up to date?

Have you grabbed performance data from the mysql db? See what's hot.

Is it possible you have something misconfigured? How well do you know apache/mysql?

Are you using SSD or HDD? What filesystem? What mount options?

What forum software are you running? Is it up to date?

Can you check you apache logs to see what the googlebot was doing?

It's good to block the ips now so you can keep the system going but like others have said you need to find the cause.
 
Old 10-02-2015, 05:09 AM   #14
Habitual
LQ Veteran
 
Registered: Jan 2011
Location: Abingdon, VA
Distribution: Catalina
Posts: 9,374
Blog Entries: 37

Rep: Reputation: Disabled
Quote:
Originally Posted by landysaccount View Post
Didn't mean those last ip addresses, I refer to the ones I blocked earlier (see my previous posts)
Why block an "obviously spoofed" ip address at all?

Google
Code:
Crawl-delay
You're coming up short with any solid evidence of an "attack"

End Trasmission.

Last edited by Habitual; 10-02-2015 at 05:10 AM.
 
Old 10-02-2015, 10:50 AM   #15
landysaccount
Member
 
Registered: Sep 2008
Location: Dominican Republic
Distribution: Debian
Posts: 188

Original Poster
Rep: Reputation: 18
Hello.

I appreciate the time all of you have taken to assist me with my server's problem.

Answering questions:

The server is running Debian 7.8 (Wheezy) and everything is up to date as of yesterday. This is a dedicated server we contracted two years ago for hosting our websites only. As mentioned before, we have several websites running Joomla but, one of these have a lot of traffic with over 20000 articles.

We do not have any panel or forum software installed.

I believe there might be some misconfiguration with apache and mysql since I've seen these processes at 99 cpu usage.

The server also have squid installed as an accelerator.

Here's apache2.conf doing virtal hosting:
Quote:

Timeout 10
KeepAlive On

MaxKeepAliveRequests 200
KeepAliveTimeout 4

<IfModule mpm_prefork_module>
StartServers 6
MinSpareServers 6
MaxSpareServers 12
MaxClients 250
MaxRequestsPerChild 300
</IfModule>

Here's part of our mysql's my.cfg:
Quote:
interactive_timeout = 10
wait_timeout = 4
lock_wait_timeout = 30

bind-address = 127.0.0.1

key_buffer = 1024M
max_allowed_packet = 64M
thread_stack = 192K
thread_cache_size = 16000

myisam-recover = BACKUP
max_connections = 300 # was disabled
table_open_cache = 1604K # was 1604
table_cache = 2124K
read_buffer_size = 4M
sort_buffer_size = 4M

innodb_buffer_pool_size = 26M
myisam_sort_buffer_size = 128M # was 64M

thread_concurrency = 8
join_buffer_size = 4M

low_priority_updates = 1
Quote:
cat /etc/fstab
# /etc/fstab: static file system information.
#
# <file system> <mount point> <type> <options> <dump> <pass>
proc /proc proc defaults 0 0
#/dev/sda3 / ext3 defaults,usrquota 0 1
UUID=404ab653-2465-425c-adf8-cffe2220ea6e / ext3 defaults,usrquota 0 1
#/dev/sda1 /boot ext2 defaults 0 2
UUID=b4e64ebf-cd03-4a51-b385-506cbefd8e7f /boot ext2 defaults 0 2
# /dev/sda2 none swap sw 0 0
UUID=aa0bb896-2f41-4db5-89a5-78a822fe9e6d none swap sw 0 0
# /dev/sdb1 /mnt/disk1 ext4 defaults 0 0
UUID=d2e5374b-2f2a-486a-9128-ad3ef2b9b207 /mnt/disk1 ext4 defaults 0 0
Once again, thank you for taking your time to help out.

Last edited by landysaccount; 10-02-2015 at 10:51 AM.
 
1 members found this post helpful.
  


Reply

Tags
apache, iptables, security, squid3


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off



Similar Threads
Thread Thread Starter Forum Replies Last Post
I think my server is being attacked. jim.thornton Linux - Security 10 07-15-2014 09:55 AM
Server Attacked code PatriceJ Linux - Security 18 02-27-2014 04:54 PM
new server got attacked before launching agriz Linux - Security 8 07-23-2012 08:52 AM
Help me. My server is attacked DDoS ndduy Linux - Security 12 11-29-2009 02:47 PM
qmail server getting attacked lsimon4180 Linux - Software 41 10-15-2004 03:44 PM

LinuxQuestions.org > Forums > Linux Forums > Linux - Security

All times are GMT -5. The time now is 09:19 AM.

Main Menu
Advertisement
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
Open Source Consulting | Domain Registration