LinuxQuestions.org

LinuxQuestions.org (/questions/)
-   Linux - Security (https://www.linuxquestions.org/questions/linux-security-4/)
-   -   DDOS, please help! (https://www.linuxquestions.org/questions/linux-security-4/ddos-please-help-861624/)

Seregwethrin 02-09-2011 09:20 AM

DDOS, please help!
 
I'm really tired of this for one week i'm dealing with this issue

I've a vps with 512mb ram which is enough to handle my website users

I moved my server to a provider with CISCO firewall but that didn't help

I moved from apache to nginx with nginx limit-http-requests module but that didn't help

Also i was tried apache with mod_evasive, result was failure.

I set iptables to accept only established and related connections, that didn't do any good.

Attacks are to port 80. Problem is not bandwidth, problem is there's a lot of connections so the ram become full and vps could't handle any connections, event i can't make ssh connection.

Example netstat -avpn result is here after as soon as making web server (nginx) online:
https://docs.google.com/document/d/1...thkey=COCup6sM
And this is nothing, when i make the web server online for 10 minutes there are dozens of those ips

What can i do? Can I do anything at all? Yes I've already told the isp about this and they'll look into but my old isp could't do anything except null routing my vps's ip.

I'm really tired of it, too angry, too weak to deal with more... Do you have any suggestions?

Note: if publishing those ips is not allowed by forum rules please pm me, but i'm gonna remove them after i get rid of this problem anyway.

iuselinux 02-09-2011 09:27 AM

You sure its a DDoS and not a DoS?

Try switching to litespeed web server, they are supposedly good with (D)DoS

Seregwethrin 02-09-2011 09:33 AM

It may be DoS but there's definitely more then 1 ip. Maybe not thousands of ips but more than 1 and they are from different locations, from even different cities.

xeleema 02-09-2011 09:35 AM

Whoa, wait a sec.
According to that file, only the following IPs are throwing the bad traffic;
(Location Data pulled from http://www.geoiptool.com)
Code:

78.162.28.14  (Unknown Hostname)              Antalya, Turkey (Long: 30.6897, Lat: 36.9125)
78.166.122.127 (Unknown Hostname)              Mersin, Turkey (Long: 34.6442, Lat: 36.7328)
78.188.30.155  (dsl78.188-7835.ttnet.net.tr)  Istanbul, Turkey (Long: 28.9647, Lat: 41.0186)
78.188.50.142  (dsl78.188-12942.ttnet.net.tr) Istanbul, Turkey (Long: 28.9647, Lat: 41.0186)

Now you can either use iptables to drop all incoming traffic from these guys, or you can use something like blockhosts.
"Moving" your server does nothing if they're following your Domain Name. Depending on your ISP for security is like depending on a Car Salesman to change your oil.

Seregwethrin 02-09-2011 09:40 AM

But they have dynamic ip addresses. They are using classic adsl. Probably they are some jerk teenagers who keep changing their ips.

Also I moved my server because I need a more secure datacenter and the old isp did't make any help for dealing with the attack. So I didn't move the server to block attack, I moved because of for better protection. But as situation states, I didn't get.

Seregwethrin 02-09-2011 10:01 AM

Check it out, new ips
https://docs.google.com/document/d/1...thkey=CNPHyP4L

I get those netstat result as soon as opening nginx. Maybe one second passed.

I have to close nginx after because if I not those connections increases.

xeleema 02-09-2011 10:03 AM

1 Attachment(s)
There's a few options for dealing with those kinds of people.

1) Blockhosts to cut them off (that's what I use).
I setup each of my webservers to ban an IP for one month when it receives two errors from the webserver.
2) Start blocking whole IP Ranges that they're coming from.
(I've attached every IP range assigned to Turkey)
3) Limit every IP address to 10 connections to port 80 (via iptables)
(I'm looking up how to do this now...saw it in an LQ thread once)

NOTE: They may be doing more to you than just (D)DoS'ing. Check the logs for your other services (especially SSH!).

xeleema 02-09-2011 10:05 AM

Checking out the file now, however a few questions about how the site is built.

1) Is this a MySQL+PHP site? (aka: Does it have a lot of Dynamic content)
2) How much (if any) of the site is static pages?
3) Are you using this site to generate income, or is this a pet-project (how critical is this?)

xeleema 02-09-2011 10:11 AM

You must have seriously pissed-off the Turks :)
Just kidding, but all those IPs are Turkish except for one.
Code:

188.56.230.69 - Unknown. Probably a spoof.
Anything in your access logs?

jcalzare 02-09-2011 10:12 AM

You could try blocking the addresses allocated to Turkey. Here's a list of CIDR addresses from http://www.countryipblocks.net/country-blocks/
If you block them all for a while, the attackers will probably move on.

Edit: Ack nevermind. xeleema already attached a list while I was typing. My post is now less obnoxiously long :)

Seregwethrin 02-09-2011 10:16 AM

They can't be doing more to me because I've two ips and I use one to connect to ftp ssh etc, and the other has only port 80 open. They couldn't have known the other ip. I set restrictions via iptables.

Well I can't block ips from Turkey because my website is for Turkish gamers.

And yes it's a php mysql website, actually it is vbulletin forum just like here. To be more specific it's a website (moreover it's a forum) for world of warcraft turkish gamers but it's the biggest in turkish language. Of course the forum has its competitors and probably they are responsible.

There is like no static pages. Every page is php.

And no I'm not getting any a income. Yes I get something in return at avg 150-200$ dollars in a month but I'm giving more then I get as time and work basis. I just don't want to close it because it is useful to people, I don't use it actually, I even don't play games. I opened it in while I was still in high school 6 yeas ago so it has a nostalgic history for me too.

jcalzare 02-09-2011 10:28 AM

Well, in that case it sounds like it's not worth investing serious monetary resources into keeping this thing online through the attack. I would suggest replacing your index.php with a simple html maintenance page. This will keep the website online, and the attackers will eventually just give up, as they shouldn't be able to overwhelm the server resources when you have a simple page up.

xeleema 02-09-2011 10:29 AM

Sounds like an awesome site, let's see if we can give these PFY's a run for their efforts. However...
Do NOT assume they don't know the other IP. Be sure to check your logs for any SSH login attempts. They could be trying to distract you with a (D)DoS while they attempt to take over the server.

Code:

#Detect DoS attack by limiting to 12 connections/sec
iptables -N syn-flood
iptables -A syn-flood -m limit --limit=12/s --limit-burst 24 -j RETURN
iptables -A syn-flood -j DROP

WARNING: Be careful with iptables! You can lock yourself out of your box if you're not careful.

EDIT: This LQ thread has a good post :)

Seregwethrin 02-09-2011 10:50 AM

@xeleema

Thanks I applied iptable rules.

Well we can see the ips, why there's no software to check them for example every minute and block them? I still didn't dig into BlockHosts but I'll definitely. I hope it does what I want and works with nginx.

And I checked netstat with your iptables rules. I still can see hundreds of attacking ips but they look like they don't harm the server at least for now.

xeleema 02-09-2011 12:24 PM

So your site is up? Cool! (What's the address? I kinda wanna check out the site :) )
If there were any other helpful posts, be sure to click "Yes" in the bottom right-hand corner of those posts.
Also, if your problem has been solved, use "Thread Tools" at the very top of the page to mark this as [SOLVED].

EDIT: Blockhosts is used to parse log files and throw a ban on any IP that shows up in the logs.
I've not worked with nginx before, so I don't know if the connections from the "bad guys" are showing up a certain way. If they're doing something besides opening a bunch of connections, the logs should show it, so you can ban 'em.

Seregwethrin 02-09-2011 04:00 PM

Hi,

I just wake up :)

you can check it here

I now just checked and it bans normal users too. For example disconnects me from ssh and I can't get a proper connection from nginx, my internet browser says connection was reset.

I just want to ban ips if an ip has 10 or more connections at given time point. Couldn't we just do that?

xeleema 02-09-2011 05:20 PM

Change this line;

Code:

iptables -A syn-flood -m limit --limit=12/s --limit-burst 24 -j RETURN
To this:
Code:

iptables -A syn-flood -m limit --limit=10/s -j RETURN
If that doesn't work, up the "limit" as needed.

Seregwethrin 02-09-2011 05:40 PM

Hi,

I'll try it but before I just tried this except rootkit
http://www.topwebhosts.org/tools/apf...os-rootkit.php

If that doesn't work, I'll try it.

unSpawn 02-09-2011 05:57 PM

Quote:

Originally Posted by xeleema (Post 4253179)
There's a few options for dealing with those kinds of people.

The best first thing to tell an OP in case of (D)DoS is that in essence the endpoint can never solve a (D)DoS completely: succesful mitigation requires hosting provider or upstream cooperation (search LQ for threads on the topic?).


Quote:

Originally Posted by xeleema (Post 4253179)
Blockhosts to cut them off

Please see this (in short: blocking through the firewall is more efficient and safer than blocking requiring service connections).


Quote:

Originally Posted by xeleema (Post 4253206)
Code:

iptables -N syn-flood
iptables -A syn-flood -m limit --limit=12/s --limit-burst 24 -j RETURN
iptables -A syn-flood -j DROP


This doesn't "detect" attacks but limits any connection (no "-m state --state NEW") to any port (no "-m tcp -p tcp -dport 80"). Why not use "-m recent" to achieve "x connections per interval"?

Seregwethrin 02-09-2011 08:19 PM

I THINK I DID IT!

Woha I'm really happy. Apf, Bfd, DDoS Deflate runs perfect!

I tried a little to make apf work on vps but it is really effective with DDoS Deflate. DDoS Deflate wasn't effective enough with iptables (or maybe I could't do it right).

Hope it works like this without any problems.

But I'd like to ask this: Does Apf firewall has it's own network monitor or does it use iptables to work?

xeleema 02-10-2011 12:33 AM

Quote:

Originally Posted by unSpawn (Post 4253548)
Please see this (in short: blocking through the firewall is more efficient and safer than blocking requiring service connections).

@unSpawn
Perhaps I should have been more clear, I use blockhosts to stop attacks via iptables, not /etc/hosts.allow and hosts.deny.

salasi 02-10-2011 09:10 AM

Quote:

Originally Posted by Seregwethrin (Post 4253634)
Woha I'm really happy...Hope it works like this without any problems.

That's important; will it stay this way?

Quote:

Originally Posted by Seregwethrin (Post 4253634)

But I'd like to ask this: Does Apf firewall has it's own network monitor or does it use iptables to work?

APF, like all of these things that I can think of, is a script that Does Stuff (reg TM) with iptables. Well, actually, it is a small collection of scripts and you can look inside the .tar.gz package to see what they are, if you are interested. Or, indeed, you can list your iptables rules to see what the outcome has been, in your particular case.

I have one further comment; it looked as if you had lots of connections that were in the process of being opened, and that would seem to be a symptom of a slow_loris type of attack, rather than a plain DoS/DDoS. Is that the way that you see things? If so, a look at the wikipedia page on slow_loris might be a worthwhile investment.

Seregwethrin 02-10-2011 09:22 AM

Yes they look like slow_loris attack. The computers which are sources of attacker connections don't have large bandwidths. They are just adsl connections, at best they should have 8mbit downstream 1mbit upstream connection. So it is really likely to the attack is a type of slow_loris attack. And their achieved result is filling resources of my vps. I did't even connect to ssh because the memory got full.

The attack still continues, but the ips are banned continuously. Hope they back down at the end!

iuselinux 02-11-2011 07:17 AM

Heres the deal man...

DDoS is an arms race, its how much bandwidth you have and how much they do, whos pipe is bigger... Other than upgrading, you cant do too much...

Try this..
[LINK REMOVED BY MODERATOR]

Or what you can do (this is what ive done), is just duplicate your website onto another server, and add another A record into whatever DNS provider you use to start some DNS level load balancing. Two things to take into consideration about this..

1) Do you use MySQL? Im sure you do, if so, you need to put it on another server somewhere. But now you need to think about DoSing yourself, you might flood your MySQL server with connections...
2) DO you use php based sessions or cookies? If its cookies you should be good, as it will be kept int he browser if the TTL expires and they hit the other server, if not, you may want to mount a LVM and configure PHP to put sessions on that, and share it between both servers... Although, this may not be a problem if the TTL is long enough, because it will cache the DNS on the viewers computer.

You said it wasnt a DoS attack, which means its not slowloris, but the dude above me seems to think it is... But... its hard to tell if it is, seeing as how the slowloris attack kills its connection before the header request is complete, meaning it wont log it to the access_logs... But if it is, you can install httpready, or you can add some iptable rules, or you can install CSF/LFD.

2 slowloris solutions at the bottom of this page: http://famousphil.com/blog/2010/02/s...p-dos-attacks/

Seregwethrin 02-11-2011 09:47 AM

Thanks all, I was able to lighten the attack. Of course it is not possible to totally get rid of it.

GrapefruiTgirl 02-13-2011 09:43 AM

Moved: This thread is more suitable in Linux Security, and has been moved accordingly to help your thread/question get the exposure it deserves.

Kind regards!

salasi 02-13-2011 10:14 AM

Quote:

Originally Posted by salasi (Post 4254179)
...it looked as if you had lots of connections that were in the process of being opened, and that would seem to be a symptom of a slow_loris type of attack, rather than a plain DoS/DDoS. Is that the way that you see things? If so, a look at the wikipedia page on slow_loris might be a worthwhile investment.

While searching for something else, this interesting discussion came to my attention.

xeleema 03-11-2011 01:55 PM

Not to drudge up an old topic, but I've recently seem some strange behaviour on a friends webserver, so I threw this up into his existing /etc/sysconfig/iptables (RHEL).
Note that the thresholds (bolded text) are tuned to their relativly small webservers' logs. Should their readership grow, they'll have to have this expanded.
Code:

##############################################################################
# HTTP PROTECTIONS - port 80
##############################################################################
# Log and drop clients that hit us with 20 connections per sec, or burst 30.
-A INPUT -p tcp -m tcp --dport 80 -m state --state NEW -m limit --limit=20/s --limit-burst 30 -j LOG --log-prefix "30 conns/20s: "
-A INPUT -p tcp -m tcp --dport 80 -m state --state NEW -m limit --limit=20/s --limit-burst 30 -j DROP
# Log and drop any one IP that has 50 conns open
-A INPUT -p tcp --syn --dport 80 -m connlimit --connlimit-above 50 -j LOG --log-prefix "Slowaris Hit: "
-A INPUT -p tcp --syn --dport 80 -m connlimit --connlimit-above 50 -j DROP

P.S: Props go to unSpawn's earlier post. I failed to notice the very last line until today. Good point!


All times are GMT -5. The time now is 02:35 PM.