LinuxQuestions.org

LinuxQuestions.org (/questions/)
-   Linux - Security (https://www.linuxquestions.org/questions/linux-security-4/)
-   -   DDoS trough a VerliHub bug, apache is dead (https://www.linuxquestions.org/questions/linux-security-4/ddos-trough-a-verlihub-bug-apache-is-dead-533834/)

c00kie 03-02-2007 05:25 AM

DDoS trough a VerliHub bug, apache is dead
 
Okay, so.. I'm ddos'ed.

I have linux box, configured by a friend of mine and me, hosted at some ISP we have here... It's a webserver ( apache with php + mysql ).

Problem is like this, I'm gonna make it simple...:
Apache/MySQL Starts -> in 1 second they're down.
Why? Becuse there is a tool(discussed here: http://www.webhostingtalk.com/showthread.php?t=574701) that connects to DC++/oDC Hubs powered by VerliHub, and then sends a certain message to ALL the users on the hub and makes them access my site ( don't have the tool, but I have something similar.. anyway ) and my port 80 is down... all the users send a very small amount of packets to the site, and block only port 80. We have to stop this!

What should I do... cause I'm desperate, I have like 17k members that are very pissed off right now...

Thanks.

live_dont_exist 03-02-2007 07:32 AM

Looked through that thread you've on the website above. There's a lot of opinions being floated out there already.Nothing much can be said that hasn't already been discussed there.

Here's my 2 cents:
1.Recreate the environment.
2.Start Ethereal/Tcpdump/Some Sniffer
3.Start the DDOS tool
4.Capture the packets
5.Find out what traffic is hitting the Main HUB.
6.Find out what Traffic the HUB is sending all the DC++ clients..all you need is the initial string.
7.Put all the DC++ clients into a group
8.Write a Snort rule(Content rule) which will detect the traffic. If you get hits on this rule you'll know that the rule is correct.
9.I havent personally done this but I believe you can configure Snort in something called inline mode where it works with iptables and drops packets if an alert is triggered.

Unless you find out what exactly this exploit does, there's no way you can block it.Thats how new exploit code/viruses are restricted. The reason why IP blocking is not working is probably becasue the code is configured to randomly choose IP's and send spoofed packets with this random source IP address.Just guessing..

The ideal thing though is for a few affected customers to send POC's of packet dumps to VerliHub/DC++(whichever one the exploit is for) and prove that there is a genuine problem so that they can come out with a patch. Everything else is Just a quick fix.

All the best and hope this helps a bit.

Cheers
Arvind

nx5000 03-02-2007 08:01 AM

Quote:

Apache/MySQL Starts -> in 1 second they're down.
And how is the bandwidth? Down also?

If the packet arrives, it has already overflowed your bandwidth and the ddos is effective.
Blocking it with iptables will let the kernel breath a bit more, but won't help your connection.

The traffic has to be blackholed upstream. It's complex and cost money. Depending on your ISP and his link with his upstream providers, he's the only one who can do something.

Otherwise
Change IP?
Or change the DNS name?

Good luck!

c00kie 03-02-2007 08:19 AM

It doesn't kill the bandwith, it kills only port 80.
I know how it works, I'll try to do what live_dont_exist recommended ( thanks m8 ), but i think it will take a while..

This tool.. doesn't work on IP, so changing IP is no good... i have to filter this, since is not killing my bandwith, it can be stopped, that's what I know...

I'm wainting for other people to gimme some info, maybe it will help.

Thanks.

nx5000 03-02-2007 08:33 AM

Quote:

Originally Posted by c00kie
It doesn't kill the bandwith, it kills only port 80.
I know how it works, I'll try to do what live_dont_exist recommended ( thanks m8 ), but i think it will take a while..

ahh ok

Quote:

Apache/MySQL Starts -> in 1 second they're down.
Hum this was not clear then. Mysql on port 80?

Quote:

This tool.. doesn't work on IP, so changing IP is no good... i have to filter this, since is not killing my bandwith, it can be stopped, that's what I know...
Well if it doesn't by IP then it's by DNS, my second solution.
Anyway, your problem is different than what I thought.
Try first what is said above then.
Linux kernel also has syn flood protection but all this depends on the rate and the context of the attack.

live_dont_exist 03-02-2007 08:38 AM

Quote:

Originally Posted by c00kie
This tool.. doesn't work on IP, so changing IP is no good... i have to filter this, since is not killing my bandwith, it can be stopped, that's what I know...

Thanks.

Think what nx5000 was mentioning was change your website IP / DNS not block the offending IP's. This though is probably your last resort and you'd need to work with your ISP to change routes etc.

However hold on..are you saying its hitting Port 80.Hardcoded???Are you sure???

Or is it intelligent enough to do something like a nmap -sV and find out which port Apache is running? If not then a quick change of port + a broadcast message/some other better way to tell your customers that they need to go:

Code:

www.website.com:53467 instead of www.website.com
might give you some breathing space. Not much good in the long run but buys you a bit of time.

Cheers
Arvind
p.s.... Oops .. a bit late..nx5000 had already posted :) .. all the best dude.

Capt_Caveman 03-03-2007 04:24 AM

From the discussion on the linked forum, this looks like a DoS against Apache rather than a bandwidth consumption or syn_flood attack. Can you access other ports on the system? Could you post some entries from the Apache logs showing the URL request?

In the meantime you may want to take a look at the Apache page on performance tuning against DoS attacks. The default Apache settings aren't optimal in this situation, so it may help to implement a few of the tweaks. You may also want to take a look at mod_dosevasive. AFAIK, it isn't being developed anymore and the main site is down, but you may still find it useful.

c00kie 03-03-2007 05:51 AM

@Capt_Caveman, yea, it attacks port 80, not a bandwith consumption. I can access any other ports, but this tool.. can be configured to attack any port the script kiddie wants.

Capt_Caveman 03-03-2007 10:41 AM

Hmmm, those kernel messages would suggest that it is a syn_flood or at least has the same effect. The fact that you don't get true connections to Apache would support that. The first kernel message is the netfilter conntrack table getting filled and subsequent packets getting dropped, along with syn_cookies getting triggered (only comes on when the conntrak table gets close to being full).

If you don't absolutely need to do statefull packet filtering, you can unload the ip_conntrack module which will get rid of the whole conntrack table altogether. Alternatively you can increase the max number of entires in the conntrack table. See this on how to find the optimal numbers to use. You can also tweak the tcp timeout values. In particular reduce the amount of time a connection can remain in SYN_RECV. Which reminds me, take a look at /proc/net/ip_conntrack and see if most of the connections are in SYN_RECV.

c00kie 03-03-2007 06:16 PM

alot of them, like 90% are in TIME_WAIT..

Capt_Caveman 03-03-2007 08:14 PM

Regarding tcpdump, use -x and -X options to get a hex and ASCII dump of the payload. Also use -c to specify the number of packets to capture. So the full command would be:
tcpdump -vxX -c 50.
See if you can find anything common among the DoS packets. I have a feeling these packets don't have a payload. If these are the results of the DC++ tool, you should be seeing the DC++ clients trying to issue a logon with "$Lock <key> Pk=<pk>". I believe the attack works by spoofing an $OpForceMove command and specifying port 80 on your server as the new hub for all of the hubs clients to logon to. Most of these connections don't even appear to get passed the SYN part of the TCP handshake.

It's also probably good to get a rough number idea on the number of packets/sec that are hitting your box as well.

Capt_Caveman 03-03-2007 08:27 PM

Quote:

Originally Posted by c00kie
in /proc/net/ip_conntrack most of the connections are like this

PHP Code:

tcp      6 74 TIME_WAIT src=194.126.185.100 dst=194.126.185.213 sport=35785 dport=62029 packets=4 bytes=216 src=194.126.185.213 dst=194.126.185.100 sport=62029 dport=35785 packets=3 bytes=168 [ASSUREDmark=0 secmark=use=

alot of them, like 90% are in TIME_WAIT..

Yeah, I would try unloading the ip_conntrack module and see if that helps any. If it does work, you may be able to get away with using the conntrack module if you decrease the TIME_WAIT setting from the default of 2 minutes down to something shorter and then increasing the max number of conntrack entries.

c00kie 03-05-2007 03:07 AM

ok will do tcpdump -i eth1 -vxX 50

live_dont_exist 03-05-2007 03:36 AM

If you notice all the packets seem to start with either E..0 or E..< or E..( or E..4. You might be able to identify further patterns if you look closely.

If you have a Snort box set up by any chance or any other device which can look into the content of the packet you might want to write a rule which drops any packets with those specific start patterns.

Cheers
Arvind

c00kie 03-05-2007 04:09 AM

Ohhhh, and do you know how could I ban those IP's? DROP them.. ?


All times are GMT -5. The time now is 12:31 AM.