DDoS trough a VerliHub bug, apache is dead
 

Okay, so.. I'm ddos'ed.

I have linux box, configured by a friend of mine and me, hosted at some ISP we have here... It's a webserver ( apache with php + mysql ).

Problem is like this, I'm gonna make it simple...:
Apache/MySQL Starts -> in 1 second they're down.
Why? Becuse there is a tool(discussed here: http://www.webhostingtalk.com/showthread.php?t=574701) that connects to DC++/oDC Hubs powered by VerliHub, and then sends a certain message to ALL the users on the hub and makes them access my site ( don't have the tool, but I have something similar.. anyway ) and my port 80 is down... all the users send a very small amount of packets to the site, and block only port 80. We have to stop this!

What should I do... cause I'm desperate, I have like 17k members that are very pissed off right now...

Thanks.

Looked through that thread you've on the website above. There's a lot of opinions being floated out there already.Nothing much can be said that hasn't already been discussed there.

Here's my 2 cents:
1.Recreate the environment.
2.Start Ethereal/Tcpdump/Some Sniffer
3.Start the DDOS tool
4.Capture the packets
5.Find out what traffic is hitting the Main HUB.
6.Find out what Traffic the HUB is sending all the DC++ clients..all you need is the initial string.
7.Put all the DC++ clients into a group
8.Write a Snort rule(Content rule) which will detect the traffic. If you get hits on this rule you'll know that the rule is correct.
9.I havent personally done this but I believe you can configure Snort in something called inline mode where it works with iptables and drops packets if an alert is triggered.

Unless you find out what exactly this exploit does, there's no way you can block it.Thats how new exploit code/viruses are restricted. The reason why IP blocking is not working is probably becasue the code is configured to randomly choose IP's and send spoofed packets with this random source IP address.Just guessing..

The ideal thing though is for a few affected customers to send POC's of packet dumps to VerliHub/DC++(whichever one the exploit is for) and prove that there is a genuine problem so that they can come out with a patch. Everything else is Just a quick fix.

All the best and hope this helps a bit.

Cheers
Arvind

And how is the bandwidth? Down also?

If the packet arrives, it has already overflowed your bandwidth and the ddos is effective.
Blocking it with iptables will let the kernel breath a bit more, but won't help your connection.

The traffic has to be blackholed upstream. It's complex and cost money. Depending on your ISP and his link with his upstream providers, he's the only one who can do something.

Otherwise
Change IP?
Or change the DNS name?

Good luck!

It doesn't kill the bandwith, it kills only port 80.
I know how it works, I'll try to do what live_dont_exist recommended ( thanks m8 ), but i think it will take a while..

This tool.. doesn't work on IP, so changing IP is no good... i have to filter this, since is not killing my bandwith, it can be stopped, that's what I know...

I'm wainting for other people to gimme some info, maybe it will help.

Thanks.

ahh ok

Hum this was not clear then. Mysql on port 80?

Well if it doesn't by IP then it's by DNS, my second solution.
Anyway, your problem is different than what I thought.
Try first what is said above then.
Linux kernel also has syn flood protection but all this depends on the rate and the context of the attack.

Think what nx5000 was mentioning was change your website IP / DNS not block the offending IP's. This though is probably your last resort and you'd need to work with your ISP to change routes etc.

However hold on..are you saying its hitting Port 80.Hardcoded???Are you sure???

Or is it intelligent enough to do something like a nmap -sV and find out which port Apache is running? If not then a quick change of port + a broadcast message/some other better way to tell your customers that they need to go:

might give you some breathing space. Not much good in the long run but buys you a bit of time.

Cheers
Arvind
p.s.... Oops .. a bit late..nx5000 had already posted :) .. all the best dude.

From the discussion on the linked forum, this looks like a DoS against Apache rather than a bandwidth consumption or syn_flood attack. Can you access other ports on the system? Could you post some entries from the Apache logs showing the URL request?

In the meantime you may want to take a look at the Apache page on performance tuning against DoS attacks. The default Apache settings aren't optimal in this situation, so it may help to implement a few of the tweaks. You may also want to take a look at mod_dosevasive. AFAIK, it isn't being developed anymore and the main site is down, but you may still find it useful.

@Capt_Caveman, yea, it attacks port 80, not a bandwith consumption. I can access any other ports, but this tool.. can be configured to attack any port the script kiddie wants.

Hmmm, those kernel messages would suggest that it is a syn_flood or at least has the same effect. The fact that you don't get true connections to Apache would support that. The first kernel message is the netfilter conntrack table getting filled and subsequent packets getting dropped, along with syn_cookies getting triggered (only comes on when the conntrak table gets close to being full).

If you don't absolutely need to do statefull packet filtering, you can unload the ip_conntrack module which will get rid of the whole conntrack table altogether. Alternatively you can increase the max number of entires in the conntrack table. See this on how to find the optimal numbers to use. You can also tweak the tcp timeout values. In particular reduce the amount of time a connection can remain in SYN_RECV. Which reminds me, take a look at /proc/net/ip_conntrack and see if most of the connections are in SYN_RECV.

alot of them, like 90% are in TIME_WAIT..

Regarding tcpdump, use -x and -X options to get a hex and ASCII dump of the payload. Also use -c to specify the number of packets to capture. So the full command would be:
tcpdump -vxX -c 50.
See if you can find anything common among the DoS packets. I have a feeling these packets don't have a payload. If these are the results of the DC++ tool, you should be seeing the DC++ clients trying to issue a logon with "$Lock <key> Pk=<pk>". I believe the attack works by spoofing an $OpForceMove command and specifying port 80 on your server as the new hub for all of the hubs clients to logon to. Most of these connections don't even appear to get passed the SYN part of the TCP handshake.

It's also probably good to get a rough number idea on the number of packets/sec that are hitting your box as well.

Yeah, I would try unloading the ip_conntrack module and see if that helps any. If it does work, you may be able to get away with using the conntrack module if you decrease the TIME_WAIT setting from the default of 2 minutes down to something shorter and then increasing the max number of conntrack entries.

ok will do tcpdump -i eth1 -vxX 50

If you notice all the packets seem to start with either E..0 or E..< or E..( or E..4. You might be able to identify further patterns if you look closely.

If you have a Snort box set up by any chance or any other device which can look into the content of the packet you might want to write a rule which drops any packets with those specific start patterns.

Cheers
Arvind

Ohhhh, and do you know how could I ban those IP's? DROP them.. ?

I'm not sure. I believe that the packets should ideally get dropped even before they are processed by the kernel itself. Even if you dont drop the packets..the kernel will eventually drop it due to too many timeouts/re-transmissions..only it wont be quick enough to stop the DOS. SO ideally I'd say whatever filtering device you use - should drop it before the Kernel processes the packet.

I'm not certain though ; if someone who knows more about this can back up what I said I'd feel more comfortable.

Cheers
Arvind

You should begin by the easier solutions.
I would first try to tweak some kernel options, as said by Capt_Caveman
Also he mentionned modevasive, did you try it?

Do you also have burst limit on SYN packets in iptables?

Snort is supposed to be an intrusion detection system, not a firewall. It can be very resource intensive. Running it on the same machine as apache could make things worse IMO.

Yes, we have mod_evasive
We also have SYNWatch on that interface.
Don't have syn burst limit (iptables), but we tried it days ago and was not working.
Will try the kernel mods.


Any other ideea will be great, and about snort ... yeah ... at our traffic rating
it will be a great resource consumer.


Regards,

That thread is closed and I think this is why?
feels like i am watching a movie.
<sorry for contributing nothing>

this is the output from snort .. please give us a rule to drop $MyNick .....

If you have Snort-Inline working then you can simply use the 'drop' action. Once you are comfortable that everything is working properly you can switch to sdrop if you like (probably using a threshold limit would be better).

Though the string "Pk=DCPLUSPLUS" might be useful triggering as well. Also make sure to edit the config file properly so that $HTTP_SERVERS and $HTTP_PORTS variables are accurate.

If you are still working on getting Snort-Inline working you may find this helpful.

If you are concerned about the resources consumed by Snort, you can also use iptables string match. I believe it's still not part of the basic iptables installation, so you'd need to do the whole patch-o-matic and kernel recompile to use it.

a little questions .. all that $EXTERNAL_NET and so over must be replaced with my ip / port etc?

ty for help ...

I usually define $HOME_NET with whatever IPs or IP ranges are on my network and then just set $EXTERNAL_NET as !$HOME_NET. $HTTP_SERVERS and $HTTP_PORTS are what they sound like. You can use actual IP addresses or ranges of IPs (in CIDR notation) in place of any of those variables if you like.

The Snort docs are excellent resources:
http://www.snort.org/docs/faq/1Q05/node34.html

now ... what should i say to iptables:

iptables -A INPUT -p tcp --dport 80 -j QUEUE

or

iptables -A OUTPUT -p tcp --dport 80 -j QUEUE

?

and the final rule for me is:

snort loggs a lot of data in log folder but the site is still unreachable ... any updates pls/.

You'll want to send packets coming into the box from the internet to the QUEUE, so this will either be INPUT or FORWARD. If Apache is running on that machine then you'll send packets in INPUT to the QUEUE, if the webserver is on another machine inside the network and DNATed then you'll want to use FORWARD. You use OUTPUT for analyzing packets going outbound from the system/network to the internet (which you don't need and adds overhead).

i've done all the above but still the site can't be reached .. and in console i still see Possible SYN Flood on port 80 messages.

any other ideea or a better rule.

Yup, me and ipv6waiting are gettin' crazy over here xD

/var/log/snort is like this:
But, no good :( !

You may want to change the 'drop' action to 'reject' to eliminate retries. Also, take a look in /var/log/snort and see how many unique IPs are you seeing? How many alerts per IP? What I'm getting at is that if these are coming from a limited number of IP addresses, you can block them out-right with IPtables.

If you've legitimately tried all of the above recommendations (TCP/Netfilter/Apache tuning) then you're really at the limits of what you can do by yourself to mitigate it. The next option would to be to throw several load-balancing proxy servers in front of the webserver and let them split the connection load and then only the valid HTTP requests would be proxied to the internal webserver.

I think you're also at the point where you should contact your ISP and see if they can help you mitigate the attack on their end. These are real connection attempts, not spoofed packets, so they should be able to null route much of it. So I would definitely contact them and let them know what's going on. Make sure to explain in detail the type of traffic that you are seeing (DC++ connection attempts on port 80) and why (malicious redirection of DC++ clients using the ForceMove command). Send relevant logs as well.

All this time.. we still h ave'nt fixed this issue, if you look here :

Snort received 118092 packets
Analyzed: 13297(11.260%)
Dropped: 104585(88.562%)
Outstanding: 210(0.178%)
===============================================================================
Breakdown by protocol:
TCP: 13244 (99.601%)
UDP: 42 (0.316%)
ICMP: 6 (0.045%)
ARP: 5 (0.038%)
EAPOL: 0 (0.000%)
IPv6: 0 (0.000%)
ETHLOOP: 0 (0.000%)
IPX: 0 (0.000%)
FRAG: 0 (0.000%)
OTHER: 0 (0.000%)
DISCARD: 0 (0.000%)
===============================================================================
Action Stats:
ALERTS: 0
LOGGED: 13297
PASSED: 0
===============================================================================
Snort exiting

I think that... we have a bunch of packets, don't we? lol. Oh.. that happened in only 5 SECONDS.

You might want to disable all your rules except the DC++ one. That will drastically reduce the number of alerts Snort is triggering and setting off.

Yeah its normal ..the number of packets you're receiving ;) .

Cheers
Arvind

This too?!

netstat -s
Ip:
35475696 total packets received
1 with invalid headers
3484 with invalid addresses
0 forwarded
0 incoming packets discarded
35470557 incoming packets delivered
34932772 requests sent out
4 reassemblies required
2 packets reassembled ok
Icmp:
1614 ICMP messages received
1258 input ICMP message failed.
ICMP input histogram:
destination unreachable: 339
timeout in transit: 921
source quenches: 1
echo requests: 353
8107 ICMP messages sent
0 ICMP messages failed
ICMP output histogram:
destination unreachable: 7754
echo replies: 353
Tcp:
0 active connections openings
1215 passive connection openings
17 failed connection attempts
134 connection resets received
504 connections established
35435997 segments received
34899221 segments send out
12 segments retransmited
71 bad segments received.
34821220 resets sent
Udp:
24246 packets received
7572 packets to unknown port received.
0 packet receive errors
25961 packets sent
TcpExt:
2338 SYN cookies sent
5327 SYN cookies received
3660 invalid SYN cookies received
17 resets received for embryonic SYN_RECV sockets
497 TCP sockets finished time wait in fast timer
129 delayed acks sent
Quick ack mode was activated 3 times
17326 times the listen queue of a socket overflowed
17326 SYNs to LISTEN sockets ignored
11 packets directly queued to recvmsg prequeue.
8 packets directly received from prequeue
4455 packets header predicted
9954 acknowledgments not containing data received
8465 predicted acknowledgments
2 times recovered from packet loss due to SACK data
1 TCP data loss events
10 fast retransmits
2 other TCP timeouts
3 DSACKs sent for old packets

Cheers m8's..

Can you try with iptables to drop all traffic coming to port 80 but only let your IP get in on port 80?
Then try to see if you can reach your website.

That's not a definitive solution, just a try to see if iptables alone can help.

I was never ddosed, so it's just suposition.


So what's the attack exactly?

It connects to port 80 and sends something if I got it right?
Or is only (potentially spoofed) Syn attack?
If it connects, can you give a line of these connections from your apache log?
Also how many different IPs dos you? For this you will need some self-made scripts.



Yes, that would be a good thing to try also.
In the same idea, you could use the tarpit rule. It sometimes gives good results.

Hmm, I'll try that, to block all the other IP-s and allow only myself...nice idea.

yea you got it right, it sends some crappy message, but nothing goes to the apace_log, nothing :(... so I belive it's not a valid request, for apache, I mean yea.. because the packet isn't a GET, is a $MyNick .. bla bla...

Oh.. and A L O T of IPs ddos me, cause it's really easy to find some hubs with lots of users...

Basically maliciously redirecting legitimate DC++ clients from some DC++ hub to victims webserver. Attacker spoofs/sends a ForceMove command to all the legitimate clients on the hub who are redirected to a "new hub" by the attacker. Except instead of redirecting them to a new DC++ hub, the attacker points all the clients at the victims webserver on port 80. All the clients then send DC++ login packets (the whole $MyNick thing) to the victims webserver which overloads it. It sort of resembles a Smurf attack. Though in this case there are soo many connection attempts that it's hard to actually see if all of them are the DC++ logins or if it's a mixed attack with DC++ and SYN_flood. They are seeing a lot of packets, which is why they should contact their ISP. Look at all the Snort logs, all those packets are arriving in less than a second.


*If* any of these packets are getting through (and if Apache isn't fubared), then it should likely be serving up HTTP 501 "Method Not Implemented" responses as there isn't a valid HTTP method (GET/POST/FIND) in these DC++ packets

I would be careful in how you do that. It will decrease the amount of data transmitted per connection (by shrinking window size to 0), but each connection will retransmit until timeout so that you'll see and overall increase in number of packets. Plus connections will stay in the conntrack table longer than if the system just sent an RST. If you do, don't load the conntrack module or use the NOTRACK target in the raw table.

Ok, let's say I use:

iptables -A INPUT -m string --algo bm --string "DCPLUSPLUS" -j DROP

But I don't want to DROP only the packet, I want to DROP or BlackList the IP.
How do I do that?

I understood the vector (p2p.. bad..) but not the method. So easy... Looks like a big flaw in this DC protocol. Thanks for explaining.
Yes that's a lot. Poor snort :D
Modifiying a DC++ and connect to these hubs could be funny. Depending on the celverness of the protocol and of distribution of nodes... the new era of worms.
Legitimate traffic. Characterising is important for fighting against it.
Yes sure, tarpit is a trick to handle with care.. works in very few case, looking at the bandwidth it's probably not adapted.
It's why I would run some scripts with tcpdump output to know howmany source ip there are.
And sure yes, not any connection tracking, it's useless at this point and adds too much processing.
Huhh that's harder than my idea.
Poor iptables :D

cOOkie, your ISP contact is on holiday or what?
He probably has more tools and more ways to act than you???

My ISP kind of sucks, he says he cannot help me ;).

So... what to do what to do... ipp2p is no good.

You could do a log message before the DROP rule. Use pretty much the same rule but put a LOG target on it and use the --log-prefix "SOME MESSAGE". You'll need to use the 'limit' match so only a few packets are logged (otherwise your log will probably get bombed). Something like:
Then have script parse the log file every 15 min and add the new IP addresses. I'd also recommend that you use the REJECT target, so that the DC++ clients get an immediate icmp unreachable message and don't keep trying to resend the same packet.

There are probably some scripts in the "Failed SSH login attempts" thread near the top of the forum that you can alter for your purposes. Also make sure that when you add a new iptables rule to ban an IP, make sure that it is added above the LOG rule that way you don't see that same IP in the logs anymore and you'll be getting fresh unique IPs to ban.

You could likely do the same thing using the directories in /var/log/snort. You still haven't said how many unique IP addresses you are seeing, which is an important question. If you have a zillion IP addresses that you have to ban then you are going to have some problems doing that with IP tables (though I guess slow packet handling is better than the current situation).

I thought about that, but I think the catch is that you can't tell what hubs these clients were connected to beforehand based on the packets alone (there's no 'referer' or anything). Otherwise it might be interesting to see where the ForceMove packets are coming from.

Ok man, I'll try to create some rules, thx again for your help.

Well, I can't do it.. I need some help, again..

From what I've seen, I have to LOG all the IP's with the STRING, and DROP them...
How do I do that? I need to log them, that's easy, but how do I read the IP's from the LOGS and block them? Thats where I'm stuck...
I don't want to filter only port 80, or I donno.. I want to -s BLACKLISTEDIP -j DROP :).

Can you help? I'm not really into this logging stuff.. i'm stuck..

Not too much into iptables so cant help you there but I believe Capt Caveman was saying that you needed to hack together a perl script or something. You need to write this ;) Heres a start:

1.You've logged all those connections and you have all the bad IP addresses
2.Let your script parse the log file and read only all the bad IP addresses
3.Put all the bad IP addresses into a file
4.Create your exact iptables rule for dropping all packets from a specific IP.
5.Put this in a loop each time only changin the IP...each IP will be replaced by a bad IP
6.This way you have 1 trillion bad IP blocking iptables rules.
7.Once you have these rules ready dump them into the iptables config file or wherever your other rules are. Make sure you still allow legitimate traffic like yourself or you'll end up blocking yourself OUT

Hope this clarifies things a bit? Anyone please correct me if I'm wrong.

All the best and keep that chin up :)

Cheers
Arvind

This is just an idea to block offending IP's using (requiring) Snort, Guardian and iptables ("ipset"). I haven't read *ANY* data on this attack: no Apache logs and no packet captures. It may or may not work so YMMV(VM) as usual.


Short overview
Tune Snort performance-wise only looking at TCP/80 traffic and only using the custom ruleset, make Guardian fetch the IP's and block using iptables' ipset.


Caveats
You need to use iptables "ipset" module. Blocking using other iptables modules may have a resource starvation risk (memory).


What you need
I. Ipset.
See http://ipset.netfilter.org/features.html for default instructions. We'll opt for blocking whole netranges here,
- configure a drop list:
(Do check syntax yourself thrice, I'm not responsable for errors: you are.)

II. Snort
- Copy your /etc/snort/snort.conf to /etc/snort/snort_current.conf
- Edit /etc/snort/snort_current.conf and disable preprocessors you don't need like rpc_decode, bo, telnet, portscan, arpspoof, perfmon, xlink. Set "config detection: search-method lowmem". If all tests out well you can disable this later on. Disable unified and payload logging and make Snort log to it's own snort.alert file in fast mode. You only need the offending IP after all.
- Comment out all the rulesets.
- Create a custom ruleset (say "/etc/snort/dcplusplus.rules"):
Now add that "/etc/snort/dcplusplus.rules" to your config.
- Enable "include threshold.conf" and edit accordingly (alert per IP once per 3600 secs):
- Add the mapping (for reference):
- Add reference (for reference):
- Edit or add to your BPF config (to keep from checking traffic from "known good" ranges). Do substitute the subnet with your management IP ranges, `man tcpdump` for BPF filter details:
Edit your Snort commandline to include "-c /etc/snort/snort_current.conf -F /etc/snort/bpf.conf -A fast -N". Remove any commandline options that log packets or do unified logging. Test adding the "-T" switch.

III. Guardian.
See installation and configuration instructions in the tarball: http://www.chaotic.org/guardian/guardian-1.7.tar.gz
- Change Guardian's block.sh script iptables line so the subnet gets added to the set:
IV. TEST WELL BEFORE USING. Do use common sense and check syntax yourself. Check thrice. I'm not responsable for errors: you are.

HTH but remember YMMV(VM).

might be a tad off track but..

can't a rule be created to set the firewall to drop packets for say 5 minutes from an ip where that ip sends say, 30+ requests per second, and list that ip as suspicious.

then once an hour say, if they reappear we move those into a day long banlist, then if the 'usual suspects' keep appearing, move those to a permaban since those are the worst offending ips, probably botted, and maybe inform their isp?

i've no idea how to go about that in the software you mention tho.

Okay...its almost impossible to defend yourself against this attack. Did i say almost?...no...its impossible. Heres how it works:

You have 2 clients connected to a hub( C1 C2 ). When C1 tries to connect to C2 it initiates the Client2Client handshake specific to DC++. C1 sends a $ConnectToMe to the hub containing its IP address and port. The hub then sends the same command with all of C1's info ( IP and port ) to C2.
C2 initiates a connection with C1 and sends $MyNick and immediately after that $Lock. C1 responds with $Key and $Lock and waits for C2 to send its $Key command after which C1 finally sends its $MyNick back to C2. So to make a long story short its like this:

C-->Hub: $ConnectToMe
Hub-->C2: $ConnectToMe
C2-->C1: $MyNick
C2-->C1: $Lock
C1-->C2: $Key
C1-->C2: $Lock
C2-->C1: $Key
C1-->C2: $MyNick

This exploit basically takes advantage of $ConnectToMe and sends multiple requests to the hub with the IP and port of the victim server as its own. The Clients on that Hub receive the requests and initiate a connection back to the IP and Port they receive( which can be a web server or an SQL server).

The only solution to this is to block all the IP's connecting to you and feeding you this bogus string. It will NOT solve your problem entirely because you will still experience DOS until those IP's get droped and that happens only after u already receave a connection and an apache child gets spawned. Decrease the "Timeout" limit to say 30 seconds in httpd.conf and hope for the best. At least this way you wont have a bogus connection to your server for 6 mins. We ended up dropping over 8000 IP's in less then 30 mins.

Good Luck!

Hmmm..

One solution is to blacklist ip's, the other solution.. I think could be to filter the http requests so if it's no GET / POST, etc. to block it.

ok, the blacklisting ip's sounds ok .. but what firewall or something that to that is arround.. any info would be great ... thanks.

that would be an idea...its just that the initial connection would still be made and the apache child would still spawn. A colleague of mine suggested writing an Apache module that would drop the connection after 1 or 2 seconds if no GET / POST would be sent in addition to blackinsting the source IP of the originating malformed string. Thus far its pretty anoying defending against this kind of attack.