Linux - ServerThis forum is for the discussion of Linux Software used in a server related context.
Notices
Welcome to LinuxQuestions.org, a friendly and active Linux Community.
You are currently viewing LQ as a guest. By joining our community you will have the ability to post topics, receive our newsletter, use the advanced search, subscribe to threads and access many other special features. Registration is quick, simple and absolutely free. Join our community today!
Note that registered members see fewer ads, and ContentLink is completely disabled once you log in.
If you have any problems with the registration process or your account login, please contact us. If you need to reset your password, click here.
Having a problem logging in? Please visit this page to clear all LQ-related cookies.
Get a virtual cloud desktop with the Linux distro that you want in less than five minutes with Shells! With over 10 pre-installed distros to choose from, the worry-free installation life is here! Whether you are a digital nomad or just looking for flexibility, Shells can put your Linux machine on the device that you want to use.
Exclusive for LQ members, get up to 45% off per month. Click here for more info.
Well if you have Msn/ Skype or stuff, maybe I could show you.
My email address is available if you need to share nfo you rather not pass in public but other than that, thanks for the offer but I couldn't do that.
Quote:
Originally Posted by AsadMoeen
V1.00 is a separate game and all other updates are separate so no problem with the game.
OK, if you say so...
Quote:
Originally Posted by AsadMoeen
I also run a Cod4 server and on testing, it can also be flooded.
You talk about flooding but nothing can be inferred from the little traffic you've shown in this thread. Game servers use ports in the 20K range and AFAIK UDP/27015 or near are used for beacons like server game listings and game status updates. I need to see a lot more traffic, especially from before any (perceived) attack starts.
Quote:
Originally Posted by AsadMoeen
When I restart my machine, it works for 2 hours.
Show us proof it does instead. Show us the iptables recent module is available ('cat /proc/net/ip_tables_matches'), loaded ('grep recent /proc/modules') and that your default bucket actually has items ('wc -l /proc/net/ipt_recent/DEFAULT').
Quote:
Originally Posted by AsadMoeen
I can pay a bit to someone who can help out, he may access my machine.
Such work requires root privileges. You have absolutely no clue if somebody offering help is trustworthy, capable or able to fix the problem (post #6 again). And IMHO you shouldn't have to pay people to fix things.
I have double checked the syntax and the working modules.
What was thought to be working with a restart temporarily was actually not the case but something similar.
It does work partially with a restart, here is more information:
The following commands do the job very well in blocking but there is a big issue.
Quote:
-A INPUT -p udp -m udp --dport 0:65535 -m state --state NEW -m recent --set --name UDP --rsource
-A INPUT -p udp -m udp --dport 0:65535 -m state --state NEW -m recent --update --seconds 1 --hitcount 10 --name UDP --rsource -j DROP
-A INPUT -p tcp -m tcp --dport 0:65535 -m state --state NEW -m recent --set --name TCP --rsource
-A INPUT -p tcp -m tcp --dport 0:65535 -m state --state NEW -m recent --update --seconds 1 --hitcount 10 --rttl --name TCP --rsource -j DROP
These would block automatically any IP on any port that makes more than 10 connections in a second. But the problem I am having here is that the rules do partial blocking. Lets say 5 different IP's are attacking me, no IP is getting blocked although iptables rules are intact. I restarted the machine with the same iptables rules that previously didn't block the attacker's IPs loaded on startup and suddenly they start blocking all the 5 attackers. But then lets say if attacker uses 2 more IPs to make a total of 7 IPs then the newer 2 do not get blocked while the previous 5 are blocked. This was the reason why I thought restart blocked the attacker for 2 hours but actually it just blocked the previous attackers.
So a restart once again would block all those 7 IPs and not the ones who attack after. I am not sure why this is happening.
And the attack is basically a UDP Flood, I've been under it before and the commands I mentioned are used to block it down. I have a tool myself for testing written for Debian, I've attached it in case if you wanna look at it. However, there are versions of this tool available for other Operating Systems too. You could just chmod +x Flood.sh and do "./Flood.sh IP port length time" for testing the attack on a UDP port. The attacker is using a lot of IPs ( surely botnets ) and the attack is being generated from http or some game-server ports.
Packet length is constant ( 14 ) so we could also use
-A INPUT -p udp -m udp --dport 0:65535 -m length --length 14 -j DROP
Which is a more simple but a wider command and the problem with this is the same as mentioned above.
While I'm waiting for you to finally provide me with packet captures I asked for here's an idea: how about you shove a machine in front of those VPSes that actually has all the modules required (like a physical host or a XEN machine) and route traffic through that? That way you don't have to deal with uncooperative hosting companies or OpenVZ deficiencies.
While I'm waiting for you to finally provide me with packet captures I asked for here's an idea: how about you shove a machine in front of those VPSes that actually has all the modules required (like a physical host or a XEN machine) and route traffic through that? That way you don't have to deal with uncooperative hosting companies or OpenVZ deficiencies.
I am about to send those details with screenshots to your Email.
To your other concern: One of the machines I use is Xen HVM which already has everything installed but that is also a victim sadly.
You have reverted to useless dialogue. You have been asked for specific information and consistently pay no attention and fail to respond with the requested information.
I think you don't have a clue, sorry to be so rude but that is the way you come across.
I for one will bail out from any further interaction with you.
* where "eth" is your 'net-facing Ethernet device name (like "eth0") and "/path/to/eth.pcap" is the path and file name on a partition with enough space to support storing 3 packet captures of 50 megs each.
Once you have got these packet captures please send me an email to discuss where I can download the files for analysis.
** No other information or email is needed at this point.
What do you mean? I have provided everything that was asked in the topic while some private information about packet captures was mailed to unspawn with screenshots.
Per pcap file there are roughly 140K packets. Of those 140K approximately 110K is UDP traffic, specifically game status messages (wireshark filter "data.data contains 74:61:74:75:73"), packet length divided equally between between 40-80 and 640-1280 or a 15 or 680 byte payload.
Thank you for posting this, here is some more information for the rest:
Quote:
I checked those packet capture files too and what it appears to be is that the "getstatus" string is flooding the game-servers. I'll give you my opinions on that one since its usually the case with game-servers because they start sending the output of the query ( the output that includes server information like name, slots and stuff ) and since they flood the query, the output reaches high file size. The attacker's IP is spoofed and they can use other game-server's ports to flood another server. A guy came to me that I was attacking his Cod4 server although he was attacking my SOF2 server and basically we were both attacking each other due to the real attacker's IP being spoofed on his machine. He gave this solution for firewall:
I have finally found the problem and an answer to all my questions, although still waiting to find a solution.
Here is the crack down:
After probably testing 7 days that included restarting, attacks from my own test machines and a large number of iptables rules I have finally reached the conclusion. The iptables rules are working and doing exactly what they should but they work only when the game-server is stopped which is why after a system restart the following things happened:
Attacker was sending "getstatus" requests to the game-server port.
The machine freshly restarted, so the game server wasn't running.
Tcpdump just sent "Port Unreaachable" to the attacker which hardly made 200kb/s output.
Output was "Zero" when rate-limit rules were applied so no output to the attacker.
As we all know that incoming packets can never be blocked by iptables, the output can be controlled but depending upon the application running at the port. So now when the Game-Server started, then because of the server itself; rate limit wasn't working. The game-server was replying to the attacker's "getstatus" requests which caused the large output to the attacker's IP. Please correct me if I am wrong with this conclusion.
Now some people have fixed this in the game server's executables already by manually applying some rate-limit in the patches like 1 request/sec while some are using their own scripts. Since I can't create any such patches or use newer version of the game, I'll have to use some scripts to do the job or some work-around the iptables.
So most of you now will be fully aware of what I mean to say here, any suggestions to fix the issue or why the solution in the above link doesn't work for me would be greatly appreciated. Probably just the way the application works, not sure.
His script I cronjob'd to run every 1 minute and automatically bans the address flooding "getstatus" requests. But I do also need to know a direct and instant solution, in case he floods harder; he would already have given my gameservers some piece of lag and it would take 1 minute to ban his IP.
iptables is the only instant solution so if anyone can come up with some direct commands or answer why the above ones don't work for me. I probably think that when the Game Server is running, it has some higher precedence than iptables rules so the iptables rules work only when the game-server is stopped.
The iptables rules are working and doing exactly what they should but they work only when the game-server is stopped
I do not believe this is a correct analysis.
Quote:
Originally Posted by AsadMoeen
Tcpdump just sent "Port Unreaachable" to the attacker which hardly made 200kb/s output.
tcpdump doesn't send anything: it just captures traffic.
Quote:
Originally Posted by AsadMoeen
As we all know that incoming packets can never be blocked by iptables,
Your analysis makes you believe that. However it was not correct and therefore your conclusions are not correct.
Quote:
Originally Posted by AsadMoeen
I can't create any such patches or use newer version of the game
That's problem number #0: you are not willing to patch or use a patched version of the game. This basically means you yourself allow part of the problem to continue unnecessarily.
...and that's problem #1: because, for reasons unknown, your iptables modules don't work as advertised you won't be getting any use out of those rules until you fix the reason for the iptables modules not working. (Should you have been able to use ipset then you could try a suggestion as posted here but since you already have problems with iptables modules that's probably out of the question.)
Quote:
Originally Posted by AsadMoeen
His script I cronjob'd to run every 1 minute and automatically bans the address flooding "getstatus" requests.
There's a few problems with the script. It is inefficient for 0) running tcpdump repeatedly, 1) disabling resolution only partially with "-f" (and not "-n -nn -N"), 2) running tcpdump without a BPF filter (like 'udp dst 20100'), 3) running "grep|awk|cut"-like chains where you could run awk with "-F" and a '/searchterm/' filter and 4) using the filter table INPUT chain for traffic that would be better dropped in the raw table PREROUTING chain. Most importantly though a server is required to send "getstatus" traffic to clients and other servers, so rate limiting at least allows for some functionality as opposed to the script which blocks IP addresses completely. Then again you're not the only one having OpenVZ-related trouble. So if blocking IP addresses based on packet payload is all that remains then at least run something more efficient like Snort:
- create a directory "/etc/snort" (if the Snort installation didn't do so already) and save this completely self-contained configuration file as "/etc/snort/snort_game.conf":
Code:
var HOME_NET nnn.nnn.nnn.nnn/32
var EXTERNAL_NET !$HOME_NET
preprocessor stream5_global: track_icmp no, track_tcp no, track_udp yes, max_udp 1052672
preprocessor stream5_udp: timeout 60
portvar GAME_PORTS [20100:29999]
alert udp $EXTERNAL_NET any -> $HOME_NET $GAME_PORTS (msg:"getstatus_thresh"; content:"getstatus"; flow:to_server, established; threshold: type both, track by_src, count 2, seconds 10; reference:url,www.linuxquestions.org/questions/linux-server-73/iptables-rate-limit-to-block-ddos-931931/; sid: 999999998; rev:1;)
- create a directory "/var/log/snort" (again, if the Snort installation didn't do so already) and ensure the user Snort runs as can write to it,
- run Snort as 'snort -A fast -q -c /etc/snort/snort_game.conf -k none -K none -l /var/log/snort'.
This will generate and update a plain text /var/log/snort/alert log file, without requiring a kludge like the "Q3-Engine-GetStatus-Flood-Fixer", which is only written to if the rule is triggered at least 2 times in 10 seconds (feel free to adjust) and which guardian, fail2ban or whatever continuously log-reading application you favor, can read and create 'iptables -t raw -A PREROUTING -s $IP -j DROP' rules for.
Quote:
Originally Posted by AsadMoeen
if anyone can (..) answer why the above ones don't work for me.
That would minimally require IPTABLES_SAVE_COUNTER, IPTABLES_STATUS_NUMERIC, IPTABLES_STATUS_VERBOSE and IPTABLES_STATUS_LINENUMBERS to be set to "yes" in iptables-config,
running 'iptables-save > /path/to/iptables`date +%s`.log' at some interval and regularly checking /proc/net/ipt_recent/* lists so there's actually hard evidence of traffic. Your iptables rule set might need some adjusting beforehand. Here's a suggestion:
Code:
# Generated by iptables-save v1.4.2 on Wed Feb 1 17:28:45 2012
*nat
:PREROUTING ACCEPT [0:0]
# Block non-routable traffic: comment out any LAN ranges you use.
-A PREROUTING -i venet0 -s 0.0.0.0/8 -j DROP
-A PREROUTING -i venet0 -s 10.0.0.0/8 -j DROP
-A PREROUTING -i venet0 -s 127.0.0.0/8 -j DROP
-A PREROUTING -i venet0 -s 169.254.0.0/16 -j DROP
-A PREROUTING -i venet0 -s 172.16.0.0/12 -j DROP
-A PREROUTING -i venet0 -s 192.0.0.0/24 -j DROP
-A PREROUTING -i venet0 -s 192.0.2.0/24 -j DROP
-A PREROUTING -i venet0 -s 192.168.0.0/16 -j DROP
-A PREROUTING -i venet0 -s 198.18.0.0/15 -j DROP
-A PREROUTING -i venet0 -s 198.51.100.0/24 -j DROP
-A PREROUTING -i venet0 -s 203.0.113.0/24 -j DROP
:POSTROUTING DROP [0:0]
:OUTPUT ACCEPT [0:0]
# Block non-routable traffic: comment out any LAN ranges you use.
-A OUTPUT -o venet0 -s 0.0.0.0/8 -j DROP
-A OUTPUT -o venet0 -s 10.0.0.0/8 -j DROP
-A OUTPUT -o venet0 -s 127.0.0.0/8 -j DROP
-A OUTPUT -o venet0 -s 169.254.0.0/16 -j DROP
-A OUTPUT -o venet0 -s 172.16.0.0/12 -j DROP
-A OUTPUT -o venet0 -s 192.0.0.0/24 -j DROP
-A OUTPUT -o venet0 -s 192.0.2.0/24 -j DROP
-A OUTPUT -o venet0 -s 192.168.0.0/16 -j DROP
-A OUTPUT -o venet0 -s 198.18.0.0/15 -j DROP
-A OUTPUT -o venet0 -s 198.51.100.0/24 -j DROP
-A OUTPUT -o venet0 -s 203.0.113.0/24 -j DROP
-A OUTPUT -o venet0 -s 224.0.0.0/4 -j DROP
COMMIT
# Completed on Wed Feb 1 17:28:45 2012
# Generated by iptables-save v1.4.2 on Wed Feb 1 17:28:45 2012
*filter
:INPUT ACCEPT [0:0]
:IN_VENET ACCEPT [0:0]
:FORWARD DROP [0:0]
:OUTPUT ACCEPT [0:0]
# Always allow localnet
-A INPUT -i lo -j ACCEPT
# Move everything that isn't localnet to another chain:
-A INPUT ! -i lo -j IN_VENET
# Allow traffic on connections already in use:
-A IN_VENET -m state --state ESTABLISHED -j ACCEPT
# Log and drop traffic that conntrack can't link to existing connections:
-A IN_VENET -m conntrack --ctstate INVALID -m limit --limit 1/minute --limit-burst 3 -j LOG --log-prefix "REJ_INV "
-A IN_VENET -m conntrack --ctstate INVALID -j REJECT --reject-with icmp-admin-prohibited
# This could limit UDP traffic in the 20100-29999 port range to 10 packets per second and burst up to 30 per IP address + destination port (if you have "hashlimit"):
# -A IN_VENET -m udp -p udp -m multiport --dports 20100:29999 -m hashlimit --hashlimit 10/s --hashlimit-burst 30 --hashlimit-mode srcip,dstport --hashlimit-name SOF2 -m state --state NEW -j ACCEPT
# Use a 20100-29999 port range for your "recent" module traffic:
-A IN_VENET -p udp -m udp -m multiport --dports 20100:29999 -m state --state NEW -m recent --set --name SOF2 --rsource
-A IN_VENET -p udp -m udp -m multiport --dports 20100:29999 -m state --state NEW -m recent --update --seconds 1 --hitcount 10 --name SOF2 --rsource -j LOG --log-prefix "REJ_SOF2 "
-A IN_VENET -p udp -m udp -m multiport --dports 20100:29999 -m state --state NEW -m recent --update --seconds 1 --hitcount 10 --name SOF2 --rsource -j DROP
-A OUTPUT -o lo -j ACCEPT
COMMIT
# Completed on Wed Feb 1 17:28:45 2012
I was busy last days so sorry about the late reply.
Thanks for the suggestion but I guess your thinking that I only have an OpenVZ machine that doesn't have the required iptables modules. As I posted above, I have a fully loaded Xen HVM machine with all required modules and even on the OpenVZ machine, required modules are loaded by the provider.
So what makes you think that the modules are not loaded because I posted a list of modules output in my previous post?
About the Patch:
The current version is the only patched version of the game by the provider and it does not follow any updates, for it being a very old game. All servers run this version but use a firewall solution to defend against the attack.
When I said tcpdump sending output, I actually meant the captured output.
About my Analysis:
Why do you think the iptables would work just after the restart? The only thing in common was the Game-Server not running. Just when the system restarted, game-server was definitely turned off because I have to start that manually ( no auto-startup ). All the attacks were blocked because at that time the server was just sending output of "port unreacheable" and when iptables rules were applied, I myself saw that no output was being made even for the newer IPs. Now as soon as the Game-Server started, people attacking before the game-server started were blocked but anyone attacking a running game-server port now was not being blocked but instead game-server was sending out "getstatus" replies. I tested this at least 10 times before coming to the conclusion so obviously this has something to do with it otherwise what else do you think caused it? because you also didn't explain this happening.
About Rate-Limit and the Script:
The script does block the attacker but has the functionality to unblock it as well. Rate-limit also does actually quite the same thing because when you go ahead of your limit, you are COMPLETELY BLOCKED so the only difference that makes is that when you fall below the limit, you are automatically unblocked. Although I understand your script or alternative to that is parallel but with the current script running, I am facing no issues and the clients are working good as well but your way being better so I would definitely try that if a direct iptables solution is not reached.
Your iptables Save File
I don't have the "hashlimit" module yet so I will be getting that loaded first onto the machine and then try your script considering that it will allow the traffic to pass through the iptables instead of allowing the attacker to bypass directly to the game-server port.
I am running a Custom Kernel so that I would have to repatch it and generate the config with the modules or is there any method to do so without it?
LinuxQuestions.org is looking for people interested in writing
Editorials, Articles, Reviews, and more. If you'd like to contribute
content, let us know.
