Game server under attack
 

I run a gaming server on Centos. It's the most popular server in my game, and some people who don't like that have been attacking it daily. It's being done by spoofing or spamming some kind of packet to the server.

The game server process continues to run on the Centos server, but it no longer appears on the game list and players can not connect to it.

I have no idea where to begin figuring out how this is being done and how to stop it. Any ideas? [REMOVED]

Thanks for any guidance you can provide. I've been dealing with this for months and it's starting to drive me crazy (and ruin my server).

you should have iptables firewall up and running to allow only the ports the server needs for your 'game'
check your LOG files (/var/log/httpd etc.)
post the results

post your "iptables -L" or "cat /etc/sysconfig/iptables"

I'm curious why you wrote 'game' instead of game, lol. If you're actually interested I could PM you the server info and you can check it out. Anyway, thank you for your help so far.

I looked in the /var/log/httpd folder and there are a few files there. I'm not sure exactly what I'm looking for though. Sorry - could you be more specific please (I'm an admitted noob)?

This is what I get when I do "iptables -L" (I assume it's default since I've never touched it):

And here is "cat /etc/sysconfig/iptables" (again assuming this is default):

First off, welcome to LQ Security. The output of those two commands shows that you are not running any sort of firewall on your system. In terms of administering a public facing server, running a game or 'game' server is tantamount to putting a bulls-eye target on yourself as they are frequently targeted for all sort of ill activity, including DOS (Denial of Service) type attacks. As an admitted noob, you will face extra, but not insurmountable, challenges in dealing with it.

The first order of business will be to determine the type of attack you are being exposed to. You mentioned that some form of packet is being spammed or spoofed to your server. Could you please be more specific? How did you determine you were being attacked, what do these packets look like, what effect are they having, are they coming from one particular source or multiple sources, etc. Please be verbose in your description and provide as much detailed information as you possibly can. LQ Security has some extremely knowledgeable people and we can certainly help you with this problem, but we prefer and will need to deal with facts. To that end, as much specific information as you can provide will be helpful.

I'll try to answer all of your questions.

I know I'm under attack because these same people have been attacking my server for about a year now. Originally it was a chat flood, where they used a little script to spam in-game chat messages fast enough to make the game server crash (not the centos server, but the "game server" running on it... not sure if there is better terminology for that distinction). This is an independently developed game, so I have access to the developer and was able to get him to patch that.

Then they proceeded to start "spoofing" other players' in-game logins, and since those credentials are used to assign administrative rights to certain players, spoofing those players allowed them to do all sorts of things like mass banning players, changing server settings to the point of crippling the game and making it unplayable, etc. The developer was able to patch this also.

As far as the current attack, I'm less clear on exactly how they're doing it. The effect is that my server is unresponsive to the main-gameserver-tracking-server-the-developer-owns, so it disappears from the in-game server list, and it also becomes unresponsive to players who try to play on it or join it. However, the process continues running on centos. I do know that it's this same group with another attack, based on forum chatter and "the grapevine", and I say that they are doing it by sending packets for a few reasons. 1- I'm hearing that it's done via a winsock application. 2- I'm also hearing that these same people have been DOSing other servers (not sure whether that conflicts with #1). 3- The game logs (not the centos logs, which I'm pretty unfamiliar with, but the mediocre game server logs that document the in-game action and minor game server stuff) don't really show anything, unlike the previous chat spam attacks that were visible in the logs.

I don't know what the packets look like per se. I also don't know whether they're coming from one source or multiple sources.

Since I don't use this server for anything but gaming, what I'd really like to do is completely lock down everything else on the server... so that it basically ignores any packets that aren't sent to this specific application, but also logs any packets that are spammed at this application in case that's what is happening. I also want to make sure nobody can connect to the server in any way except for the game... meaning email services, http stuff, cron, anything that would allow my server to be controlled remotely (again, not that familiar with centos yet), and any possibility of other users or accounts existing on the system.

Of course, I don't know what I'm talking about in the slightest, but a complete lockdown sounds really good right now, and like I said, I'm not using any of that stuff.

EDIT: I should have also mentioned that I'm running centos on a VPS, since that might matter.

If you have a gui installed (something like a windows desktop) you should be able to use the system-config-firewall to get you started with locking down the system. Just remember to make sure you don't block the remote desktop port you're using! (Made that mistake a number of times myself.) If it's not installed just do a to download and install it. It's easy enough to use that as long as you know the port(s)/protocol(s) you need you should be good to go.

I don't.

Anyway, I've received some more information indicating that this is being done with a UDP dos attack, presumably on the port from which the game is running on my server. Does anyone know how I can log and stop this type of attack?

First things first. Until we have a clear picture of what you need please stop and disable any publicly accessible services you don't need for server maintenance like web-based server management panels, FTP, VNC, SMTP, etc. You need SSH but ensure you use an unprivileged account with pubkey auth and not log in as root (more here: http://www.linuxquestions.org/questi...tempts-340366/). As you don't run a default firewall (the rules listing in chapter 2 http://wiki.centos.org/HowTos/Network/IPTables is) we need to set one up first and as quickly as possible. To help you set up /etc/sysconfig/iptables please post a list of ports in use by services and obfuscate your IP address with something like "1.2.3.4". Also list enabled services and as much technical details about the game or games you're running. The more nfo the better and the quicker your reply the better.

If you have that kind of access to the developer, you really ought to be able to find out what ports and protocols should normally be in use for the game.

I like wireshark for getting a screen dump of all traffic, but how useful this is will be a function of whether you can capture data when there isn't much 'good' traffic and a lot of 'bad' traffic.

It will be very helpful to know what the attack packets actually look like, because, for some cases there will be simple iptables blocks, and for others it would be more involved. Obviously. you should be trying to progress to a more secure situation, and a firewall will be an important part of that whatever the current attack looks like, but, if you are under attack currently, that must have a degree of priority.

You seem to be referring to attempts to crack passwords, by something like a brute-force or a dictionary attack. This is not the normal use of the word spoofing in connection with networking, so be careful with this. (And, if it is an attack from one, or a small number, of ip addresses, iptables can help here, too. Particularly important to look at the actual data, and see what it looks like, then.)

iptables also has some logging capabilities, so it could help there too. The trouble is defining what is a 'good' (ie, normal game packet) and what is a 'bad' packet. Again, looking at the actual data, it could jump off the page. Or, not. You won't know until you've actually looked at some data.

Too much there that is hearsay, and which doesn't give actual details of the packets.

All that 1) means, if true, is that they using Windows. Not that helpful as a way of countering the threat.
2) means that there is a rumour that the same people have been using a denial of service attack of some sort. Even if it can be relied upon, that doesn't tell you very much (can you rely on the idea that it isn't a DDoS...I doubt it, unless you know more about the technical capability of the person who gave you the report, and whether it is a DoS or a DDoS does make a difference in how easy it is to take short-term measures against.)
3) isn't all that helpful, either, unless the implication is that game developer can again enhance the logs to help you.

netstat -ntulpe

chkconfig --list | grep "$(runlevel|awk '{print $2}'):on";

The game is *edited to avoid attackers finding this in Google*. I'm not too aware of all of the technical details though. If there's anything specific you need to know, I can talk to the developer and get that information for you.

I am certain that you're right. But I wouldn't know how to go about locking down everything else that should not be in use. Also, my (limited) understanding of the current attack is that it's going to the port that has to be open in order for the game to run, and that it's sending a packet that the game accepts (but flooding the server with it).

If that is correct, then I will need to be able to log when packets are flooded that way, check what IP the packets came from, and then completely block any future packets from that IP so that they don't enter the server and reach the gaming software that is running on it.

Well, they were sending the server a packet that the game client normally sends the server, which tells the server what player it's dealing with. Player logins are done in the game client, and all of the servers simply listen to the login that the game client sends them. So they spoofed that and didn't need to use any kind of password cracking. Luckily, the developer patched that pretty quickly.

I'm not really sure if that's the case or not. There are definitely a lot of packets flying around though, as this is a real-time shooting game.

I'm not entirely sure if the packet itself is any different from the packets that are normally sent to the server, or if it's the same kind of packet but just flooded to make the attack.

OK, so only TCP/22,80 and UDP/369... Lets see if we can build you an /etc/sysconfig/iptables replacement before moving on to packet inspection:
Save this as /etc/sysconfig/iptables.new then activate like this (could use 'at' instead) since all your IPTABLES_SAVE.*= variables are set to "no":
this allows you five minutes for testing / looking at logging after which the default script (having no rules :-( ) will be enabled. If you think this rule set works then - Note that SSH should really be confined to your management IP range or at least protected by 'fail2ban' or equivalent. The same would go for HTTP if you only use it for web-based service or server management. Also note the above rule set explicitly denies SMTP traffic (even though a standard sendmail.cf should have the MTA running on only loopback).
- Do adjust limits if you find a lot of logged "IN_lim " lines in /var/log/messages.
- Also see http://wiki.centos.org/HowTos/Network/IPTables, http://www.centos.org/docs/5/html/De...-US/ch-fw.html and http://www.frozentux.net/iptables-tu...-tutorial.html for a basic understanding of Netfilter / iptables.
- While you're at it run 'yum install logwatch' and run your logs through it. Not related to you getting DoSsed but it would be good to see if there's other things to address.

When I try to enter:

/sbin/iptables-restore < /etc/sysconfig/iptables.new

I get this message:

I'm no longer getting that error now that I input the iptables text you gave me directly in centos using the vi tool, instead of saving it in a text editor. Now I am getting an error that -i can't be used with output. When I remove -i, I get another error saying "lo" is a bad argument.

Oops.

0) It's missing a "# Generated by iptables-save v1.3.5 on Sat Oct 15 00:00:01 2011" line before the "*filter" line,
1) Should be "-A OUTPUT -o lo -j ACCEPT".

If you change the loopback line then you should be able to execute the script like before. If that doesn't work try

I'm getting the message "line 32 failed" after applying those last two changes. Does that indicate some problem with this iptables text, or is it fine and I should just use that other method of activating the iptables file from your last post?

The only thing missing are "-i eth0" items in lines below the ESTABLISHED,RELATED rule. Try adding it similar to that line.

Still "iptables-restore: line 32 failed."

And this is the FIRST step? I'm grateful that you're helping me, but YIKES.

OK, this is taking too long. Save as appropriate and run it like this standalone shell script:

It froze on the fifth line and I can no longer log into my server via ssh.

Not back up yet? No access to port 80 (meaning it's inaccessible completely)?

How do I access port 80? Through my ssh client, just change port 22 to 80?

I need to stress that you're dealing with a noob here.

EDIT: That's not working.

OK, that plainly sucks. Just to be sure: the iptables rule set is really simple and unless your VPS doesn't come with the most basic of modules ('grep -i iptables /etc/vz/vz.conf'), or used an ethernet device other than eth0, I can't see why it wouldn't have worked. Anyway, since it remains inaccessible the only alternative is to ask your provider to either log in and fix things for you if they can or gracefully shut down and reboot it. After that you'll have to find out if iptables is enabled by your provider and if so which modules are offered.



No, use a browser for port 80.


Thanks, I sometimes tend to forget that.

Okay, apf firewall is set up and configured. I did have help with this (I'm sorry for not following your advice via email, but I'm also trying to maintain my job and it was either pay someone to set that up for me or shut down my gaming server.) My current iptables contents are below.

Unfortunately, the attack I described previously is still happening multiple times per day.

I have some more information about what is going on when the attack happens. The targeted application actually continues to run... it even continues producing log files. But it is dead to the outside world. Apparently this attack causes the application's socket to terminate.

I read that this is an exploit for attacking some other games as well. An oversized udp packet is sent, and it causes the application's socket to terminate. That seems to be what is happening here. So I had a rule added to the firewall:

-A INPUT -p udp -m length --length 2001:65535 -j DROP

(2000 is the number the game developer recommended, saying anything over that could be dropped.)

This rule is not stopping the attack. But wouldn't an oversized packet mean one which is larger than 65535 anyway? However, I can't enter any number higher than that, or I get an error when I restart iptables. So, can this be reversed and changed to ACCEPT with a criteria of 0-2000, so that ANY packet over 2000 (no matter HOW large) is automatically not accepted? If so, what is the syntax for that, and if not, do I have any other options?

I also noticed when I first looked at this configuration that there is no LIMIT rule for the rate of udp packets. What should I put for that kind of rule, so that if more than a certain number of udp packets come per second from the same IP address, they are automatically ignored and the DOS attack fails while the IP address is added to the deny hosts list?

I'm not an iptables guy by any means. However, shouldn't the "-A INPUT -p udp -m length --length 2001:65535 -j DROP" go before the below rules that permit traffic to the port your game service listens on?

Thanks, I changed that. Hopefully it helps.

It's really hard to find qualified people to work on this stuff. The same person who did that also had logging set to 0 so that it was not logging dropped packets, even though I explicitly said I needed that. At least I'm starting to understand things in the process, but still, that's pretty annoying.

No problem, I hope it works out for you. But as I said, I'm by no means an iptables guy. My post was a question. I'm a bit rusty since I've been focusing on other areas of security so you may want to wait for verification or double check yourself.

In general, there are two major philosophies with IP tables that are helpful to keep in mind when working with the rules. One, is that the rules are process in order from top to bottom and if a match is found, it stops comparing. As Ol'Roy pointed out, placing a drop rule based upon packet size beneath one that accepts traffic to that port won't work, because the length check won't even get analyzed. The second philosophy is to set your firewall so that traffic is dropped by default and only accepted when it meets certain parameters. The syntax for the rules is almost identical to your drop rules, but instead of -j DROP, use -j ACCEPT. This practice is commonly referred to as white-listing versus black-listing and it gives you a real leg up on the would be attacker, who can adapt to get around your reject lists. There are two ways to achieve this, one is to set the policy to DROP and then add the rules that you want to accept, the other is to set the policy to ACCEPT, add the rules you want to ACCEPT and then at the end put a catch all DROP rule. The primary difference is in how the filter will behave when the rules have been flushed, in which case it will act according to the policy. If you don't have physical access to the machine, setting the policy to accept is safer in that if the rules get cleared, you can still get at the machine via SSH.

So as an example of how this would impact your rules, instead of having 20+ rules to drop specified ports, you could have three or four rules to only accept the ports, or port ranges you wish to accept. You can also get creative to enhance your security on your SSH port or other management interface. Say for example, that you have either a static IP or you can identify the network+mask that identifies the range that YOU will be using, you can restrict access to only these source IP addresses, effectively cutting off traffic from everyone else. I also noticed that you have duplicate rules for TCP and UDP on the same port, you should be able to simplify your rules by just specifying the port without the protocol. For example, your three rules:
The 3rd rule is really superfluous because traffic will be dropped by one of the two rules, but you could also eliminate the first two and just use the 3rd.

There are some really great tutorials on IPTables, that might help you with these rules. It looks like you have or are getting a grasp of the basic syntax and are starting to get into the art of rule writing. Once you get past the initial learning curve, writing iptables rules is actually enjoyable.

To address your other question about rate limiting: a quantitative suggestion is hard to provide. You might want to ask the developer for some guidance. Otherwise you will need to do some trial and error and expect to fine tune it for a while as a balance can be difficult to achieve.

You might find this link interesting: http://blog.bodhizazen.net/linux/pre...with-iptables/ It is an IPTables tutorial specifically geared towards preventing attacks and it discusses both length and limit functions. While somewhat cursory, it might at least be a good common reference point for refining the discussion.

Thanks for the help, Noway2. I'm looking through the information you linked to and trying to wrap my head around it.

I think unSpawn may have written me off due to the fact that I paid someone to do the initial configuration on my server.

UnSpawn, if that is the case, please understand where I'm coming from.

I realize that it's not apparent here, but I do have a love for knowledge, and I spend a lot of time each week learning how to do new things. I make my living on the internet, and I'm no stranger to spending days figuring out why X, Y, or Z isn't working.

Paying for someone to do the initial setup on my firewall, however, was logistically unavoidable. I did not have days to spend going through the entire process as a newb, and I also did not want to give up and shut down my gaming server. That only left me with one option. Please try not to think poorly of me. I'm still here trying to finish the job.

I don't think anyone, unSpawn included, will look down upon you for paying to have someone perform a service for you. For that matter, we all have day jobs too. Sometimes expediency is the biggest priority and hired hands can bring that to the table. I do think that most here appreciate making an effort to learn, which you are doing, and I don't think that we could ask any more of you. Sometimes, especially for those who have been using Linux for a long time, we lose sight of the new user perspective. This causes us to forget to include things that have become obvious to us. Please have patience with us in this regard.

Once you have had a chance to read up on iptables and look over some of the tutorial pages, please ask questions and don't be afraid to ask.

Be certain I do not.

And I do realize that you're trying to finish the job. I've been following the thread and thinking about things. What I'm missing here essentially is any "evidence" of an attack. Sure there's HL/CS exploits (mostly old) and even if there is no indication of everything having been fixed all we've got so far is hearsay type of information. Nothing tangible. Even though I'm hesitant about it due to expected size I think we should really start at the bottom, logging traffic and, basically what Noway2 hinted at in his first reply, combing over packet captures. If we can get a fix on things we could create one or more Snort rules and have Guardian or fail2ban tarpit the offenders. (While I don't like to get ahead of things, we shouldn't rule out the possibility that it may well be that raw data won't yield the evidence we seek, or something that can be dealt with effectively on the network layer, and what vulnerability exploits address needs to be dealt with at the Half-Life protocol level or inside cs2d_dedicated.)

It is an attack, for sure, this is a known thing among the community of server owners and the game developer has confirmed it. But I guess what you're referring to is evidence of what exactly is happening.

I'm firmly convinced that the socket is terminating, but I don't have any idea exactly what is causing it. And I could be wrong about the socket, of course, I would just be really surprised.

Is there a command that will let me check whether the socket has been terminated, and reopen it if necessary?

Regarding checking out the packets, I had Wireshark installed, but unfortunately the logs got massive within about 5 minutes (just due to regular game traffic) and the server started freezing up. After that there was trouble for about 30 minutes, and the datacenter supposedly even had to take the server off the rack and switch out the fans. WHOOPS!

I definitely want to get these packets logged during one of the attacks. How can we do that without a repeat of what happened with Wireshark?

Yeah, that's exactly why I said I was hesitant about this approach. The problem is we don't know what to look for. To tune down to "interesting" traffic as fast as possible we could initially define traffic as UDP and "-m state --state NEW". And I guess there's no other way than to run tcpdump with a BPF filter. We might need to make the firewall log too (like said earlier your firewall needs simplifying) and possibly a scoped strace ("-e") for correlation purposes too. If you happen to have another server we could use that as log host and send pcaps and logs there. If you don't then we need to think about how we're going to reduce it another way.

I do have another server, which I'm not currently using for anything. :)

If you can explain how to do the things you suggested in more detail, I will definitely try.

Hey, is it possible to move this thread to a private forum? It wouldn't be helpful if the attacker found this conversation. :eek:

Welp, the BPF below was working fine on one pcap I tested it with, and now I tested it on another and it doesn't seem to be as accurate as I thought. If anyone sees a problem with it please let me know.

So I take it the firewall rule for large UDP datagrams didn't help? If so it might not be the large UDP packets.

Wireshark is really bloated, so I'm not surprised it crashed the server. It was probably eating up your memory. If you want to test your theory it's large UDP traffic, you should be able to record just fragmented UDP traffic with the following tcpdump command... IP packets are 1500 bytes or less so to be over 2000 they are probably going to be fragmented.

ip[6] & 0x20 != 0 will match if the More Fragment bit is not 0.
ip[6:2] & 0x1fff != 0 will match if the Fragment Offset field is not 0.

I don't know how much space you have, or how much traffic you'll record. To prevent from filling up your partition you can use the -C flag to specify the max file size, AND the -W flag to set a limit to how many files you want to keep.

-C 25M will keep creating 25MB files, where -W 20 will make it so after 20 files have been created, it will start to overwrite the first. By adding those two options you'll create 20 25MB files. You can change adjust the numbers to however you want.

You can also add "dst host <your ip> &&" to the beginning of the BPF to make sure you only capture traffic being sent to your IP.

OK, sorry about that, you clearly had a whole load more traffic than I was envisaging. The trouble is Wireshark, by default, tends to grab everything and then allows you to look at the data you have grabbed in different ways. The data display is easier for a newbie to get their heads around than some other apps, but that isn't a luxury that we have here.

On the iptables front, I would point you at a few links, and one warning:

• First, the warning: this is an immediate problem, and the last thing that I want to do is to send you off for two weeks reading iptables documentation, with the result that you don't do anything about the immediate threat for that period of time. I have more in mind that you should have these things available, so that you have a quick reference, in case things do go wrong.
• Extensive documentation on iptables is available here (don't bother with the 'online' version, just grab, eg, a .pdf so that you have it available and can look at it at your leisure); when I say extensive, I mean it. Pretty complete, but using it to look up a syntax is more useful than starting to read at the beginning and finishing at the end, at this point. (Actually, the iptables man page is better than some, too, so that is also an option.)
• For a worked, explained, example of a firewall, I find that Linuxhomenetworking is the best (YMMV, etc...mind you, if I remember correctly, his examples are all for Red Hat/Fedora and so should be relevant for other distros of that family (eg, Centos)), but there is also Yolinux, which is less extensive, but also easier to get through.

(Passing note: the Linuxhomenetworking site is heavily based on the book by Harrison "The Linux Quick Fix Notebook" pub Prentice-Hall; that book, and some others in the series (the Bruce Perens open source series) have been made available by P-H as zero cost downloadable .pdfs; you could search for that, for example, but if you are interested, and you don't find anything, I can research that further for you. It may be that there is now a revised edition available as dead-tree-ware, I'm not sure, but i do get the impression that it is the older books in the series that have had this treatment.)

Thanks for your help guys!

What I'd really like to do at the moment is log all *incoming* udp traffic but with the logs being saved on a separate vps in manageable chunks, say 25mb each. Instructions on how to do this would be very helpful. Edit: I also need to log the IP addresses which the udp packets are coming from. Does Wireshark or tcpdump do that?

Right now I've got everything logging via tcpdump, and it seems to be working fine. After the next attack I'll let you guys know.

Unfortunately, no.

Doing some research I see Counter Strike, like similar Quake-engine-like games, uses Valves Source engine Half-Life protocol [0|1] except it's not HLv1 but the HLv2 protocol so any dissectors ain't gong to work [2|3]. There's been HL / derivative vulns since 2000 (IIGC) 4|5|6] and exploits too (no links obviously). If we assert server-side, version-specific ones have been addressed by the developers then running the most recent version of CS2D (ensure you do) alleviate those. However to compound things there seems to be 0) different versions of Counter Strike out (dunno, haven't kept up, I dropped out after playing QWCL and UT1 basically), there seem to be problems with 1) enabling certain server-side cvars, game 2) mods, issues with 3) fake players, player names with 4) strange characters, 5) authorized players vs unauthorized users, enabling 6) remote admin console (RCON), (apparently large UDP packets?,) and 7) small UDP packets that seem to "freeze" the server application. And then I'm not talking about Steam itself [7].

- Items 1+6 mean you shouldn't allow any player to enter RCON commands. RCON should be disabled and if you need it access to the port should be confined to your management IP range and or trusted (really) admin-capable players. Maybe bind the port to localhost and only allow authenticated users to access it over SSH? And FWIW there seem to be mods out to lock the rcon users password.
- I don't know if item 3 is fixed but IIGC item 4 is.
- Item 5 means that if you have authorized players doing Bad Things you're in a whole different world of pain. (If you compare this to say SIP then there's a lot the firewall and SIP applications can do to protect the server from unauthorized users and restrict access for authorized ones, but once a user is in there isn't much you can defend yourself with from a user doing Bad Things except watch counters).
- As for item 7 this seems DoS-related. While there are fixes out ("DoS Attack Fixer") that claim to mitigate some of those (like "AntiCSDoS") contain only a (Windows) binary which you can't run on Linux and which I wouldn't run anyway as there's no clue as to what it contains or does. Some admins swear dropping short packets works
(where CS2D_SERVER_PORT obviously is the port cs2d_dedicated runs on and LENGTH seems to be 28 or 46) and some say it doesn't. The problem is no mailing list message, forum thread or web page shows any network data so in essence there's nothing to go on analysis-wise.


Down to the practical side of things. Given there's no network data whatsoever around for analysis and given your previous bad experiences with running tcpdump (though you don't show what command you actually did run) and OlRoy's latest post I think it would be wise to restrict pcap to only the necessary ports. IIGC the following ports may be in use:
UDP/1200 #friends service (whatever that may be)
UDP/27000-27015 # HL1
TCP/27020-27050 # HL2
UDP/27015 # HLDS, SRCDS and HLTV
UDP/27020 # HLDS, SRCDS and HLTV
TCP/27015 # RCON
UDP/28000 # unknown
UDP/29000 # unknown
Unless you check which server-side ports are in use on your server (which you should) I suggest using "port 1200 and portrange 27000-27050 and port 28000 and port 29000" as BPF filter.


Excellent. As for actually running tcpdump logging to a remote server there's more than a few ways. Even though this isn't critical data I suggest not running the capture over Netcat but use encryption available. Let's assert for these examples the remote log server has IP address 10.1.1.2 and ports TCP/22 (SSH) and TCP/20000 are not blocked in the firewall.
* When I issue long-running commands on a server I usually do that inside a screen ('man screen') session as this ensures I can reconnect to the session if my SSH connection breaks. Obviously I don't log in as root over SSH (and neither should you) but a unprivileged user after which I use sudo or su to root when additional auditing is enabled.


I. Using OpenSSH
On the CS2D server become root and issue:
* When using SSH this way you should use ssh-agent to load keys or set up an IP-restricted passwordless account.
** Ensure "/path/to/" is an existing path with ample space available, "udp.pcap" a non-existing file and "remote_user" may write the file there.
*** In this command tcpdump can not segment captures. Maybe adding "-W 1 -C 100" may limit it to sending 100MB once across the wire after which you restart the job. Alternatively you could log to local file as Olroy showed but limit the amount of files saved and send files across the wire separately once tcpdump starts writing a new one (inotify).


II. Using Socat
* Same comments as above apply here.
** For some reason on the server socat starts to spit out nfo to stdout. Dunno why. Maybe "${socat_cmd} >/dev/null".


III. Using fuse-sshfs
This requires FUSE to be installed and the fuse-sshfs package. On the CS2D server become root and issue:
* Same comments as above apply here except here tcpdump should be able to split captures.

HTH

@UnSpawn, I think you meant -W 10 -C 100 to create 10 100MB... -C is the file size, -W is the number of files to create. I found HLShield is supposed to help prevent attacks on Linux game servers, but as you mentioned I don't know what it contains. It's also an experimental version.

@smallgamer, you said the actual game server logs didn't show anything with this current attack, but do you know if there is a debug mode you can use for to get more verbose logging?

Thanks, changed it.

Thanks for the additional info. Although I think a lot of it is helpful either way, CS2D is not a version of Counter-Strike of Half-Life per se. It's a top-down 2D remake of Counter-Strike developed by Unreal Software. How much it shares with Counter-Strike in terms of exploits, I'm not exactly sure. We do have rcon though, so there appear to be some similarities.

That depends if it's only the connection to the Master List server (as it seems to be disappearing from the list), user-initiated connections or server replies. If its the Master List server then an iptables -j LOG rule for the remote servers address should show it doesn't download stats and update its own status anymore. Otherwise running stace on the server might show some oddness but you probably have to tweak the strace set, something like '-etrace=\!gettimeofday,sleep,ioctl', to cut out crud. BTW your latest reply is a bit short and doesn't address what Olroy asked earlier on about debug mode and if you really want to say "Thanks for the additional info" then you best do that by finding out and posting back if any of the items mentioned above are an issue with CS2D or not. While you're at it also report back what ports your CS2D uses and which tcpdump command you actually run: be complete and be verbose.

I'm sorry for not replying to it all. I'm trying to wrap my head around what for me is a boatload of new information, while maintaining focus on stopping this attack.

I don't think there is a debug mode, as the developer mentioned trying to put out a debug release soon to figure this out. I really don't know whether any of the items you mentioned are an issue with this game or not. I can certainly ask the developer.

As for this attack, the tcpdump worked and I've got a log file now of the latest attack. It is clearly visible when the attacker starts flooding the server with packets, and his IP address is from the same ISP as an attacker I had earlier this year when he was using a different kind of attack. I don't think I can post the pcap file here due to the fact that it contains a lot of players' IP addresses and I don't want that on a public forum. However I will upload it and PM it to you.

EDIT: Pcap file has been sent via the "send an email message" button, unSpawn.

Looking forward to seeing the pcap!

Thanks, I have sent it to the email you asked me to.

So based on the PCAP this guy is just flooding the server with thousands of packets per second. I'm trying to limit the rate of packets one IP address can send, but it's not as straightforward as I had hoped.

I just want to make it so that the same IP address can't send me more than 100 packets per second. I found this:

http://www.debian-administration.org/articles/187

But I've learned that the "hitcount" can not be set higher than 20 (which is much too low) without increasing the default value of ip_pkt_list_tot, which is located in /etc/modprobe.conf. And, OF COURSE, this file does not exist and can not be modified on an OpenVZ vps, which is what I have.

Wtf? Then how is limiting the rate of packets from a particular IP address done on OpenVZ servers? We're just a sitting duck for DOS attacks?

Will --seconds take a decimal? If I can set it to 20 hits within .2 seconds, that would be the equivalent of 100 hits within 1 second, which is what I'm aiming for.

You didn't post the command you ran. And the capture encompasses a very short time frame. Less than half an hour is not exactly a rich data set on a well-used game server.


Next time please contact me first to make drop-off arrangements. I don't care much for file sharing sites.


On with the pcap:

In the stats above "client.15" (last octet) has a significant larger packet count compared to the two following players. Generating statistics for the first 3 players (and unless you managed to start tcpdump at the end of an attack the first line definitely is a capture error) with we get :

So discarding the first line it doesn't seem like client.15 exceeds any thresholds. Looking at data using Snort doesn't trigger anything. Wireshark doesn't show any interesting payload, except using a filter of or in tcpdump does show a sh*tload of payload-less packets. I wouldn't call it flooding, more like a set of bursts, as other players manage to get packets in as well. Now I haven't got any in-game experience wrt lag and players response times don't seem optimal anyway (we're talking the Unreliable Data Protocol) so I can't see how this would influence the game. And again, twenty minutes ain't exactly the amount of data to draw any conclusion from.


modprobe.conf sets module loading parameters. If you can't load modules yourself then you can't change ip_pkt_list_tot yourself. Ask your provider. And please don't b*tch about OpenVZ here: nobody here forced you to rent one.


Should you still want to block these packets, and given your past experiences with iptables modules, I suggest an alternative and that's using fail2ban plus Snort. You'll have to do the legwork yourself (install fail2ban and Snort, add your UDP ports to /etc/services, configure ports in /etc/fail2ban/jail.conf and an appropriate action in /etc/fail2ban/action.d/ for a rule searching for "CS_pkt_sz_lt_1", test the rule, set server IP address in snort_cs.conf) but as promised here's a Snort rule. You don't need any snort rule sets except this config and run snort with the (suboptimal: "-A fast") "-c /path/to/snort_cs.conf -A fast -q -K none -l /var/log" args:[EDIT]
I. Alternatively you can also use a third party app like Guardian instead of fail2ban.
II. If you choose to block using Netfilter you could use a rule something like
though you've got to find out if --length needs the IP packet size (28) or the UDP size (8).
[/EDIT]

That's it for now.

It appeared to be long enough to see the attack and plenty of standard game packets surrounding it. Perhaps you would like to tell me the minimum time you need, information I was not given before.

I already complied with your request and sent it to your email, so I'm not sure what your point is here.

I apologize profusely - I didn't realize my logical questions regarding OpenVZ's apparent dead end in this area would be so offensive to you.

I didn't come here to be berated, and while I appreciate your help, I would also appreciate it if you would show me the same common courtesy that I show you. I assume that we're both adults here and that's not too much to ask.

I'm not sure I understand. My attacker, Client.15, sends 512 packets per second, over ten times as many as the others. Is that not the threshold in question?

Do you mean blocking these packets from other players? No, I'm not concerned about that. I want to block the udp packets that exceed 100/second from any one IP address, i.e., the ones sent by Client.15. Do those instructions for snort and fail2ban refer to blocking the packets from other players, or the DOS attack from Client.15?