Be certain I do not.

And I do realize that you're trying to finish the job. I've been following the thread and thinking about things. What I'm missing here essentially is any "evidence" of an attack. Sure there's HL/CS exploits (mostly old) and even if there is no indication of everything having been fixed all we've got so far is hearsay type of information. Nothing tangible. Even though I'm hesitant about it due to expected size I think we should really start at the bottom, logging traffic and, basically what Noway2 hinted at in his first reply, combing over packet captures. If we can get a fix on things we could create one or more Snort rules and have Guardian or fail2ban tarpit the offenders. (While I don't like to get ahead of things, we shouldn't rule out the possibility that it may well be that raw data won't yield the evidence we seek, or something that can be dealt with effectively on the network layer, and what vulnerability exploits address needs to be dealt with at the Half-Life protocol level or inside cs2d_dedicated.)

It is an attack, for sure, this is a known thing among the community of server owners and the game developer has confirmed it. But I guess what you're referring to is evidence of what exactly is happening.

I'm firmly convinced that the socket is terminating, but I don't have any idea exactly what is causing it. And I could be wrong about the socket, of course, I would just be really surprised.

Is there a command that will let me check whether the socket has been terminated, and reopen it if necessary?

Regarding checking out the packets, I had Wireshark installed, but unfortunately the logs got massive within about 5 minutes (just due to regular game traffic) and the server started freezing up. After that there was trouble for about 30 minutes, and the datacenter supposedly even had to take the server off the rack and switch out the fans. WHOOPS!

I definitely want to get these packets logged during one of the attacks. How can we do that without a repeat of what happened with Wireshark?

Yeah, that's exactly why I said I was hesitant about this approach. The problem is we don't know what to look for. To tune down to "interesting" traffic as fast as possible we could initially define traffic as UDP and "-m state --state NEW". And I guess there's no other way than to run tcpdump with a BPF filter. We might need to make the firewall log too (like said earlier your firewall needs simplifying) and possibly a scoped strace ("-e") for correlation purposes too. If you happen to have another server we could use that as log host and send pcaps and logs there. If you don't then we need to think about how we're going to reduce it another way.

I do have another server, which I'm not currently using for anything. :)

If you can explain how to do the things you suggested in more detail, I will definitely try.

Hey, is it possible to move this thread to a private forum? It wouldn't be helpful if the attacker found this conversation. :eek:

Welp, the BPF below was working fine on one pcap I tested it with, and now I tested it on another and it doesn't seem to be as accurate as I thought. If anyone sees a problem with it please let me know.

So I take it the firewall rule for large UDP datagrams didn't help? If so it might not be the large UDP packets.

Wireshark is really bloated, so I'm not surprised it crashed the server. It was probably eating up your memory. If you want to test your theory it's large UDP traffic, you should be able to record just fragmented UDP traffic with the following tcpdump command... IP packets are 1500 bytes or less so to be over 2000 they are probably going to be fragmented.

ip[6] & 0x20 != 0 will match if the More Fragment bit is not 0.
ip[6:2] & 0x1fff != 0 will match if the Fragment Offset field is not 0.

I don't know how much space you have, or how much traffic you'll record. To prevent from filling up your partition you can use the -C flag to specify the max file size, AND the -W flag to set a limit to how many files you want to keep.

-C 25M will keep creating 25MB files, where -W 20 will make it so after 20 files have been created, it will start to overwrite the first. By adding those two options you'll create 20 25MB files. You can change adjust the numbers to however you want.

You can also add "dst host <your ip> &&" to the beginning of the BPF to make sure you only capture traffic being sent to your IP.

OK, sorry about that, you clearly had a whole load more traffic than I was envisaging. The trouble is Wireshark, by default, tends to grab everything and then allows you to look at the data you have grabbed in different ways. The data display is easier for a newbie to get their heads around than some other apps, but that isn't a luxury that we have here.

On the iptables front, I would point you at a few links, and one warning:

• First, the warning: this is an immediate problem, and the last thing that I want to do is to send you off for two weeks reading iptables documentation, with the result that you don't do anything about the immediate threat for that period of time. I have more in mind that you should have these things available, so that you have a quick reference, in case things do go wrong.
• Extensive documentation on iptables is available here (don't bother with the 'online' version, just grab, eg, a .pdf so that you have it available and can look at it at your leisure); when I say extensive, I mean it. Pretty complete, but using it to look up a syntax is more useful than starting to read at the beginning and finishing at the end, at this point. (Actually, the iptables man page is better than some, too, so that is also an option.)
• For a worked, explained, example of a firewall, I find that Linuxhomenetworking is the best (YMMV, etc...mind you, if I remember correctly, his examples are all for Red Hat/Fedora and so should be relevant for other distros of that family (eg, Centos)), but there is also Yolinux, which is less extensive, but also easier to get through.

(Passing note: the Linuxhomenetworking site is heavily based on the book by Harrison "The Linux Quick Fix Notebook" pub Prentice-Hall; that book, and some others in the series (the Bruce Perens open source series) have been made available by P-H as zero cost downloadable .pdfs; you could search for that, for example, but if you are interested, and you don't find anything, I can research that further for you. It may be that there is now a revised edition available as dead-tree-ware, I'm not sure, but i do get the impression that it is the older books in the series that have had this treatment.)

Thanks for your help guys!

What I'd really like to do at the moment is log all *incoming* udp traffic but with the logs being saved on a separate vps in manageable chunks, say 25mb each. Instructions on how to do this would be very helpful. Edit: I also need to log the IP addresses which the udp packets are coming from. Does Wireshark or tcpdump do that?

Right now I've got everything logging via tcpdump, and it seems to be working fine. After the next attack I'll let you guys know.

Unfortunately, no.

Doing some research I see Counter Strike, like similar Quake-engine-like games, uses Valves Source engine Half-Life protocol [0|1] except it's not HLv1 but the HLv2 protocol so any dissectors ain't gong to work [2|3]. There's been HL / derivative vulns since 2000 (IIGC) 4|5|6] and exploits too (no links obviously). If we assert server-side, version-specific ones have been addressed by the developers then running the most recent version of CS2D (ensure you do) alleviate those. However to compound things there seems to be 0) different versions of Counter Strike out (dunno, haven't kept up, I dropped out after playing QWCL and UT1 basically), there seem to be problems with 1) enabling certain server-side cvars, game 2) mods, issues with 3) fake players, player names with 4) strange characters, 5) authorized players vs unauthorized users, enabling 6) remote admin console (RCON), (apparently large UDP packets?,) and 7) small UDP packets that seem to "freeze" the server application. And then I'm not talking about Steam itself [7].

- Items 1+6 mean you shouldn't allow any player to enter RCON commands. RCON should be disabled and if you need it access to the port should be confined to your management IP range and or trusted (really) admin-capable players. Maybe bind the port to localhost and only allow authenticated users to access it over SSH? And FWIW there seem to be mods out to lock the rcon users password.
- I don't know if item 3 is fixed but IIGC item 4 is.
- Item 5 means that if you have authorized players doing Bad Things you're in a whole different world of pain. (If you compare this to say SIP then there's a lot the firewall and SIP applications can do to protect the server from unauthorized users and restrict access for authorized ones, but once a user is in there isn't much you can defend yourself with from a user doing Bad Things except watch counters).
- As for item 7 this seems DoS-related. While there are fixes out ("DoS Attack Fixer") that claim to mitigate some of those (like "AntiCSDoS") contain only a (Windows) binary which you can't run on Linux and which I wouldn't run anyway as there's no clue as to what it contains or does. Some admins swear dropping short packets works
(where CS2D_SERVER_PORT obviously is the port cs2d_dedicated runs on and LENGTH seems to be 28 or 46) and some say it doesn't. The problem is no mailing list message, forum thread or web page shows any network data so in essence there's nothing to go on analysis-wise.


Down to the practical side of things. Given there's no network data whatsoever around for analysis and given your previous bad experiences with running tcpdump (though you don't show what command you actually did run) and OlRoy's latest post I think it would be wise to restrict pcap to only the necessary ports. IIGC the following ports may be in use:
UDP/1200 #friends service (whatever that may be)
UDP/27000-27015 # HL1
TCP/27020-27050 # HL2
UDP/27015 # HLDS, SRCDS and HLTV
UDP/27020 # HLDS, SRCDS and HLTV
TCP/27015 # RCON
UDP/28000 # unknown
UDP/29000 # unknown
Unless you check which server-side ports are in use on your server (which you should) I suggest using "port 1200 and portrange 27000-27050 and port 28000 and port 29000" as BPF filter.


Excellent. As for actually running tcpdump logging to a remote server there's more than a few ways. Even though this isn't critical data I suggest not running the capture over Netcat but use encryption available. Let's assert for these examples the remote log server has IP address 10.1.1.2 and ports TCP/22 (SSH) and TCP/20000 are not blocked in the firewall.
* When I issue long-running commands on a server I usually do that inside a screen ('man screen') session as this ensures I can reconnect to the session if my SSH connection breaks. Obviously I don't log in as root over SSH (and neither should you) but a unprivileged user after which I use sudo or su to root when additional auditing is enabled.


I. Using OpenSSH
On the CS2D server become root and issue:
* When using SSH this way you should use ssh-agent to load keys or set up an IP-restricted passwordless account.
** Ensure "/path/to/" is an existing path with ample space available, "udp.pcap" a non-existing file and "remote_user" may write the file there.
*** In this command tcpdump can not segment captures. Maybe adding "-W 1 -C 100" may limit it to sending 100MB once across the wire after which you restart the job. Alternatively you could log to local file as Olroy showed but limit the amount of files saved and send files across the wire separately once tcpdump starts writing a new one (inotify).


II. Using Socat
* Same comments as above apply here.
** For some reason on the server socat starts to spit out nfo to stdout. Dunno why. Maybe "${socat_cmd} >/dev/null".


III. Using fuse-sshfs
This requires FUSE to be installed and the fuse-sshfs package. On the CS2D server become root and issue:
* Same comments as above apply here except here tcpdump should be able to split captures.

HTH

@UnSpawn, I think you meant -W 10 -C 100 to create 10 100MB... -C is the file size, -W is the number of files to create. I found HLShield is supposed to help prevent attacks on Linux game servers, but as you mentioned I don't know what it contains. It's also an experimental version.

@smallgamer, you said the actual game server logs didn't show anything with this current attack, but do you know if there is a debug mode you can use for to get more verbose logging?

Thanks, changed it.

Thanks for the additional info. Although I think a lot of it is helpful either way, CS2D is not a version of Counter-Strike of Half-Life per se. It's a top-down 2D remake of Counter-Strike developed by Unreal Software. How much it shares with Counter-Strike in terms of exploits, I'm not exactly sure. We do have rcon though, so there appear to be some similarities.

That depends if it's only the connection to the Master List server (as it seems to be disappearing from the list), user-initiated connections or server replies. If its the Master List server then an iptables -j LOG rule for the remote servers address should show it doesn't download stats and update its own status anymore. Otherwise running stace on the server might show some oddness but you probably have to tweak the strace set, something like '-etrace=\!gettimeofday,sleep,ioctl', to cut out crud. BTW your latest reply is a bit short and doesn't address what Olroy asked earlier on about debug mode and if you really want to say "Thanks for the additional info" then you best do that by finding out and posting back if any of the items mentioned above are an issue with CS2D or not. While you're at it also report back what ports your CS2D uses and which tcpdump command you actually run: be complete and be verbose.

I'm sorry for not replying to it all. I'm trying to wrap my head around what for me is a boatload of new information, while maintaining focus on stopping this attack.

I don't think there is a debug mode, as the developer mentioned trying to put out a debug release soon to figure this out. I really don't know whether any of the items you mentioned are an issue with this game or not. I can certainly ask the developer.

As for this attack, the tcpdump worked and I've got a log file now of the latest attack. It is clearly visible when the attacker starts flooding the server with packets, and his IP address is from the same ISP as an attacker I had earlier this year when he was using a different kind of attack. I don't think I can post the pcap file here due to the fact that it contains a lot of players' IP addresses and I don't want that on a public forum. However I will upload it and PM it to you.

EDIT: Pcap file has been sent via the "send an email message" button, unSpawn.