I have a CentOS 5.3 server hosting our companies website. The connection to the site is through a Cisco RVS4000 security router. The website is the only device on the network that uses this router and the router only has one inbound rule which forwards port 80 to the website.

I setup the Linux environment following instruction I found here for CentOS 5.3 over a year ago. Lately, we have been getting hit with various DoS attacks. Being a Linux novice, I'm looking for some help in preventing these DDoS attacks.

When the most recent attack occurred, I actually see traffic logged coming from the internal IP of my website. For example, the IPS report shows DDOS_TYPE_UDP_FLOOD with the source IP of 192.168.0.50 which is the internal IP of website. How in the world is my website generating the traffic? The log about 30 minutes prior to the flood shows "Possible DoS HGOD SynKiller Flooding" with a source IP of 69.162.102.55.

Again, I'm very much a novice to Linux so bear with me if I haven't provided enough information to aid in solving this issue.

Vondie

Firstly, I know nothing of that particular Cisco device, so bear that in mind in considering my answer. When you say it is a 'security router', I assume that it is a router with Cisco's firewall functionality built-in.

You mention DoS and DDoS; do you mean that you are getting both, or was one of those a typo? I have doubts about this, anyway....

For a DoS (whether DDos or just DoS) attack, the identifying characteristic would be a lot of traffic coming in to your server from the outside. You don't seem to have this, so the question is at least open whether this is a DoS-type attack or whether someone has control of your server, in some way, and is using it to send out traffic. This is what you would expect if, for example, your server had been made part of a botnet.


...with the source IP of 192.168.0.50....
So, it is coming from your server. If that's a DoS, you are committing the attack, and its not an attack on you.

.... How in the world is my website generating the traffic? ...
Someone has installed a program that generates the traffic.

...The log about 30 minutes prior to the flood shows "Possible DoS HGOD SynKiller Flooding" with a source IP of 69.162.102.55.....

If your server has been co-opted into a botnet, then this packet may be the instruction as to what your server should do.

Note that one slight mystery is that, as far as I know, the HGOD piece of malware is for Windows: while this means that it should not run on Linux, if someone has installed something like Wine, or a virtual machine environment, you could probably make linux run it. Another possibility is that what you have is just similar to HGOD and it has been mis-identified.

Some general comments:

it seems that the configuration of this server has allowed some unpleasant person to get some level of control. You really, really want to find out what that mistake has been, because you don't want to make the same mistake again, when the server is reconfigured.

It seems (note again my total lack of knowledge on Cisco) that you may be able to get some temporary relief from this problem by using firewall rules (on the Cisco, or on the server...or both) to drop any packets from 69.162.102.55. This is not a solution, but may give you a bit more time to work on the problem.

If you have taken a sensible approach to website development, you will have a development machine as well as this, your production server. This may be a time when you could swap the two to advantage. That is, transfer your dev machine to host the production site and quarantine your current production machine until your have investigated the issue and put measures in place.

You need to ensure that you have put best practices in place. What I would not advise though is to assume that just putting something in place for, eg, passwords will necessarily fix the problem; if the problem is really with the ssh configuration (or some other service), then insisting on slightly stronger passwords may not do it for you. (Or, maybe it will, if the problem was that someone was getting in via ssh because the passwords were weak.)

This suggests that you put some effort into forensics, in that if you find out how someone got in, you can be sure to direct effort to that particular problem. Otherwise you just 'scattergun', and that may not hit the particular target, although you should not ignore the more general 'best practice' side, too.

Having said that, you should read some 'best practice' type guides, to put you into a better position in the very short term. I would suggest as a very minimum that you ought to be able to tick the following boxes, right now:

• strong passwords
• no root log-in
• ssh security meaures (see, eg, the samhain doc, referenced in the URL later)
• considered firewall approach (whatever that means, in context, after maybe you have modified things, see next item)

You did, in an earlier post, ask about dual NICs and security. In the normal case, you would put the server in the DMZ/'Orange' interface, of a firewall. This would not require dual NICs on the server. It wasn't mentioned there, I take it that this is not the architecture that you have considered?

You need to have a look at this; but do note the comment
You need to think about what you need now and initially concentrate on references which focus on what you need now. You can, and should, go back later and read more.

Be aware that there is not a simple, straightforward "Do this, and have security forever" position. It is a process, a journey, with new threats coming up and someone has to give it (some) continuous attention to evolving threats and more general housekeeping.

I get the impression, maybe incorrectly, that the organisation for which you work has decided to do this website thing 'on the cheap'. Perhaps the one thing that you could do out of this is to point out that the website doesn't run itself and that it does need some ongoing attention (and the budget for that) to security, backups, etc, etc. I'm also guessing, probably unfairly, that at the time that this started you had no idea whether the server box was patched and up-to-date and that the same may well apply to the Cisco.

2 members found this post helpful.

Thank you salasi for the very thoughtful and insightful response.

I do realize that there is no simple answer to this situation nor am I able to articulate all the needed information to answer all of your points. Even the simple question of DoS and or DDoS, all I have is the log information from the router which lumps them together. I do know it was a UDP packet, so I did restrict the port 80 to only accept TCP.

I would tend to agree that it was some code that was dropped into the webserver that was generating the traffic, but I have not idea how that could have happened given only port 80 is open to the website. I do have a contact us page with php running, but I tried to follow the best practices there with striping control characters and such.

To answer the dual NIC question, I opted for the recommended approach and only have a single NIC card installed. To make things a bit more complete, the machine is running under a Microsoft Hyper-V VM.

I will begin to peck away at your outline of steps.

I have made an effort to use strong passwords.

Only allowed through console, GUI is not enabled.

To review.

I know the Cisco router is up to date, that's easy to handle. The webserver hasn't been touched since it was launched over a year ago, mostly because of my worry I will break a working environment. You are correct, we are a small and lean organization. I am the single IT guy who really started life as a digital hardware guy. At best, I'm a jack of many, but I am not a master of anything. Hence my post here.

Good. Many people for whom security is not their full-time focus do not realise this and, as a consequence, never argue with their management for enough time/budget to deal with security (and then get into all sorts of problems, both management and technical when it does, inevitably, go wrong for them).

One of the problems in dealing with an issue like this 'at a distance', rather than going in (physically) and, in a more traditional model 'doing some consulting' is that you are guessing a bit about where to probe. If you were there, physically, you'd get more of a feel for the organisation and the short cuts that it might be likely to take. The Microsoft thing may be fundamental (and a subject on which I can guarantee to be of no help whatsoever) and so it would have been helpful to mention it earlier.

You also ought to have a look at 'Cross Site Scripting' to see if whatever your website serving/content solution, it might be vulnerable there.


.

An effort? You ought to be able to ensure that you use strong passwords. If, eg, you allow the use of weak passwords for ssh, and ssh is on the standard port, you will get hacked, and probably fairly quickly, too. It is not whether ssh is a 'bad' protocol (it isn't), but ssh can easily be configured to be insecure. And, if it is insecure, there are enough people trying the handles that one of them will find the problem.

When I wrote "no root log-in", I meant no root log-in. Only allow the use of root after logging in as an ordinary user and using su and/or sudo to get root rights. The would-be evildoers will target root, and if they can't log in as root, that is an additional hurdle for them to have to clear. And, you can log attempts to use su/sudo and if some user who shouldn't be doing that, block them.

And, if you use a trivial password for root (root, or root1, or toor or password....), you probably deserve to be hacked; these will be in the first few that a bad guy (or a bad guy's script) tries.

While this a serious concern, whatever software you are using (webserver, php, cms, home-written stuff, etc) will be large and complex pieces of software. All large and complex pieces of software will have bugs. That is not so bad until someone with evil intent finds out about the bugs or weaknesses and works out a way of exploiting them. Once an exploit is 'out there' in the wild, you need to do something to stop your site being vulnerable before the bad guys get a chance to use it against you.

This demands vigilance, whatever software you are using. You need to be hooked into whatever community there is for your software so that you know when a security update becomes available, you can move rapidly to put that in to your production server. (And, to make a point I made earlier; If this implies a feature update, it might kill your site, so if you can test it out on your development server, that will be easier; not having a development server is a real risk, in these circumstances.)

As an interesting exercise, maybe for quieter times, you may wish to try to calculate your worst-case response time to a security fix. And what if a critical person happens to be on vacation at the time? Or your expert doesn't happen to look at the right site at the right time? Or happens to be busy with other tasks? Even in the worst case, you don't want the outcome to be '...and then the company goes bankrupt, but a couple of weeks later the fix is in place...'.

1 members found this post helpful.