How-to secure public servers
 

Hi,
I am looking for a way to secure and monitor, ca. 20 servers all running suse 8.2 - 10.2, that have global IP addresses, and are not behind a firewall. This servers run apache, mysql, pop, smtp, etc...
I want to setup one machine as a "monitor". I have looked at Snort IDS but would like some input before i start reading, installing and experimenting with it.
Currently the only security implemented is, to disable all unused services, restrict access to certain services based on IP, disable root ssh and only allow certain users to use ssh.

Any suggestions on how i can improve the security, which IDS i should use, etc..., is appreciated

Rgrds
Markus

The short answer is check out the LQ FAQ: Security references, read, then return to ask specific questions.

IMHO there is no long answer, at least not at this point. Sure anybody could post some measures assortis right away and w/o thinking, but I'd rather assess the situation first and *then* come up with coherent advice.

We should look at all aspects of the machines involved, including purpose, network segmentation, maintenance and incident history, measures implemented, etc, etc to come up with say list of priorities. From that you can pinpoint things that need to be fixed urgently and make a top ten list of things that are necessary to reach overall "base level" security. After that you implement the rest of the measures to improve security further.

ca. 20 servers all running suse 8.2 - 10.2, that have global IP addresses, and are not behind a firewall. This servers run apache, mysql, pop, smtp, etc.
- are all the machines located in one place?
- what is the purpose of these servers? (ISP, business, other)
- what is the relationship between servers? That is, do some or all provide services *together* (think Apache + MySQL + Squid = application)? Are there some that provide services to one, two or more separate domains (LAN nets, Internet)? Are there some that provide multiple services that really should be separated? (say MTA vs LAN firewall/router)?
- is there a specific reason all have public IP addresses?
- is there a reason why the distribution release version differs and includes stale release versions?
- what is the integrity state of the machines? (Known problems, past intrusions etc)
- if there is any "evidence" of past hardening, auditing and maintenance, do you have access to relevant admin logs?
- you posted some measures taken. Can you point which ones have been done on all machines and what was used? (Just name the tool: sshd, tcp_wrappers, Xinetd, iptables),
- is there a todo list with security-related tasks that need taking care off urgently?
- are you in any way constrained wrt effort, time and money (who isn't)?

* Please take your time answering/asking. There are no "wrong" or "right" answers. In this case "more *is* more", so the more information the better it will be. If there is anything security-related you would like to mention (past incidents, unrelated events, suspicions, etc) please do also.

** If you're OTOH not interested in this approach but would rather like to get a set of generic security-related measures just say so. In that case you can skip back to the short answer.

For large amount of servers I know it is very hard to monitor and that is why you do need an IDS, if not active IDS the least a passive one.

Sure, you will need one "Server" IDS that all other IDS agents installed on the machines will report back to the central monitoring workstation based on certain rules such as for example:
If an attacker is doing a brutte-force on your SSH port (10 New connections/second) send report back to the IDS
i.e iptables -I INPUT 1 -p tcp -i eth0 --syn --dport 22 -m hashlimit --hashlimit-name SSH --hashlimit 5/minute --hashlimit-burst 1 --hashlimit-mode srcip --hashlimit-htable-expire 1200000 -j ACCEPT

In that case if 4 connections (new) per minute (averaged out within a certain period) from a specific source that tries to connect to SSH port from one instance put the offender into table and don't expire for about 120K miliseconds
now after ACCEPT no longer matches because it is over 4 connections/second you can have another rule bellow it and say:
iptables -I INPUT 2 -p tcp -i eth0 --syn --dport 22 -j DO-SOMETHING

in this case DO-SOMETHING it canbe log-it to IDS agent and then IDS agent will send that incident to you. Simple but very effective.

Another advice is if a Server doesn't have to be public (such as MySQL Server, Mail Server, etc.) place them in a VPN tunnel or behind a Firewall that way the intruders will not be able to see the server and the payload will be much, much lower because in every server where there are many iptables rules you loose performance when there is an attack because iptables cannot process rules in the Layer 2 or 3 of the OSI. They are processed at the Layer 4 as far as I know which for a large traffic to take something to a Layer 4 is critical (lots of latency=lowbandwidth=low performance).

Of course, if these servers are very critical you may want to get something like a CISCO ASA.

i am not sure "just" a passive one will do you any good ...yes you know you have been attacked but the attacker will still have compromised your box and you will need to reinstall just to be sure because anyway the IDS will tell you what attack was used but not what the guy did once root on your box.

any reason why there is no firewall1? choice ? no fund? as toni said a firewall would allow you not to have iptable on your box in case of an active IDS.

if possible upgrade all your box to the same version if only for ease of managing them. apply Bastille and configure SElinux if not done already. for public servers it is the minimum you can do.

I dont know why people post threads like this, it is like "Dont you read the books?"

The point of a Passive IDS is for a 24-7 IT Personel who upon receiving the notice (e-mail/message) in their Blackberry or e-mail they can imediatly take the neccessary measures. Now, it is unlikely that the attacker will be able to gain access in just 5 minutes. They have to assess the security layers that are in place before they can lounch an attack--that is if you have any security layers in place such as strong passwords, not leaving things by default, placing Web applications in a sandbox folder, and controlling the application's user/group permisions.

Thanks for all your responses.
I have updated all servers to same version, and also installed tripwire on each.
The next step will be to install an IDS, most likely SNORT.
The security reference page is extremely helpful.
I will post back with more specific questions.