Lets suppose the server runs in an enterprise environment.
OK. Then for me the trail would have to start way before hardening, at the point of aquisition or right thereafter. My first question would be: who commisioned the box (or for whom was the box commissioned), what's it's purpose and where is it located?
-
Who: leads to firing off enquiries with respect to business requirements, deliverables, risk, product/data and timeline. For instance this may be a box prepped by a hoster which is a different set of variables compared to prepping one yourself and delivering it to colo. It may also mean we learn that, due to specifics of the business deal, we'll have to apply some extra steps to ensure data protection or fix some problems.
-
Purpose: gives scope on what the standard outline for prepping the box should include as well as any special points of attention. Ask yourself: what are we securing against? After all you don't want your Snort bridge set up the same as your Oracle box located behind two firewalls. (Should also check for interaction or "trusted relationships" with adjacent boxen. Kerb? RADIUS? PDC? And *exactly what* kind of traffic/behaviour must be expected in this "trusted relationship"?)
-
Location: is a trigger for things to check and do especially when dealing with remote systems, from Out of Band access and subnet separation / existing firewalling and NIDS to standard operating procedures and necessary people to liase with or alert.
Information leads (or it should) to insight in what you're dealing with and their relations. This data can be broken down into tasks. The deadline shows you how much time you have for delivery. Tasks set off in time, reshuffled for risk and other points of attention gives you a checklist of tasks to perform, a planning. Planning and checklists allow you to focus on the stuff that really is necessary, leaving the rest for later. Without it becomes rather inefficient and more errorprone.
This all may seem a bit awkward to introduce this all when talking about simple host, network and application hardening and auditing (wel'll leave Pentesting out of this, more on that later on) but I add this to emphasise security always will be a trade-off. Not only between total system usability and total system security but more due to (or lack of) capacity planning, required information, and time.
Admittedly this will only be interesting for those of you who are involved in project-based setups: before I go on I'd like to mention that the information you gather could lead to answering) questions (or noticing pitfalls) in a wider scope, like was it really necessary to dump those five none too similar tasks on one box? What if the relationship with box X or net Y is broken? Should I alert on NAS failure? Was this application (well-)tested? What I want to say is that if you have a chance to make yourself be heard during inventarisation and design phases your knwoledge and experience can make a difference with regards to the quality of the project, in the end the quality of what you'll have to deliver (and maintain).
That said, let's look at your list and reshuffle. Not I don't add all details for full system hardening, you know where to get the info:
Group one
0. Firewall inbound. Independant of where the box is located this is the first thing to do in my opinion. The rationale behind raising the firewall instantly is that if you only allow traffic to and from your management IP (or range) you'll limit risk of network based attacks (and get traffic logging on the side). This is also why knowing the purpose and location can be important: if the box was set up by your friendly colo people you probably asked them in advance, based on the purpose of the box, to only allow pub traffic to port X and Y, which saves you time when firewalling. If you didn't and you did't bring your hardening kit to the box you could even suffice having a remotely accessable script and running "lynx -dump proto://file|/bin/sh". One could argue stopping unnecessary services is the first thing to do, but it takes time to figure out what to stop and you can only stop one at a time (unless you have a script, OK). Of course there are scenario's where the box would have been handed over a long time before you are able to devote attention to it. That's where the next point comes in.
1. Information is everything.
Having baseline data enables you to check for differences. This means backing up and extracting data from the package management databases (if the distro provides them), installing software to list , keep track of and alert on changes. Decide, based on requirements and position of the box in the network if you need Samhain (which can actively track changes but requires a kernel module) or passive tools like Aide or md5deep or even tripwire. Consider running Servdoc which gives you basic configuration overview (you can probably fit that in if you need to work on rudimentary configuration management). I like to add RCS as a minimal tool to help me store and revert configuration changes. It also a good time, and I can not emphasise this enough, to set up and keep an admin log. Not only does it help you keep track of problems, changes and notes, it also helps anyone cooperating in or taking over management.
With backup in effect, baseline data saved off-site and changes traced you'll have a good basis for making changes and reverting back if necessary.
Group two
2. Purge. Unless there are more urgent security issues that must be addresses first, purging any software and configuration not used
now should be done before updating. Less software means less things to check and manage, a smaller footprint of the box with respect to remotely exploitable conditions (and maybe even performance) and having less packages makes upgrading (somewhat) faster. The more evolved a package managers is the easier to accomplish this task is. It's also a good time to ask yourself if you really need compilers. As a rule of thumb they have no business on production systems. If the box really can't do without mark those packages for special treatment later on. This also shows the necessity for a controlled development or staging box you can build and test packages on. You'll need one if you need to introduce features your distro doesn't come with, like building a GRSecurity-patched kernel package.
3. Upgrading and installing. After upgrading and installing always check for purgeables. Luckily you tested installing your custom kernel on the staging box, so installing it now won't make you twitch.
4. Initial assessment and system configuration. Running your filesystem integrity checker shows a lot was changed after packages where removed and added and users where added. Change means potential problems, so assess using checklists from SANS, your company/hoster and project and application-specific checklists. Fix the system inward-out: start with the kernel and kernel sysctl's and work your way through directory and file access and users (sudo, rootsh). The details are in the LQ Security references. When done check with Tiger, NSAT (if supported: not Mixter's but Number9's) and env_audit. You could run Bastille-linux, but personally I like to stay in touch with what I change. If I ever run it it'll only be in report-only mode. This is also the time to look at system auditing: make sure system auth like PAM and syslogging (remote logging?) gives you all you need: having too much info isn't a problem because it can be filtered down using say Logwatch, but having not enough info will make you miss vital alerts. If you can't run GRSecurity/LSM, at least check if you can use alternatives, like running Spoon's Lcap to minimise capabilities, tighten up mount flags and access permissions and get as much active checking as you can afford.
5. Assessment and configuration take two. After testing the system works as expected including a reboot and a partial restore(!), configure (kernel) and network sysctls, applications and firewalling since they are in close relationship with eachother. This is where you'll likely spend most of your time. People who set up LAMP boxen with (deemed eternally vulnerable) PHP / bulletin boards / CMS and people who (need to) expose management features of applications over the 'net should take extra care. Another tool I'd like to introduce is Monit. It can watch any application and system or application resources and alert and act on changes/limits. Anyone who has seen how much fun Apache can be after it's logsize exceeds 2GB knows what I mean. Especially if you find out from the customer (and not alerts, nooo) calling you outside of office hours. Rerun your filesystem integrity checker, assessment tools.
Group three
6. Remote assessment. With the box reloaded and rebooted you could say it's time for a remote assessment. I favour running Nessus. If your box is in colo, check with them first. I don't advertise for paid services, but if you can't run Nessus try
www.securityspace.com even their free test is worth as much as a Nmap scan. Other tools to throw at it are Hping, Nemesis and Scapy.
To me securing a box means protecting resources and maintaining a constant level of operation in the face of remote and local threats. Auditing then is checking and adjusting to changing levels of threat. Penetration testing, in relation to the previous points, goes against the grain as it's all about trying to break stuff in every conceivable way with the added bonus (risk) of hitting every system along the path. In my humble opinion that's better off in a controlled environment, where applications can be tested before they are rolled out. Anyway. If you need still more then you're probably heading for pentesting country. Take it from here:
OSSTMM Open Source Security Testing Methodology Manual:
http://www.isecom.org/osstmm/
OWASP: The Open Web Application Security Project:
http://www.owasp.org
Metasploit Framework:
http://metasploit.com/projects/Framework/index.html
8<---------------- I'll cut off my mind dump here since I'm tired writing.
In closing I'd like to thank you for the trigger to write about this. I don;t know if it comes in handy, if the intention is clear at all, if it answers any questions or if it can be part of a discussion. Also your (anyones) knowledge and experiences will differ from mine, resulting in a different POV or approach. That's a good basis for discussion I hope. Constructive comments are welcome, as always.
I also want to make sure the point comes across that, next information, preparation and planning should be considered important. Preparation makes sure you get a grip on what to focus on. Planning shows when you have to do it. If you don't have the knowledge for a specific task and if you can't hire anyone, you know you'll have (time) to increase your knowledge by research and practice before the task comes up. Pace things. Don't hurry.
I'd like to remind you all (and myself equally) that
knowledge isn't only what you know but knowing how and where to find it.
