Hi,

I'm trying to secure the CentOS servers on our company network as the current situation is, shall we say, less-than-ideal: remote root logins with the same password across several servers (behind a firewall, on non-standard ports, but still) and several key processes running as root.

My proposal to amend this consists of the following:
- setup a bare as possible SSH-gateway with only the normal user accounts to handle remote access
- disable the root login from anywhere else but LOCAL and create special accounts with root permissions for our ~4 system administrators, like admin.foo admin.bar that can only login from inside the company network, using SSH-keys.

So far my biggest obstacle seems to be creating the administrative users, how do I go about and do that? When I simply create a user adminfoo with uid=0 it will show on my shell as root, which makes it useless as a way to make our admins accountable for their actions.

BTW, my initial proposal to use sudo unfortunately met with strong resistance, because it compromises usability.

I'm also very interested in any opinions about this setup.

Sorry, but SUDO is the way to go. Zero-level accounts may just as well be root, so why bother? The single SSH entry point is the best starting point, in this case.

From that box, each user would be able to connect to the other boxes, as their regular, non-root/non-special user. From there, they can SUDO. Reduces usability? BullS__T....all they have to do is type in "sudo <whatever command name>". Take the "sudo -s" option away from them. Why? Because...once they start a "sudo -s" shell, their commands aren't tracked, except through shell history (which can be altered). The "sudo <command>" is logged, and that log can be mirrored to a remote syslog server, so you've got accountability.

If YOU are responsible for these systems, then YOU set the policy. Who cares if the user is a bit inconvenienced? They can do their jobs, and if typing an extra 4 characters is too much for them....they've got problems. Besides, they shouldn't be working as root all the time anyway. I'm guessing that what they REALLY don't want is accountability. Now, they log in as root, do what they want, and can say, "Hey, wasn't me!" if something went wrong. SUDO changes that.

If it's your neck on the line, cover it. Use SUDO, mirror the logs, and make sure you've got a good audit trail for what happens, and who does it. Because right now if something happens...your company will have YOU as the scapegoat, if it's major. Because after all...YOU set up the security, right? And YOU are the one who couldn't tell them who crashed the box....

You'll want to stick to your guns on this one. As mentioned, sudo provides the flexibility to fine-tune administrative privileges, and it leaves an audit trail.

Let me begin to say that, as a user of Ubuntu since its inception, I wholeheartedly agree with you about the advantages of sudo and I greatly appreciate your input. However, since I have a junior position in my company, I'm not in the position to tell people how they should work. Furthermore, my main task is to make our network more secure from attacks from the outside, accountability comes second at the moment.
As TB0ne mentioned, an SSH-gateway will improve the situation greatly in that respect. Every admin has his own workstation here, so we can use originating ip's to identify who's who. SSH-keys will keep our servers secure, even if the gateway is compromised.
That said, sudo could be good solution for our application-administrators. They are not busy editing system files all day, they just need to do things like restarting a process sometimes. If some of these applications weren't running as root, they wouldn't need root access anyway.

System administration in my opinion should be a good balance between usability and security. So I'm curious now, what kind of disaster scenarios could you come up with that could compromise this setup (no remote root, ssh gateway, admin-accounts with ssh-keys for 3-4 persons and sudo for the rest)?

Sorry, I totally disagree. You may have a junior position, but you have to realize that YOU are the one that's accountable. They WILL throw SOMEONE to the wolves if something goes seriously wrong. If you're responsible, you set the policy. If you can't, get your boss, and your boss's boss, to sign off on something (in writing), that says you are in the clear.
Right. Even if they ARE running as root, sudo can still fire them up. Even as a sysadmin, I use sudo. Keeps mistakes from happening, to a point. No one is perfect.
Well, you'd be pretty well covered, but if you leave the workstations open, you're still swinging in the wind. Put good workstation policies in place, to lock the screen, etc. Also, give those workstations static IP addresses, and tie each user account to an IP. So Joe can only log in from 11.22.33.44 or the console, no where else.

Then let's work it from the security angle: what happens if one of your sysadmins' boxes gets pwned? You have no way of enforcing passphrases on their private keys (used for Pubkey Authentication). If one of your lazy sysadmins uses a passphrase-free key, it's game over for you. An industrious cracker has a direct root login to your server.

If instead your sysadmins were set up as sudoers, under the same scenario the cracker would only have a sysadmin login to your server. (That's not a good thing, but it is a much better situation for you, because you will have more time and opportunity to detect a problem while an exploit is prodded for from a regular user account.)

With respect, this should be standard operating procedure where multiple sysadmins are administrating the same host(s).

Well, this is something I thought of today: since most of the admins use Windows 7 on their workstations (actually, I'm the only one with a linux workstation), I figured key-management would become a lot easier if I let them SSH as a regular user to the servers and then SSH with their special account to localhost, so both the public and the private key would be stored on the server. As I'm writing this, I realise this sounds like a security hole, but then again, the keyfiles are required to be readable only by the user that owns them.

To summarise: when a workstation is owned, the attacker only has user-access to the servers. I'm pretty confident we can enforce a no-empty-passphrases policy. To own a workstation is not impossible, but hard, because it typically requires user interaction. To gain user access from outside you would first need to gain user access to the SSH gateway and then to a (production) server. Preferably then, the OS on the gateway should be a different one than on the production servers, to prevent crackers from using the same exploit. What remains then is the threat of a lost or stolen password: there this solution is on the same level as sudo.

Well, you got those opinions from a couple seasoned sysadmins on this thread. It sounds like you generally do not agree. I've nothing more to add.