I'm rather glad that there is no one 'guide' text that you can read and which pretends to tell you everything about starting up a new free shell service. New admins that followed it would gain a false sense of security and crackers would have an instant guide to their security setup. On the plus side, we get a lot of different setups between providers, but on the bad side, I've seen some truly terrible shells providers during my time as a user (and later, as an admin).
As most admin skill comes from experience and independent learning, I feel the best thing I can offer you is a short list of tips that are ways into your own learning experience.
1) Reading : Read the Debian Security Guide, and for that matter, all the 'secure server setup' guides you can find. Security starts with your install.
Don't be afraid to install many times until you have the best setup, your final install stays with you a LONG time in many cases, so it pays a lot to get it right. To be brutally honest, 90% of the work involved for a shell server is in the setup, only 10% is in the maintenance.
2) OS Choice : Debian and the *BSD distributions are the only *nix variants that I've used that I would reccomend for use as a public shell server. Make sure you avoid any 'desktop' distribution, even the better ones such as SuSE...... they're good, but the sad fact is they're just not secure enough to be a shell server. When you're giving a lot of completely unknown, untrusted people shell access to your powerful server, security is #1 priority. Your users may not tell you that it is, but I've had users leave other shell providers who value "freedom" (read : have lax security) and ask for accounts with my service because the other services were going up and down like yo-yos due to their lack of security.
3) Attitude : When setting up a public shell server, you're letting yourself in for a whole world of pain (aka : "users"). This world pretty much follows sod's law. Admins of public servers where users tend to fall into two groups in my experience, the paranoid, and the rooted. I've lost count of the number of providers that have started up with a fresh faced, newbie admin at the root prompt, trusted their users, and crashed and burned a few months later as someone roots them 'unexpectedly' and thrashes their system. Security is a process, not a product, a state of mind, not a method. When you start, go SLOWLY, and CAREFULLY. You can always do braver things later when you have more experience. Whatever you do, don't start off thinking "I don't really need to secure that......", and/or the classic "it won't happen to me", becuase sooner or later, it will!
4) People : Beware of unknown users who want to 'help'. Frequently, they just desire root access, or worse. You will probably need a lot of help in the first few months if you aim to setup a good shell service. Fortunately, there are a lot of existing shell service admins who are around and will help you if you ask. (by "help" I do not mean doing anything on your server, I mean offering suggestions as to how to solve some difficult problems which you have already read about extensively before asking for help).
One big tip is that the harder you make it to get an account on your system:
a) the more good users will respect your rules/accounts
b) the more difficult bad users will find it trying to get onto your system
c) the more effective deletions/bans are
d) the better the quality of users you tend to get
e) the more likely you are to get donations from your users
f) the more likely it will be that your users will be people you actually SEE, rather then people who only show up to ask support questions.
do not have a fully automated signup process! I've seen a lot of providers do this, and I can't remember the last time where it did not lead to evil things happening later! Worst of all is signups via a suid root script you write yourself......... PLEASE don't do this! I've seen so many people do it and it lead to disaster, that there is just no way I'd ever suggest anyone do this! Checking users out before adding them is a GOOD IDEA!
If you want to chat to me or many of the other admins of large free shell providers currently on the internet, try hanging around ShellsNet.org, and making it known you are a new admin, and we will add you to the 'admin' forum/channels, that we use to help each other.
Admining can be really fun - I know I enjoy it
The most important thing to remember though, is that it comes with a lot of responsibility!
That, and you'd be welcome on ShellsNet
We do try to help out the newer admins a lot, both directly (by offering them help with various things) and indirectly (by filtering out abusers from the IRC network so that the crackers/spammers/evildoers are kept away).
Good luck with your project, and I hope I'll see you around sometime
-Founder - moonlightglade.net, themlg.net - aka "MLG", the free shells provider
-Founder - shellsnet.org, aka "SN", the free shells network, as promoted by bylur.net