If I deploy Linux, who will support it when I leave?
 

I'm a sysadmin for a small nonprofit which currently runs a mostly-Windows environment on Server 2003 in dire need of a hardware refresh.

I've addled my brain for the past several weeks, coming up with all kinds of amazing things I could offer them with a Linux/BSD-based setup, all for very little cost compared to their commercial counterparts.

My ideas involve, among other things, deploying Xen or KVM to virtualize the new server; replacing Server 2003 with CentOS & LDAP or Samba4; and handling storage with FreeNAS/ZFS as either a single-node or as a 2-node HAST failover cluster, depending on budgets. The system would include failover of most layers to keep uptime high and maintenance time low.

Overkill? Maybe, but I suspect that investing upfront into a good solid infrastructure like this will pay dividends in less ongoing maintenance, great performance, no licensing costs, and plenty of options for the future. That besides, I long for a chance to hone my skills in many of these areas, so it is a mutual benefit.

The only question is support if/when I leave. I live in a fairly rural part of Northern California, and I don't know a soul who could support this setup, even with meticulous documentation. I don't want to leave this organization holding the bag for my pet project that only I can support.

What have others done in situations like these?

So I would start with the premise that ANYTHING you implement must be documented - even if a redo of Windoz.

A non-profit that requires meaningful IT operations must have a budget (or KNOWN volunteers with capability) to support the IT services that are required to meet the needs of the organization. That review will be the truth test of whether the IT services are truly required.

For instance, if you are providing 20 hours of support per month, assume an hourly rate and establish a budgetary target that must be satisfied to complement or replace you. Raise that as a question to the business/operations coordinator. "If we build what you require, it has an effective cost of $20K per year to support it. I'll volunteer (or be paid) for my time to get it running. But you need to be sure you have an ongoing commitment to pay someone to care for this over the years - including some planned investment in refreshes of hardware and software".

Those conversations are really hard in a small non-profit, but it's an essential gut check when making the investment in the first place. Basically, if you build it cheap and slick and then it breaks down and becomes useless - there's an argument to be made to not build in the first place.

All of this budgetary planning is a foundation to find a foundation for which external/remote support can complement some paid on-site care that might need to be hired in. With time and training I would expect that this becomes supportable.

thedaver pointed out that ANYTHING you do will have to be supported and well-documented. Honestly, if you turn over a huge binder full of notes on a Windows setup, will anyone be able to support THAT?? If there's a dearth of IT folks in the area, whoever they get will be over their heads with WHATEVER you load.

If you deploy Linux, and document it well, anyone who knows IT in general should be able to decipher what you leave. If they need a consultant, all they'll do is change "Must have experience with Windows servers" to "Must have experience with Linux servers" in the ad. Also, you could always leave a maintenance account on the system for remote support, and put all your notes/docs in that users home directory...anyone who logs in can see what runs where, why, and how to do things to it. Doesn't get more simple than a .txt file. :)

With CentOS, you could even write a small shell script using SUDO, to lock that account and disable ssh, and another one to UNlock it, and enable ssh. That way, only if someone from that site calls for help, will the service/account be available. Better for security, easy for them to work with. I'd load Linux, and maximize the non-profit's profits, and keep them current.

Let me be the third person to emphasize documentation. Linux as a server platform is no harder or easier to make sense of than Windows, provided there's some half-decent and reasonably up-to-date documentation.

The answer to your question, "who will support it when I leave", is hopefully "someone with the necessary skills". That goes for both Windows and Linux. It may seem easier to find someone capable of supporting and maintaining a Windows system, but you may find that a lot of those who claim to "know Windows Servers" have quite limited knowledge and/or little experience.

<minor rant>

One problem with the Windows platform is that somebody with only the vaguest idea of what he or she is doing may be able to keep the network limping along for several months, perhaps even years, as anyone reasonably comfortable with the Windows desktop can figure out basic stuff like resetting passwords and adding accounts by simply exploring the Start Menu or type semi-intelligible sentences into Google ("help windows password doesnt work").

Some see this as an advantage, but I believe it only postpones the inevitable. Sooner or later a real problem will surface, often one that could easily have been avoided if the person responsible had known what he/she was doing, and then the organization will suffer downtime. Also, they will most likely have to pay a consultant a small fortune to get them out of the mess.

</minor rant>

I have worked for some not for profit and I know what you mean.

My suggestion would follow any common business plan. Use a 1,2 and 5 year plan to decide which way to go. Only you can plan this out.

If you find that the 5 year plan holds for you going to linux and they have used their money for the best use then I'd think it wouldn't matter what happens. I'd guess it would be possible to get a linux or windows person to take over.

I can't agree more with the suggestion for keeping good records of your actions.

I actually have thought about this at home. I gave up Microsoft entirely in 2004. All the computers in my home are Linux. So what happens if I die? What will my wife and daughter do?

This bothered me for a while, however, they are pretty competent at 'using' Linux so the worst case would be they would have to copy files onto a Windows machine.

But more than that: realize that a Linux box will stay STABLE for years and years. Over time, Windows can and will get clogged up and may eventually fail due to this. Linux won't.

You have to ask yourself, would I be asking this same question if I bought a printer? or some other piece of hardware? No. Because the printer is stable. You know its not going to go bad just from use.

Windows computers can go bad just by "using" them. So you can't apply Windows mindset to other OSs.

That's my $0.02.

Good Luck and have a great day.

I suppose it pertinent to mention that I am myself quite invested in this nonprofit. I don't serve it coldly from a distance; I thoroughly love the work they do in our community, and so what money they do pay me isn't my primary reason for concerning myself with this. I want to make sure they have what they need to do what they do for as long as possible. This research, as well as much of the work I do, is pro bono.

At this point the most important consideration is what is best for this organization (not for me or my interests).

I'm going to answer this somewhat out of order:

(others have reiterated this)

Absolutely. I have both documentation of the overall configuration as well as daily logs of what I've done. I have a KeePass database with all necessary credentials and I've left the second-in-command with its passphrase. All of this is stored on-site in a known location. I won't belabor the point that I agree 100%.

A few others also mentioned, at least in an implied way, that any competent IT tech should be able to handle a well-documented Linux setup or a well-documented Windows setup. I have pondered this long and hard.

I can not speak for the rest of the world, but I can speak for my area: I've worked at one of the (now only) computer repair shops in town, and I'm currently employed in an organization with perhaps one of the largest IT budgets in my county. Suffice it to say I've had a fair sampling of the local talent, and talented indeed they are: but not in Linux. I have one co-worker who could take over the setup I described in my OP, but every other one is steeped to the core in, shall I say, "corporate computing." VMware, MS, Bomgar, Aruba, you name it. It was with difficulty they configured our only Linux boxes, a few DNS servers running CentOS (which I helped configure during my first week on the job).

On the contrary, I know for a fact the current Server 2003 setup was deployed by a local consulting firm, who could no doubt administer a well-documented Server 2008 infrastructure.

As much as it pains me to say it, it would take a certain kind of naivete to assume that any competent IT consultant in my area would be just as well-off managing Linux as Windows. At least, for now--hopefully that will change!

At first I considered this a strength of the Windows option, at least from the perspective of "what's best for my client." I thought it was best that they could at least find some volunteer with basic Server knowledge who could "limp it along." Now I'm not so sure, because you're absolutely right: that volunteer will most likely mess things up, and when things do go south, it'll all be for naught (more expense, as you say).

This is a point I will certainly return to in my decision making, because it holds sway.

This argument was for me, in the beginning, one of the strongest arguments for going the Linux route. Especially Debian stable "just works." If nobody's fussing with it, it *will* work today if it worked yesterday, assuming the hardware hasn't faulted. The same can't be said about Windows, not by a long-shot.

That was my original vision: set up a very advanced yet robust, tried-and-true platform on Linux, then sit back and watch it pur along indefinitely. With Linux there is no pressure to upgrade, no "defective by design" crap, and it can squeeze the most out of hardware.

This, along with licensing cost savings, I think is the strongest argument in favor of the Linux option (along with the "sky is the limit, don't paint us into a corner" argument).

You reach a sticking point with this org. I love them, but I loathe them on this point. They have even said it: "the squeaky wheel gets the grease." Basically, there is very little strength in the organization/admin side of things, especially when it comes to IT. It's basically "when it breaks, we'll find some way to fix it...hopefully." That's where I've come in during the last 3 years and been the silent knight, keeping things moving at a bare minimum of expense. Half of me has often thought that backing off and letting them fall flat on their ass would be better in the long run, so they can see what it really takes to keep an org going the way they want. But that is another discussion.

Here are a few things I can do to make the decision easier:

- Research available support. Indeed, I've already found http://www.debian.org/consultants/#US, some of which are in San Francisco, 3 hours away but still manageable in a pinch. If I could get a good, vetted backup tech in place, I would feel 100% better about deploying Linux.

- Pilot my project at home. I suppose much of my trepidation here is that I've done a lot of this, just not all of this together. I keep envisioning a house of cards collapsing, but I work with Windows for a living (gah..) which is par for the course there. Maybe if I could get a small test lab running the way I envision it, and it works stably, I'll feel more comfortable.

- Lastly, after all this research, I can put it by the client as two options, and let them decide. After all, it is their network.

Thanks all for your input, I'll be sure to report back what comes of it!

I agree with you...but. :)

I don't think they could just step in and fix things, because I'm not that naive; but, I DO think that if it's documented and the person has a bit of common sense (which can be VERY uncommon, I'll grant you), they could figure it out. I don't think they could fix a problem in a short time, but could follow the breadcrumbs left in the documentation. That, combined with Google and forums like this, could get them going again.

Could they immediately configure different DHCP zones with static addresses? No...but someone who is technically competent could follow the terminology, and read the documentation, and piece things together. Granted, a ten-minute fix may take an hour or so, but it's not a show-stopper. Could they restart a service, or restore a file from backup? Certainly.

I don't know how many users you're talking about, but I'd consider going with two (?) physical servers instead of VM, if maintenance is an issue. A warning-light on a physical box is easier to troubleshoot than a VM warning if you're unskilled with it. Using DRDB could solve your disk sharing issues as well, eliminating the ZFS. Heartbeat can fail services over, and is also easier (in my opinion) to deal with. Even if you DIDN'T use heartbeat, you could even just lowball things, and write a script that someone could manually run to start the needed services on the second server, if the primary fails. Downtime would be limited to just a few minutes. Total guesswork, since I don't know the number of users/goals/etc....

Having given careful consideration to the status of your organization as a "non-profit" and that it is running now on a Windows 2003 environment, I would not recommend that you switch it to Linux. Instead, I would recommend that you upgrade the hardware ... to equipment that is reasonably faster and in all cases "electrically and mechanically more stable" ... and that you either continue to use Windows 2003 on it, or else upgrade it to a more-recent version of Windows. :eek:

(Be sure to get a server edition of the Windows product ... don't skimp on the software.)

My reasoning? "Your alternative proposal represents Disruptive Change." The most important thing that the organization needs to receive is predictable continuity of operation ... that is to say, the least amount of disruption and especially the least risk of the worst-outcome scenario: the effective loss of ability to pursue its core-mission (which is not "running a computer!"), and unjustifiable difficulty in timely recovery from that scenario in your absence. (Sux about that bread truck, pal ... enjoy your harp lessons.)

To me, it's not a question of "whether Linux could do it." I see no compelling business reason to move the organization so far away from where it is today. The ROI isn't there, and the risks are unwarranted. No matter how "good" Linux is (no questions there), or how "good" you are.

I have a similar situation at the museum that I volunteer at. Some time ago someone donated a couple of laptops running Ubuntu. After getting frustrated with my efforts at updating Open Office to the latest version, I installed Slackware on them and set them up as fool-proof as possible. I use slackware all the time so it was no bother.

When I turned on one of the laptops later I found it was stuck in the 'boot into installation cd and perform a fsck' position. Obviously someone didn't know how to shut down from kde and did a hard power off. The problem is that I'm the only IT guy there so nobody else has any kind of boot CD or would know what to do even if they had one. Sometimes Windows is just so much easier on users...

It is still a business plan decision. Don't put your feelings into this.

There will be someone available for linux just as there will be for windows. Use the plan. Stick with a 1,2,5 year plan. If that plan calls for windows or mac then use them. If it calls android use it. Who knows what will be around in 5 years.

Consider the poor users of Novell that may still be running out there.

Why would the statement and question become important?

Because there is a dearth of linux persons in the area while presumably windows doesn't have this problem. This is the OPs reality.

He should first get rid of the idea of longing for a chance to "hone my skills". If he isn't sure exactly what he intends to give, this wish list may end up saddling the user with systems requiring manual intervention. I mean why even think virtualisation and fail over if they have a single server (currently running windoze).

Better first find out what applications they use. Proceed only if the application will run on CENTOS. If you are lucky and the organisation is incurring cost on W2k3 (ie. not gifted a license by a sponsor) thenn (then and only then!!) proceed to make a play for cost cutting and future growth.

Ok