Although using Slackware in the enterprise has been discussed in the past in this forum, I don't recall the logistics being discussed.

If you support Slackware in an enterprise environment, how do you maintain the systems? Please notice I wrote "support" and not "use." By that I mean you support Slackware systems other people are using and depending, not maintaining your own Slackware system.

I am interested in those of you who support computer systems for other people. How do you automate your maintenance of those systems? Patches, tweaks, help desk support, whatever? Especially long distance.

Are you using thin client connections?

Are you on call 24/7?

What is your basic routine like?

What is your worst experience?

Slackware does not have the same repository eco-system like Debian. How do you handle software installation requests by clients and users?

Etc.

Thanks.

I cannot speak for my friend, but, I know he has a business based on Slackware.

http://www.microlinux.fr/

As hitest pointed out, my business is almost entirely Slackware-based, with some RHEL installations. Maintenance more or less follows the KISS principle.

I have a couple of public servers with Slackware64 14.0, and these are kept in sync with Slackware updates. Meaning whenever I get an email from slackware-security, I apply the patches.

LAN servers and client desktops are less critical, and I update these locally, let's say every couple of months.

All desktop clients are running MLED, my personal blend of Slackware + Xfce: http://www.microlinux.fr/mled.php. Third-party packages are all in one centralized repo (http://www.microlinux.fr/slackware/) configured with slackpkg+, so upgrading stuff is only a matter of 'slackpkg upgrade-all'. Prior to Slackware, I had been using CentOS, Debian and Ubuntu LTS for these tasks. Slackware is much work, but a) much more flexible and b) fun. Plus, it's actually great for teaching (which I also do).

For communication with clients, I'm using a) email and b) a simple landline with an answering machine. I don't have a mobile phone, nor will I ever have one. Clients are puzzled at first, but most of the time, they end up telling me I'm often more responsive than some big shot companies around here.

No "worst experiences" so far, except of course handling the odd client from hell, but I've gotten quite proficient in detecting those and sending them quickly back where they came from. My choice of Slackware is primarily because it's ultra-robust and ultra-reliable, and it doesn't give me any headaches. More recently, more and more client want "this Slackware thing" on their home computers, as a Windows replacement.

2 members found this post helpful.

Installing a Linux workstation with bells and whistles for an officer of the French army. The guy had a very precise idea of what he wanted, so I spent quite some time assembling a state-of-the-art machine, installing and configuring the system and the applications, sanding down the edges, etc. He seemed quite satisfied with it.

The next day he was on the phone yelling at me about this crappy Linux system. I eventually found out the guy had wiped clean the computer partitions and reinstalled - well, tried to - some Linux CD he had bought somewhere with a magazine, only to see if he was able to do it. When I asked why he would have done that, he explained something like "IT'S MY COMPUTER AND I CAN DAMN WELL DO AS I PLEASE WITH IT!!!" I politely explained all the options he had now and the documentation he would eventually have to read in order to restore everything, which he didn't like. "I WON'T READ ANY DAMN DOCUMENTATION! I'M NO IT EXPERT AND I HAVE BETTER THINGS TO DO!!!" I eventually gave him one piece of final advice: go see a shrink, and never call me again.

That was my worst client, I guess.

2 members found this post helpful.

Isn't Slackware better than that?

Back on topic: My worst mistake was failing to properly update my network settings before rebooting after an upgrade over a ssh connection. Everything had to wait until I could physically access the upgraded server.

The last time I had the freedom to use all-Slackware infrastructure was 2001, and it wasn't much -- less than a dozen servers at its peak: one bastion/load-balancer/mailserver, two mysql database servers, and a few (five? six? can't remember) webservers. It was a small company, and I was the only technical person -- programmer, webmaster, system administrator, hardware procurement. Just about the only thing I didn't do was write good-looking html -- the VP did that.

The company eventually went under, but I was expecting it to grow quite a bit larger, so my administrative setup was a bit overkill for so few machines: Nagios (which I could monitor from home), and ssh to the bastion server. Once on the bastion, I could either ssh to any other server, or mssh to run commands on all of them concurrently. I had a cron job to rsync volatile data around so if a server went down I had all its stuff on at least two other servers. The database and webservers failed-over (if not always smoothly or transparently), but everything else was manual.

There are better tools today, and I would have availed myself of them (like ipvs, keepalived, chef, tungsten, glusterfs, ganglia; some of these existed in 2001, but I didn't know about them yet). The Nagiosgraph plugin would have been welcome, as well. I used cvs (in lieu of a better revisioning system) for both code and configuration files, primarily apache-httpd configuration -- I'd make + test a change on one webserver, then commit it to cvs, then pull it from cvs on each of the other servers. It made for easy reversion too.

I was on call 24/7, and was usually the first to notice problems, but would sometimes get calls from the VP (who was the primary contact for our japanese clients, who were awake when the rest of us were sleeping and would sometimes call him in the middle of the night). My wife and I were living in a cottage at the time, and I could see my desktop's monitor from almost everywhere. I left Nagios up, and glanced at it from time to time throughout the day. If something went wrong with the infrastructure, rows would go red, and I'd notice them.

Worst experiences .. our demographic data collection software was all homespun, and sometimes wouldn't get everything into the database. The VP (who was also the statician) would notice a hole in our data via our realtime data dashboard webapp, and call me, and I'd fix the problem and then write some perl to extract the missing data from the web logs and inject it into the database, before the customers noticed and freaked out (they also had access to the dashboard app).

Once, before we had redundant database servers, the database went down (a hardware problem -- I don't know if you remember the rupturing capacitors problem with some year-2000 motherboards, but that was what happened), and I had to rush in in the middle of the night, figure out what was wrong, yank a webserver from the http pool, transfer the database server's disks to it so we had a working database server again, fsck, then transfer the demographic data from the webservers' logs to the database with some hurried perl. It was night, so I knew our japanese client was freaking out, and it needed to be done posthaste.

Other than a few of those, my experience there was fairly benign and low-stress. It helped that I was using Slackware for everything, so the system software was rock-solid, bug-free, and already had almost everything I needed installed (and I was writing the rest). A system would only go down from hardware failure, power failure, or me turning it off.

I'd keep the VP, President, and Project Manager appraised via email, first telling them there was a problem and I was looking into it, then telling them what the problem was and how I was fixing it, and then telling them everything was working again and no data was lost. I didn't know it at the time, but only the VP actually read those emails, and he'd call the President on the phone with one-line summaries: "There's a problem, we're on it", and "Everything's okay now".

I applied no patches to our software, as I believed (and still do) that patching incurs risk of introducing bugs to the system. That was thirteen years ago, though, and I doubt you could get away with such practice today. Patching security vulnerabilities is necessary.

For larger and more "enterprisey" environments, I'd use the same tools that I have with non-Slackware shops: A ticket-tracking system where issues, features, and bugs are submitted and tracked (Trac, JIRA, and Redmine are all good systems for this). Chef or puppet for centralized configuration management. Rsyslog-ng for centralized logging. GlusterFS for distributed and redundant file sharing.

If I were to support nontrivial Slackware-based infrastructure today, I'd also want to set up a Slackbuilds cache on the local network, so that if I ran sbopkg on a hundred servers at once, they'd pull the sbo's and source tarballs from the local cache, and not each of them reach across the internet to pull the same data a hundred times. Ditto with CPAN packages.

2 members found this post helpful.

ttk,

A very interesting story! Thanks for sharing that with us.
Are you doing system administration work these days?

Quite welcome! Not much, no. I've always been more Software Engineer than System Administrator, though I did play the Architect role some in 2005 and 2009. These days I'm a straight-up distributed systems engineer.

http://ciar.org/ttk/resume.html

2 members found this post helpful.

This is hands down the most impressive sysadmin résumé I've ever read.

Indeed! It is very impressive!

You automate the updates or ssh and update manually?

Automated or manually?

You use Slackware in your classes? What do you teach in the classes?

Do you teach other subjects?

I have one for emergencies, mostly for when I travel. Only immediate family members know the number and 95% of the time the phone is off.

Do you find there is a market for home clients? Perhaps your neck of the woods is different, but over here where the intelligence quotient is directly related to the proximity of the remote control, I envision the following happening:

"Hey, this Lee-nucks stuff works great! Thank you!

Within a few days:

"Hey! WTF! How the hell do I download Netflix?"

"You can't."

"Then get this sh-t off my computer!"

What about your business clients? How do you handle vertical software? Over here QuickBooks reigns, for example. If you use VMs, then why not just keep them on Windows directly?

Slackware 14.1 seems enterprise "ready" as is, other than offering paid support and a ton of GUIs, what else does Slackware need?

Actually you can, fairly transparently, using the "pipelight" browser plugin, which uses a modified private version of Wine to do the rendering of Netflix streams and other non-Linux stuff.

Eric

5 members found this post helpful.

Many thanks, Eric for your work enabling us to watch Netflix on Slackware. Works like a charm for me.

http://alien.slackbook.org/blog/pipe...inux-browsers/

It needs something akin to Kickstart and Spacewalk, for centralized mass-installation and management of hundreds or thousands of servers. (The post-installation functions of Spacewalk are better served, imo, with chef, nagios, and other tools.)

It needs out-of-the-box support for more of the kinds of infrastructure software used in the enterprise, like ElasticSearch or Solr for search, Zookeeper and Gearman or SGE for job dispatch and management, GlusterFS or LustreFS for distributed filesystem, Hadoop (plus Hive or Pig or both) for the Map/Reduce weenies, JIRA or Redmine or Trac for ticket tracking (Redmine also provides a wiki, git management, and other nice features), and sbopkg standard with installation (and an on-site mirror of all the SlackBuild packages).

Finally, it needs documentation oriented toward the system administrators who live in the enterprise world, where "Linux" means either RHEL or CentOS, and there's an official process document for everything (and if there isn't, a task doesn't get done until there is).

I've often thought I'd like to make a Slackware framework package (not so much a distribution fork as an overlay, so anyone could trivially apply it to any Slackware release) that gave Slackware more of the capabilities of Turbo Linux and Oracle Linux, for the datacenter. Enterprise is less about "we need a server" than it is about "we need an Oracle Exalogic cluster". But then all my unfinished projects grow hands, point fingers at me, and laugh.

2 members found this post helpful.