Initial OmniOS impressions by a BSD user
 

I had been using FreeBSD as my main web server OS since 2012 and I liked it so much that I even contributed money and code to it. However, since the FreeBSD guys (and gals) decided to install anti-tech feminism, I have been considering to move away from it for quite some time now.

As my growing needs require stronger hardware, it was finally time to rent a new server. I do not intend to run FreeBSD on it. Although the most obvious choice would be OpenBSD (I run it on another server and it works just fine), I plan to have a couple of databases running on the new machine, and database throughput has never been one of OpenBSD's strong points. This is my chance to give illumos another try. As neither WiFi nor desktop environments are relevant on a no-X11 server, the server-focused OmniOS seemed to fit my needs.

My current (to be phased out) setup on FreeBSD is:

• apache24 with SSL support, running five websites on six domains (both HTTP and HTTPS)
• a (somewhat large) Tiny Tiny RSS installation from git, updated via cronjob
• sbcl running a daily cronjob of my Web-to-RSS parser
• an FTP server where I share stuff with friends
• an IRC bouncer
• MariaDB and PostgreSQL for some of the hosted services

I would not consider anything of that too esoteric for a modern operating system. Since I was not really using anything mod_rewrite-related, I was perfectly ready to replace apache24 by nginx, remembering that the prepackaged apache24 on FreeBSD did not support HTTPS out of the box and I had ended up installing it from the ports. That is the only change in my setup which I am actively planning.

So here's what I noticed.

Preparations:

Before deciding on an operating system (I ruled out DragonFly BSD early - I could not install sbcl there, neither from pkg nor from DPorts), I spend quite some time reading about its motivations, heritage and development. I already feel thirty years older by just having read about general Solaris concepts. I was too young when Solaris blossomed, and two decades of (quite some) Linux and (quite some) BSD have spoiled my understanding of Unix. To say the least, I never had a real UNIX on any of my machines and no system except Windows which was that different from FreeBSD and/or Debian. Never settle, I guess.

The installation:

The most important reason to pick OmniOS over other illumos distributions was that OmniOS officially supports being installed on a system which is controlled over ssh. OpenIndiana seems to require "clicking" on stuff. I don't want that.

Most of OmniOS's "Kayak" installation reminds me a lot of OpenBSD's: I am asked a number of questions and I would probably get a working operating system by just accepting the defaults. Unlike OpenBSD, OmniOS wants numbers instead of "y" or "n" and sometimes asks for confirmation of what I have just entered, but that is not too bad since I don't think I will have to run the installer ever again. Updating OmniOS seems to be done without running this QA thing from the start. I consider this to be a significant advantage over OpenBSD, to be honest. I had to change the networking defaults on the post-installation screen though: As the default setting is "static" instead of "DHCP", I could not reach my server from the outside. Changing that fixed it. (So, yes, I needed to install OmniOS twice. On the plus side, I felt smarter on the second try.) I also like that the other options (adding users, enabling sshd etc.) are just a "Enabled"/"Disabled" switch - no giant blue fullscreen "dialogs" required.

OmniOS informs me that my new user account can be added as a primary admin and to sudo. So sudo is optional on Solaris - good to know. (According to the web, there is pfexec instead. I really prefer sudo for the time being.)

First impressions:

Hooray, a BSD boot loader! Finally an operating system without grub - I made my experiences with that and I don't want to repeat them too often.

It is weird that the installer won't accept "mydomain.org" as a hostname but sendmail complains that "mydomain" is not a valid hostname right from the start, OmniOS sent me into Maintenance Mode to fix that. A good start, right? So the first completely new thing I had to find out on my new shiny toy was how to change the hostname. There is no /etc/rc.conf in it and hostname mydomain.org was only valid for one login session. I found out that the hostname has to be changed in three different files under /etc on Solaris - the third one did not even exist for me. Changing the other two files seems to have solved this problem for me.

Does the network still work?

Uh... I think so, at least.

On to the packages!

Package woes:

OmniOS's pkg is quite different from FreeBSD's: It has a more verbose help screen and its packages have subpackages and something like flavours built-in from what I understand. A good number of core packages comes with OmniOS and I had updates (pkg update - no extra pkg upgrade required) to some of them even after a fresh installation.

The number of officially provided packages is not really overwhelming though: No Python 3, no sbcl, a lot of other things are missing as well. I could probably find some of them in third-party repositories - or I could go the extra mile and install Joyent's pkgsrc on OmniOS which combines the battle-tested pkgsrc package management with commercial-grade support by a company which actually needs it to be good enough for enterprises. Guess what I did?

The instructions from the website gave me a working pkgin, except that I added the PATH and MANPATH exports to my .profile file as well - I wanted them to be available automatically, of course. I managed to get all relevant applications - including nginx and php72-fpm - installed quickly.

Random findings:

~ I was wondering how many resources my (mostly idle) new web server was using - I always thought Solaris was rather fat, but it still felt fast to me.

Ah, right - we're in Unixland and we need to think outside of the box. This table was really helpful: although a number of things are different between OmniOS and SmartOS, I found out that the *stat tools do what top does. I could probably just install top from one of the package managers, but I failed to find a reason to do so. I had 99% idle CPU and RAM - that's all I wanted to know.

~ Trying to set up twtxt informed me that Python 3.6 (from pkgin) expects LANG and LC_ALL to be set. Weird - did FreeBSD do that for me? It's been a while ... at least that was easy to fix.

~ SMF - Solaris's version of init - confuses me. It has "levels" similar to Gentoo's OpenRC, but it mostly shuts up during the boot process. Stuff from pkgsrc, e.g. nginx, comes with a description how to set up the particular service, but I should probably read more about it. What if, one day, I install a package which is not made ready for OmniOS? I'll have to find out how to write SMF scripts. But that should not be my highest priority.

~ The OmniOS documentation talks a lot about "zones" which, if I understand that correctly, mostly equal FreeBSD's "jails". This could be my chance to try to respect a better separation between my various services - if my lazyness won't take over again. (It probably will.)

~ OmniOS's default shell - rather un-unixy - seems to be the bash. Update: I was informed about a mistake here: the default shell is ksh93, there are bogus .bashrc files lying around though.

~ Somewhere in between, my sshd had a hiccup or, at least, logging into it took longer than usual. If that happens again, I should investigate.

Conclusion:

By the time of me writing this, I have a basic web server with an awesome performance and a lot of applications ready to be configured only one click away. The more I play with it, the more I have the feeling that I have missed a lot while wasting my time with FreeBSD. For a system that is said to be "dying", OmniOS feels well-thought and, when equipped with a reasonable package management, comes with everything I need to reproduce my FreeBSD setup without losing functionality.

I'm looking forward to what will happen with it.

I gave up on FreeBSD a long time ago, for the same programs installed, OpenBSD was half that of a FreeBSD installation! :)

Sounds like you've found what works for you, & that is all that matters, thanks for the insight.

:eek: ... sorry didn't get to the end.
I've tried a couple of times to get into non-linux *nix, but the frustration of not knowing where anything is just got the better of me.

This is to remind myself to come back and read the entire initial post sometime. Thanks anyway for taking the time to let us know your thoughts.

FreeBSD is incredibly user-friendly, really. But at a certain point, that does not matter anymore.

YesItsMe,

Thanks for the post. Been running Solaris since the early 90's and we have just decided to stop paying for support, which means no updates or patches. Was going to migrate everybody over to CentOS but I have been getting a major push back from my users.

Since our group has a very large code base of scientific applications this seems like an ideal solution. I'm going to start playing around with in a VM first, can't wait to get started.

Thanks again!

On desktops, OpenIndiana might be your first choice.

I'm glad that I could inspire you!

Thank you for the detailed post, quite inspiring.

Our company decided to do the same, more or less for same reasons.

It is interesting you mention sbcl, as we are into common lisp as well.

Some of our engineers have been running OpenIndiana on their laptops (ThinkPads) with success. Some of our workstations unfortunately do not seem to run well, or allow OpenIndiana to be installed (for example, X399 motherboards with AMD threadripper).

We are still testing OmniOS on servers.

The documentation, the "feel" of the OS, some of the technologies (like Zones) are leaps and bounds superior to FreeBSD, in same way FreeBSD was superior to others.

Please continue writing on illumos/OmniOS, it is inspiring and helpful for all like us wishing to use something better. Thanks again.

By the way, a (almost) recent article on installing X11 and a Desktop Environment on OmniOS

I already like your company. :)

Sadly, nothing new to tell though.

I was surprised to see this thread linked on /r/illumos:

This reply was quite inspiring, and also similar to the one at our company.

Aw, fame!

Not quite un-interesting:
OmniOS forces me to understand how ZFS works by creating a new "/" container on each upgrade. That means: If you don't perform a

at least once, you're screwed. The data's not "gone" then - but you won't have fun restoring it.

FreeBSD does not do that. This seems rather dirty when compared to the clean approach.
So: Another + for OmniOS on this one.

OmniOS r151026 has been released, it has an (optional) more colorful installer and upgrading it took three minutes and a reboot and everything still works. :)

Interesting, thank you.

Could you clarify a bit? You mean that one needs to make a dataset mountpoint explicit at creation time?

Is that somewhere in the wiki? If not, perhaps it should be added.

Thanks for the news.

I read there is a better AMD support. I am looking forward to test it on one of ours X399 motherboards with AMD threadripper.

Unless it is already defined elsewhere (like /home, by default, is per the automount service, but I screwed this one up) or you can live with losing it after an update, then one does not. Otherwise, this is what the OmniOS devs proposed to me on the IRC (unless I misunderstood a lot).

Example: If you use pkgsrc in /opt, you should probably have /opt mounted separately because the next OmniOS boot environment might not have it. (I admit I have not fully understood all of it just yet, but I'm working on that.)

Report back!

Thanks, I will. It may take a bit, though, we are testing LW vs SBCL/CCL.

I am still hoping that some day I will be able to afford a LispWorks license. It seems to be a pretty decent software. SBCL works flawlessly on OmniOS though, I was able to run one of my applications from FreeBSD without code modifications. :)

Nice! HobbyistDV's license is a bit high, but offers good productivity value (ie, producing GUI apps deployable as binaries).

HobbyistDV has no .exe deployment AFAICS - that's why I don't want it. Not all of my users have a spare Lisp runtime on their machine. Yet!

(Or am I mixing up the editions?)

You may be mixing HobbyistDV with Personal Edition.
http://www.lispworks.com/products/features.html

HobbyistDV has a compiler, and it produces GUI binaries.

Holy entity, 1200 US$ per license...

Yes, I was thinking of the Personal Edition, sorry.

I have managed to install OmniOS r151026 on a custom server - AMD Ryzen Threadripper 1950X on X399.

Unlike OpenIndiana, installation went without a glitch.

I was able to install Awesome WM via Joyent pkgsrc:

https://imgur.com/a/sTr41E0

But could not find MATE (nor gnome or kde), like in this image.

Does anybody know how to install MATE? It seems there is no meta-package, at least not on latest pkgsrc-2018Q1.

root@omniosce:~# pkgin install mate
calculating dependencies... done.
nothing to do.

Since OpenIndiana comes with MATE, it might be interesting to find out if their package repositories are public. Might break libraries though...

I have tested OI for a while, but it seems to me they are a bit different. OmniOS is more like FreeBSD or Arch, OpenIndiana more like Linux Mint. I would not like to mix repositories.

I will ask on the list as well.

Thanks.

Sorry that I can't help with that. :(

Your original post appeared in BSDNow, Episode 245. Very nice.

(I came to know via John D Groenveld on OmniOS-discuss)

Regarding not being able to install MATE and other meta-packages via pkgsrc: that is actually a regression, as explained here, which will be fixed in 2018Q2, thanks to Jonathan Perkin at Joyent.

I was pleasantly suprised with how easy and smooth Awesome install went on OmniOS.

Xorg started the first time withouth a glitch. FreeBSD on same platform was not as agreeable.

I am looking forward to test MATE, Gnome, KDE and others on OmniOS.

OmniOS feels like a very promising foundation.

Thank you for the link. It's nice to see that what I am doing is not entirely considered to be a weird thing. :) I would like to see more people considering to, at least, try any illumos. It needs publicity thanks to Oracle.

Also, yes, Jonathan Perkin is really doing great work. He gives me hope for illumos's future.

Somehow I missed this thread when it first came out. I'm moving away from desktop Linux and considering my options in BSD/Solaris land. Even the good Linux distros (Slackware, Gentoo) are starting to wear at me. They seem like a chaotic collection of parts, rather than a cohesive whole, unlike the BSDs and Solaris.

As such, I am currently testing OpenIndiana. So far, I like it.

If one cares about real UNIX, I could not recommend enough giving any illumos OS a serious look and a long try.

If OpenIndiana works on your hardware, give it a longer test trial, I am sure you will like it much.

Once I moved to OmnisOS, I cannot even start to think of going back to FreeBSD, as it seems to me FreeBSD has been slowly drifting towards a "Linux weltanschauung" rather than embracing a more UNIX-esque worldview.

I could never get OI to boot on my ThinkPad (core i5 7th gen), and even verbose booting gave me no help. Got an error about being unable to find the zfs pool on boot media. Tried re-writing the USB image and to various USBs many many times before I gave up.

I would like to give it another try if others are liking it this much.

Before going for OmniOS (desktop installed via pkgsrc) I also tried OpenIndiana.

It was installing fine on my i7 Thinkpad W541, but I was unable to login after install on a HP z620 and a custom made server. I believe it must have BIOS, and not UEFI for now?

Did you ask on the mailing list, or irc?

It is not an illumos thing, as I was able to install OmniOS on all three hardwares.

Of course, having some uncommon hardware going is a different matter, as drivers are ported from FreeBSD, but not as quick as one would love to.

I think there are many that would agree with you, particularly among the OpenBSD folks.

That's really cool, thanks for this info. So, you are running OmniOS as a desktop environment? Which DE or WM are you using?

Yes, I am running OmniOS as desktop, using Mate installed via pkgsrc.

Just recently Andy Fiddaman and OmniOS devs added VirtualBox to the omnios-extra repo, so one could run Windows from there for those unusual needs (VirtualBox is not on illumos' pkgsrc, but it is available on OpenIndiana).

By the way, Andy Fiddaman is one of the many illumos unsung heros.

I had another question regarding this: are the packages you are installing on OmniOS via pkgsrc more current than what is available in OpenIndiana? For example, the latest release of Hipster (from 10/18) has Firefox 52.9, even after updates. But then again, I guess I could just as easily install pkgsrc on OpenIndiana, but I was just wondering if one of the benefits is more current packages.

Still working my way through the particulars of this true UNIX environment, which is new to me. I have plenty of Linux experience (for better or worse!), and some OpenBSD experience, but true UNIX is new to me. I am enjoying the ride so far.

They share the same repositories. However, as illumos pkgsrc has been switched to nightly updates, none of that should be a problem anymore.

I believe pkgsrc's Firefox is at 64.0.

I use OmniOS as IMHO feels better for servers, in that way I can focus only on one illumos version for now.

But if I would have hardware that support OI, I would personally stay on that one, as it is a more mature desktop experience.

Thanks guys. Yes, I don't see a need to switch from OI at this point, I'm just a relentless experimenter. :)

Make sure you check out Time-Slider, a very nice ZFS-based backup system.

For some yet unknown reason, OmniOS stopped finding the (virtual) e1000 adapter a week ago. A few reboots later, it was back again. Did anyone of you have a similar issue once? I hate mysteries like that.

After latest update that botched a server[1], we had to stop using OmniOS. Unfortunately the developers are way too busy to take care of non-paying users. We are back to FreeBSD now, and, surprisingly, Windows Server 2019.

[1]: Panic after boot menu after upgrading to r151032

That's sad!

Yes, indeed, I was quite disheartened for several days. I would not expect much change in the foreseeable future. Such a pity! Illumos still carries a clear superiority on almost every front, but it also shows a fatal flaw from Sun times, greed and self-centred pride. Even Microsoft was less fragile in that respect.

There is not much greed in what Oracle left unburnt, I thought...

The boot sequence capture you posted shows "skipping system dump : no dump device configured". Enabling it would be the first thing to do in your case.

Without crashdump files to analyze but just a stacktrace from a screenshot, there is probably no way for anyone to successfully investigate the issue.

There's no greed at play here, just a technical limitation. With just this report, I'm almost sure you wouldn't have had more help should you had paid for support, unless the support had the very same hardware and was able to reproduce the problem.

Sorry about that, I did not mean to indicate people involved there were in any way greedy. I was expressing a distinct but general feeling I acquired after almost 2 years interacting with illumos/joyent environment. A feeling I did not experience with FreeBSD or Linux community.

It is only normal for developers of that calibre (like some of illumos/joyent's which I esteem real geniuses) to give priority to issues other than dealing with newbies like myself, well-meaning but not bringing much to the table beside eagerness to learn.

In the end, I was very sorry we had to drop illumos. To me, it is the best OS in many aspects. But without much chance to learn and grow, it is impossible to consider it as an option, unless of course one is already an expert.

But, long live illumos!

Coming back to this after 6 weeks: we did not give up on illumos yet.

In a way, I am grateful OmniOS botched that server upgrade. It forced us to either go back to FreeBSD world or try SmartOS. We are testing SmartOS and it has been quite good. Currently setting up an AMD server with Hetzner. SmartOS gives piece of mind by solving the upgrade problem tout cour.

So, if your server's hardware can use SmartOS, give it a try.

The problem seems to be a bug somewhere in OmniOS itself, finally having been confirmed in a debugging session with a friendly stranger from the #OmniOS IRC channel. Basically, if you change the (virtual?) network adapter after installing OmniOS, the old adapter will remain set up for the DHCP route (as seen in ipadm). You'll need to disable it manually and add /dhcp for the new adapter.

Confusing, I admit.