I hope that this has not been fully explored in a previous thread but I could not find one that addressed this.

I have a small ISP setup based on CentOS that has been in evolution since the late 1990's.

I want to replace all of the current servers with Raspberry Pi4s since the actual volume of transactions is very small and I want to get a better way to emulate a cloud with cheap replaceable servers. I am also old enough to know that you can easily run a whole company on a Intel 486 with 500 kb of RAM running SCO Unix so a 4 GB ARM board looks like it can do some pretty heavy lifting if it has some siblings helping out.
I am thinking that a Bitscope 20 CPU array will more that satisfy my CPU and memory requirements even if it is only half filled. I will probably keep my current RAID fileserver until a Raspberry with 16 Gb Ram and 5 SATA channels comes along.

I want to run a front-end firewall/router/DHCPD on one Pi. Single point of failure but easily replaced.
A single HTTP(Apache) load balancer with the actual web servers running under kubernetes behind it. Single point of failure ...) A lot of the web sites are only accessed once a week (lousy marketing on my part).
Other services include SVN, Joumla, Nexus, Postfix. Nothing with 10s of accesses per minute.

In my vision, the rest of the PIs will be used to support microservices (Kubernetes, Docker, Java, Spring, Consol, etc.)
Cloudstack possible but at <$100 for a real CPU is VM worth the effort?

The 4GB Pi4 makes this look very attractive given what I know about the history of computing but it is not impossible that I am missing some key point.

The question is "What OS to chose?".
CentOS seems to be available and familiar to me but I am not sure about the level of support.
Ubuntu seems to be committed to the Pi but I think of it as a Desktop OS.
There are others as well.

https://www.raspberrypi.org/downloads/ says that Raspbian is the official OS. Use that. If Centos were listed on that page, I would have said Centos because it's familiar to you, but I don't see it there.

Ubuntu is a very solid server OS, by the way.

1 members found this post helpful.

I'm partial to Arch linux for my pi3s - I see they have an image for the pi4, and the instructions are mighty similar; exact except for the image name. I don't run desktops, but that new puppy should do the job.
Might have to swap out my firewall/router and do the job properly using 2 decent interfaces ...

I also see mention of Fedora getting close, so Centos may not be far behind.
(later) looks like it's in the works according to the Centos forums and working of sorts on Centos 7.

1 members found this post helpful.

As mentioned, Raspbian is the officially supported OS. That's basically Debian. However, I have significant concerns about the long term viability of any distro infected with systemd, so you might look at Devuan instead because that's basically Debian but without the systemd.

Or sometime soon, FreeBSD will support the Raspberry Pi 4. That has a long support cycle too. Right now there as not an image for the 4 yet, just the earlier models.

As for the hardware, will you use the compute modules connected to networked storage? It is probably more cost effective, as well as saving on heat and space.

Thanks for the quick response.

Have you tried Raspian in an environment similar to what I am intending?

I have Raspbian running and it looks like a very solid desktop OS. The graphics is a bit slower that my desktop CentOS on AMD but still runs.

I have not explored all of the Raspbian sysadmin tools but do have Webmin running which I like as a simple remote admin tool.

I am more comfortable with the Red Hat tools but I will try to get Ubuntu running.

It is very nice to just replace an SD card and reboot to switch OS.

I will use networked storage for everything.
With Gigabit Ethernet, it should work well.

I am already most of the way there now with the existing setup and I will keep the same RAID server.
Not having a local hard drive on the Pis will enforce a certain amount of discipline.
It also makes it easier to plan for hardware failures.

Debian and then Ubuntu in smaller production and development environments, both. I have some CentOS machines and keep running into some of the same old problems I had with RH 5.2 which went away with a move up to Debian and then Ubuntu. Even though I was more familiar with RPM, it took me a suprisingly short while to favor APT.

I've used Raspbian a lot in other contexts and where Debian suffices, Raspbian will, too. I'd recommend at least getting familiar with one of them so you can make an informed decision as to whether to stay with CentOS or upgrade to Ubuntu. :P

I'd avoid Webmin. As tempting as it seems at first, it is a lot more work and much more limiting. I'm really lazy and thus gravitate naturally to the shell where I can get the most work done with the least effort. Anyway, as you scale up to 100's or 1000's of machines, you'll need to work with your own scripts or else Chef, Puppet, or Ansible instead. Webmin can't and won't scale. It'd be fine for some of your customers though. But then there is the overhead of constantly explaining what it can and can't do.

1 members found this post helpful.

I currently use a firewall/router/gateway based on an old AMD processor with 1GB Ram running CentOS with 3 Ethernet ports. One goes to a wireless router that servers the internal network, one to the Internet and one to the DMZ. Pretty standard.

It looks like it should be easy to add a USB Ethernet adapter to the Pi4 for the Internet access and use the wireless to connect to the internal network which is already wireless and use the Gigabyte Ethernet to connect to the DMZ.

The Internet access is 30-60 Mbs (cable modem) and I expect that USB should be plenty quick enough to handle that much traffic.

I agree about Webmin and scale.
Ansible looks interesting but I am not at that point at the moment.

Moving applications into the cloud is also a possibility at that point since it is likely that the Internet traffic would be much larger if hundreds of CPUs are required.

I also believe that once I get past this initial state, the bulk of the servers will be identical and will just be pawns in Kubernetes' game so managing their configurations would be restricted to managing IP addresses (some central control - DHCPD using MAC addresses?) and swapping out boards (hardware failure) or SD cards (OS, Kubernetes upgrades).

Not exactly as asked but peek at this idea maybe. https://www.netgate.com/solutions/pfsense/sg-1100.html

Neat little unit.
At first glance it seems that it can do at least 90% of what I want and with a bit of investigation it may be that 100% of my front door requirements are easy to do with it.

It is pretty clever bit of packaging probably done by a team that is more focused on this functionality than I am.

I like firewalld and the ability to detect intruders at the door but perhaps it has a better solution already packaged up into something easy to maintain.

I have a long history in the IT sector and am probably overly biased to solutions that I build but there comes a time when one needs to pick one's battles a bit more strategically.

Thanks for the suggestion.

I am no fan of SystemD, but I cannot second Turbocapitalist's concerns.

I've used several distros with SystemD for a number of years and had no SystemD-related issues whatsoever. These include Mageia, which has incorporated SystemD for years, Ubuntu MATE, and Debian itself.

That I don't like SystemD does not keep me from (reluctantly) admitting that it seems to work. Furthermore, a sysadmin of my acquaintance who maintains a thin client network with about 200 users has told me that his users' computers boot more smoothly with SystemD. His main beef with SystemD, besides its being binary, is that he finds journalctl annoying. (The version of RHEL on his server is, I think, still pre-SystemD, but he will be upgrading them with new hardware and an updated OS soon.)

But, with rsyslog installed, SystemD logging information gets stored in the traditional format in /var/log. At this point, I think most distros are including rsyslog be default.

I have been using systemd for a few years and have learned to like it.
It was a big change but not too hard to get into.

It seems easier to use than init.d once you get used to it.

Journalctl and systemctl status do not seem to be well thought out or tested for usability but one can always find the full logs somehow.

Documentation is weak but that seems to go with Linux and Open Source in general.
Everyone wants to code and the documentation is often written for other developers rather that SysOps (DevOps to the young) and often is more interested in bragging about how well edge cases are handled than the normal use of the utility or feature.

Small rant from an oldtimer!

I think that systemd is here to stay so I am not thinking about that in looking for an OS.

Nope. It's neither thought out nor tested, nor can one usually find the reasons for catastrophic failures in what's left of the logs. I've had it scotch two and half demos, one of which was Very Important. That demo was important enough that I removed the storage medium and spent some time doing read-only forensics and then tested on clones until I found the cause: systemd. Then I arranged one more, the half demo, but even if it had worked it would not make up for the first one. It failed in a new way, but I was prepared and spent a few minutes working around it but that was quite noticeable and derailed the flow. I also object to how it was hammered through Debian, without technical merit, and thus also into all three or four hundred Debian derivatives.

Anyway, more on topic, you can download and try PFSense on other hardware and check out its capabilities. If you have something it can run on, try it:

https://www.pfsense.org/products/#requirements

However if you are rolling your own solution, there are also other boards available with dual gigabit Ethernet. Here is one example of many:

https://beagleboard.org/x15

Not sure if PFSense will run on that, but several distros will.

One thing to watch out for is the RPi4 runs considerably hotter than its predecessors.

P.S. Not all of the RPi4 has software drivers written for it yet, so you may have to wait a while.