Request for community assistance in distro/strata acquisition strategies

The high-level goal for the next release is to make Bedrock Linux easier to try out. There are two broad steps for this:

• *Fully* automate hijacking a given install of some other distro and turning it into Bedrock Linux.
• Provide a utility to *fully* automate acquiring other distro's files for use as strata.

That latter item, sadly, cannot be done in a generalized fashion. We'll need some logic for each distro (or possibly family of related distros) we're interested in. This adds up to a *lot* of time consuming work. Luckily, this work is easily parallelizable across different people! Instead of further delaying the next release waiting for me to read up on a bunch of distros I don't know, or limiting the usefulness of the next release by skipping supporting them, I thought it best to reach out for others for help here. Odds are good ya'll know some distros better than I do.

Here's what I'm looking for:

1. Some way to check if the distro supports the current machine's architecture (e.g. `x86_64`)
   • Presumably compare the supported options against `uname -m`, maybe after mapping it if it's in another format.
2. Some way to list a distro's available releases, if that makes sense for the given distro.
   • If there's a way to filter it down to only currently supported releases, that would be ideal.
   • If the release has a number of names/aliases, all of them would be of value. This way a user can specify the name in any format and we'll grab it.
3. Some way to indicate which release should be considered the default selected one if none is specified, if that makes sense for the given distro.
4. Some way to get a list of supported mirrors.
5. Given a distro, release, and mirror, some way to get the distro's "base" files into a specified directory.
6. Whatever steps are necessary to set up the previously selected mirror for the package manager, if that makes sense for the distro.
7. Whatever steps are necessary to update/upgrade the now on-disk files, in case the above step grabbed files which need updates.
8. Whatever steps are necessary to set up the distro's locales, given the current locale
9. Any tweaks needed to make it work well with Bedrock Linux.

What makes this tricky are some constraints we'll need to use:

• If possible, this should be doable when restricted to the utilities busybox provides.
  • For example, busybox does not provide `debootstrap`, so we can't (directly) use that.
  • Assuming we find some way to acquire `debootstrap`, we'll also need its dependencies. Busybox's implementation of things like `dpkg` is inadequate.
• If the above step is not possible, we want to find a way to acquire the given distro with the minimum amount of additional dependencies.
• - For example, the best strategy I have for acquiring Fedora right now requires an xml parser. I'm considering distributing xml2 as part of Bedrock Linux. However, if we can avoid that and keep Bedrock Linux's disk overhead / distribution size smaller that'd be preferable.
• *Only* use official packages/images/etc from the given distro. A random docker image of some distro *could* make things much easier for us, but it introduces trust issues.
• *Only* use network resources intended for the given distro. If the user request Ubuntu, don't grab Arch's `debootstrap` code as part of the process to acquire Ubuntu. Presumably people who host Arch packages do so for the intent of people who want to run Arch software; using it as a stepping stone for Ubuntu could be considered rude.
• Favor https over unsecured connections. Verifying signatures is out of the scope for the next release (although should be pursued eventually), but I *think* busybox can verify https certificates easily enough.
• Everything should be as future-proof as possible. If you're downloading some web page, such as a list of mirrors, it's best if it's one we can expect not to move or change format in the foreseeable future. Ideally, the effort to maintain this Bedrock Linux utility should be minimized.

Some quick and dirty examples:

Arch Linux:

1. Arch Linux only supports `x86_64`.
2. Rolling release, no need to list releases.
3. Rolling release, no need to determine default release.
4. The official mirrors are available at https://www.archlinux.org/mirrorlist/all/ which can be trivially downloaded and parsed
5. Use a bootstrap tarball provided in the various mirrors to set up an environment for pacstrap, then use pacstrap to acquire the files
   • Given a mirror, we can find an HTML index page at `$MIRROR/iso/latest/` which contains a file in the form `archlinux-bootstrap-<date>-x86_64.tar.gz`. We can download and untar this to some temporary location
   • Add the mirror to the temp location's `/etc/pacman.d/mirrorlist`
   • chroot to the temp location and run `/usr/bin/pacman-key --init && /usr/bin/pacman-key --populate archlinux`.
   • chroot to the temp location and run `pacstrap </path/to/stratum/>`
   • kill the gpg-agent the above steps spawn and remove temp location.
   • chroot to the stratum and run `/usr/bin/pacman-key --init && /usr/bin/pacman-key --populate archlinux`.
   • kill the gpg-agent the above step spawns
6. Add the mirror to the stratum's `/etc/pacman.d/mirrorlist`
7. `pacman -Syu`
8. Append locale to stratum's `/etc/locale.gen` and run `locale-gen`.
9. Comment out `Checkspace` from `/etc/pacman.conf`, as Bedrock Linux's bind mounts confuse it. Include a comment explaining this in case users read that config.

Debian:

1. Parse https://www.debian.org/ports/, map to `uname -m` values, compare against `uname -m`.
2. Given a mirror, look at:
   • The codename and version fields in `<mirror>/dists/oldstable/Release`
   • The codename and version fields in `<mirror>/dists/stable/Release`
   • The codename and version fields in `<mirror>/dists/testing/Release`
   • Unstable/Sid, no version number.
3. Default release is stable from above.
4. Parse https://www.debian.org/mirror/list
5. Use busybox utilities to download the package list and calculate packages needed to run debootstrap. Download those, extract them, then use those to run debootstrap.
   • Download `<mirror>/dists/<release>/main/binary-<arch>/Packages.gz`
   • Parse Packages.gz for debootstrap's dependencies.
     • Packages.gz is a relatively simple format. This is doable, if slow, in busybox shell/awk.
   • wget the dependencies from the mirror and extract them to temp location
     • Busybox can extract .deb files.
   • chroot to temp and debootstrap stratum
6. Add lines to `/etc/apt/sources.list` as needed
7. `apt update && apt upgrade`
8. Install `locales-all`.
9. None needed.

Ubuntu and Devuan will likely be very similar, but they'll need some specifics. Ubuntu won't have oldstable/stable/testing/sid, for example, and they'll both need different mirrors.

Void Linux:

1. Download index page from mirror then look at filenames, compare against `uname -m`.
2. Rolling release, no need to list releases.
3. Rolling release, no need to determine default release.
4. Parse https://wiki.voidlinux.eu/XBPS#Official_Repositories
5. Get static build of xbps package manager from special mirror. Use to bootstrap stratum.
   • Download `xbps-static-latest.<arch>-musl.tar.xz` from https://repo.voidlinux.eu/static/
   • Extract
   • Run `xbps-install -S -R <mirror> -r <stratum-location> base-system`
6. Not needed
7. `xbps-install -Syu`
8. Write locale to stratum's `/etc/default/libc-locales` and run `xbps-reconfigure -f glibc-locales`
9. None needed.

I'm thinking of making `void-musl` a separate "distro" from `void` for the purposes of the UI here, unless someone has a better idea. It'll be almost identical under-the-hood, just it'll look at a slightly different mirror location.

One way to go about researching this is to look for instructions on setting up the distro in a chroot, or to bootstrap the distro. Many distros have documentation like https://wiki.archlinux.org/index.php...existing_Linux or https://wiki.voidlinux.eu/Installing...ide_Arch_Linux.

Don't feel obligated to actually fully script something up for these. Some of that effort may go to waste if someone comes up with another strategy, or if some code could be shared across multiple strata. Just enough for someone else to write up such a script should suffice for now. It would be good if you tried to follow the steps you're describing manually, though, just to make sure they do actually work and you're not missing something.

In addition to coming up with these items for distros I haven't covered and improving strategies for distros we already have, there's value in thinking of other things which could be useful that we might need per distro. Is there anything I'm forgetting which should be added to the per-distro list of information we need?

• Someone once proposed a adding a way to determining the various regions that have mirrors, to make it easier for someone to select appropriate mirror. I ended up dropping that idea as I found it finicky when I experimented with it, but I think it's a good example of something else to be considered here.
• Another possible consideration I dropped was logic to find the distro's init; I think a general init-finding logic could be used without any per-distro code. There's only really two options I've seen as a distro default: `/sbin/init` and `/lib/systemd/systemd`. We can just check both.

I know a *lot* of people have said they would be interested in contributing, but don't know enough low-level Linux nitty-gritty to code something up. This may be a good way to contribute that might be more accessible.