I have some Slackware arm 14.2 container running on a banana pi system; and I'd love to be able to maintain them - do builds etc without really "cross compiling" or waiting for the comparatively memory starved pi to do it.

I know could run a full Slackware Arm in a qemu-system-arm instance, and run the containers with an ordinary lxc system there.

I also realize I could copy the container filesystems into a qcow image and run them in qemu-system-arm directly. (I have done this) but it would really great if I could just pop the sd card out of the banana pi system and fire them up directly, with not edits or changes to configs at all.

I was thinking something like having lxc call qemu-user -> init
rather than init directly. I am not sure how to do that. I am little afraid to just set qemu-user up in misc-bifmts and as the arm elf handler and push the button because these are privileged containers. But is that maybe all that is needed?

ARM is another CPU architecture than x86(_64) and what everybody seems to gracious ignore in this forum is that LXC is not a Virtual Machine or System Emulator. It's a way to isolate in their container a series of processes run by the (host) kernel.

How the heck you run ARM binaries as processes in a x86(_64) kernel and CPU? Calling again and again the quemu-user via misc-bifmts, for every little ARM binary?

Maybe this will work, BUT I bet that it will be terrible inefficient for doing it at an entire operating system level.

It will be probably several (if not hundred) times slower than the real ARM hardware. And the memory consumption will be probably EPIC huge, because if I'm not mistaken, each ARM binary executed by a quemu-user instance will load in memory the entire hell of libraries for its own use. As example, for 1000 host processes will mean to have 1000 copies of almost all libraries in memory.

I for one, I believe that your idea is Fantasy - not even Science-Fiction, but true Fantasy, like the ones with flying dragons which breath fire, elves and orcs.

PS. I want a flying dragon too and I will prefer one of a species which is smart like humans and talks.

well running a fully emulated ARM system performs pretty decent on my FX. Having more than 2GB or ram is a big to give the VM vs the banana pi hardware generally make builds go faster than running on that system (SATA SSD on that) but only runs sata 1.5gbps mode so io performance is pretty bad for swap.

I understand its a different CPU-architecture, that is why qemu is needed, and I realize in this use case its fully emulating and not virtualization. Most of these containers are pretty light - they have a single service like apache, samba, my dvb/dvr package, etc in them. So there are not a lot of 'processes' for qemu-user to have to span for the most part. So even loading the full stack of libraries for per-process with no sharing would not be unreasonable for 'testing' things out.

I can see it not working for building maybe stick to the full vm there, but I could still build the packages there and upgrade package with an alternate root to update the container, then spin up the container for testing. Efficient no but I am not talking about some quick tests, not heavy use here.

Also typing this on M1 Mac right now - running the 86_64 build of Seamonkey for MacOs...The implementations are different but the binary translation + thunking technology inst a "fantasy". Heck it wasn't a fantasy more than decade ago when Apple did for PPC binaries on x86.

I did not said that the binary emulation is "fantasy" but the idea of running an entire operating system in binary emulation on each executed program via qemu-user and alike. I explained already why I think this way.

Well I don't really disagree with anything you're saying LuckyCyborg, except the 'entire operating system' part - its not an entire operating system, no kernel etc. Its not a fat-userland its minimal containers. Its the init binary, a handful daemons, and the target process.

In other words really not to-different from chroot + binfmt support. Which is actually the direction I am playing with right now. That might be 'enough' to do must of what I would want to - it would be nice be able to have a full namespace so I could duplicate NIC names, ip addressing, etc and not have to make temporary changes to configs but I am prepared to accept:

the current state of LXC and qemu don't make it easy to run foreign architecture container.

Theoretically, you don't need a virtual image to run a VM. qemu let's you specify any path for the filesystem image and if you specify a block device it will attempt to use that. This requires that qemu recognizes the bootloader.


I haven't really tried but it *should* be possible to run a vm with the OS on the SD card.

1 members found this post helpful.

Pithium, thanks I'll look at that. I guess the issue the OS on the SD card is the host OS for the containers I actually want to run. So I can't just pass qemu /dev/sdX as raw disk image. I'd have to mount it and pass part of the filesystem as you suggest. I don't know the emulation would do work but I'll investigate. Thanks!

In that case you could probably use qemu to virtualize the arm os, which then mounts your SD card and runs lxc. You would give your vm access to the SD card, and basically just have an x86 host running an arm VM, which is the host for an arm container located on the SD card you pulled from your pi.

You'd be taking a deep dive into the matrix here, but if you wanted to do this regularly you could set up a copy of your host OS as a VM in order to streamline the process. Anytime you want to access that container on your x86 PC, you just pop in the SD card and launch qemu with whatever settings you need to streamline it.

This is where I am currently. It works well enough; though it is a matrix-y.

Thanks for the tips and suggestions

Seeing as I'm not doing the work, why not at least try using /dev/mmcblk0pX instead of /dev/sdX as the host OS? It will be slow, but not dreadful.

Running LXC is exactly like running chroot + binfmt support. So, if you already have and ARM rootfs (directly mounted from Banana PI SD card or created on x86-64 host) with /bin/qemu-arm-static setup via binfmt, you can run LXC on this rootfs.

But, if you want to use directly the SD card, maybe you need to change at least /etc/fstab.

If you are using only for development, I think chroot + binfmt support is enough.

I use qemu-user emulation (binfmt + chroot) to compile Slackware packages for platform where I don't have a real hardware or for packages that requires too much memory or time (such as mozilla-*, rust, qt, nodejs, etc.).

The performance, with an high-end x64 CPU (such as Ryzen 7), are comparable with dedicated hardware, with the advantage that you have usually more processors and so the total compilation time is usually shorter than on real hardware.
Also memory consumption is usually not a problem on modern PC, since the required memory for each program is only few MB, with few exceptions (e.g. firefox: on x64 every new session requires about 300MB, while on qemu-user for ARM every new session requires about 800MB).

Another story if you want to run a full emulated system (qemu-system-arm): in this case it is much more slower and maybe not very usable also for development purposes.

As you're using slackware, there's 2 slackware ports to Arm, slarm64 & slackware Aarch64 There are often patches required for arm compilation, but the essential library routines will be available.

What will be wildly different is the boot process. I suggest you direct your attention there. I see getting booted as the challenge.

OMG - Now I feel dumb - I'll have try that when I get home today. I have been beating my head against the wall going down the path of trying to use the native lxc binaries to run the arm init. Never occurred to me run the arm lxc binaries...

1 members found this post helpful.