[SOLVED] Slackware 14.2, unprivileged lxc container as root--fail!

Uncle Lumpy · 08-03-2017, 07:26 AM

Hi all,

I have something weird happening and I'm not sure what additional steps to take to diagnose this.

I have 3 Slackware machines running 14.2 with the latest 4.4.79 kernel. On 2 of the machines, I'm successfully able to create, start and use (as root) unprivileged lxc containers. However, on the third machine (configuration nearly identical to the other two), the container fails to start. I've tripled checked the lxc configuration files--identical to what I did on the other two machines. I got so frustrated yesterday that I blew away the installation and re-installed 14.2 (installing all packages just to be safe)...still no luck at starting an unprivileged container.

The only "Slackware differences" between this machine and the two that work is that this one is running samba and serves as an nfs server. I don't think either of those things should impact starting an unprivileged lxc container.

The first error message that shows up is "Failed to mount none onto /var/lib/rootfs-lxc//proc" and at that point things cascade and the machine fails to start.

I've attached the logfile (using DEBUG mode) from the latest attempt. Any suggestion what to try would be appreciated!

Lumpy

Richard Cranium · 08-03-2017, 11:53 PM

/var/lib/rootfs-lxc//proc

The // bit seems odd.

franzen · 08-04-2017, 02:51 AM

Quote:

Originally Posted by Uncle Lumpy

Code:

lxc_utils - utils.c:safe_mount:1692 - Operation not permitted - Failed to mount none onto /var/lib/rootfs-lxc//proc

Looks like a permission issue.
How do you mount things in the lxc-config?
How are the permission bits set on the "real" host?
Maybe something needs to be set to the unpriviledged uid 100000, to be able to be mounted by the container on startup.

Just a guess, maybe it helps, and it's also bit more secure: for proc and sys, try

Code:

lxc.mount.auto = proc:mixed sys:ro cgroup-full:mixed

Johannes

Uncle Lumpy · 08-04-2017, 04:55 AM

Quote:

Originally Posted by Richard Cranium

/var/lib/rootfs-lxc//proc

The // bit seems odd.

I agree...I don't understand the syntax of the error message.

Lumpy

Uncle Lumpy · 08-04-2017, 04:57 AM

Quote:

Originally Posted by franzen

Looks like a permission issue.
How do you mount things in the lxc-config?
How are the permission bits set on the "real" host?
Maybe something needs to be set to the unpriviledged uid 100000, to be able to be mounted by the container on startup.

Just a guess, maybe it helps, and it's also bit more secure: for proc and sys, try

Code:

lxc.mount.auto = proc:mixed sys:ro cgroup-full:mixed

Johannes

Thanks, Johannes. I'll give this a try when I get home this evening. At one point I tried an "lxc.mount.auto" command similar to that in the config but it didn't seem to change anything. I'll have to think about your other questions...since root is trying to run the container in the standard Slackware location (/var/lib/lxc) I just assumed that all the permissions were OK (especially since it worked out of the box in the other two Slackware machines).

Lumpy

Uncle Lumpy · 08-04-2017, 03:33 PM

OK, I tried the lxc.mount.auto command in the test1 config file--slightly different form of the error message but it still failed to mount proc in the container. I've attached the latest test1.log file as well as the config file used (test1.config).

Best,
Lumpy

Uncle Lumpy · 08-04-2017, 03:37 PM

Hmmmmm, it didn't upload both files. Here's the test1.config file...with a new "log" suffix so that it can be uploaded...

Lumpy

franzen · 08-08-2017, 07:41 AM

Quote:

Originally Posted by Uncle Lumpy

Thanks, Johannes. I'll give this a try when I get home this evening. At one point I tried an "lxc.mount.auto" command similar to that in the config but it didn't seem to change anything.

Please try with commented line

Code:

# lxc.mount = /var/lib/lxc/test1/rootfs/etc/fstab

What is the Output of

Code:

ls -la /var/lib/lxc/test1/rootfs/

?

Chuck56 · 08-08-2017, 08:08 AM

I recently converted to unprivileged LXC and took some notes in the process.

When adding proc & sys mount commands to lxc-config:

Code:

  # proc & sys, remove from /etc/fstab
  lxc.mount.auto = proc:mixed sys:ro cgroup

Don't forget to modify the container's fstab and comment out (or remove) the following lines:

Code:

  #none /var/lib/lxc/guest6/rootfs/proc    proc   defaults 0 0
  #none /var/lib/lxc/guest6/rootfs/sys     sysfs  defaults 0 0

The other significant change I implemented was to stop using cgmanager (deprecated) and start using libcgroup services. I modified rc.cgmanager to non-executable, modified rc.config to executable & added the code below to my rc.local. Chris Willing's guide to unprivileged LXC was a great reference for me.

Code:

# LXC: http://www.darlo.tv/lxc/setup-unpriv-slackware.html
if [ -x /etc/rc.d/rc.cgconfig -a -x /etc/rc.d/rc.cgred -a -d /sys/fs/cgroup ]; then
  echo "Starting libcgroup services"
  /etc/rc.d/rc.cgconfig start
  /etc/rc.d/rc.cgred start
fi

Hope that helps!

Uncle Lumpy · 08-08-2017, 06:11 PM

Quote:

Originally Posted by franzen

Please try with commented line

Code:

# lxc.mount = /var/lib/lxc/test1/rootfs/etc/fstab

What is the Output of

Code:

ls -la /var/lib/lxc/test1/rootfs/

?

OK, I've attached two files (I hope). test1a.log is the error from running the command with the line you suggested commented out in the config. ls.log shows the output of the ls command...

I don't understand why this worked "out of the box" on two of my Slackware systems but fails on this one.

Thanks for your efforts!
Lumpy

Uncle Lumpy · 08-08-2017, 06:14 PM

Quote:

Originally Posted by Chuck56

I recently converted to unprivileged LXC and took some notes in the process.

When adding proc & sys mount commands to lxc-config:

Code:

  # proc & sys, remove from /etc/fstab
  lxc.mount.auto = proc:mixed sys:ro cgroup

Don't forget to modify the container's fstab and comment out (or remove) the following lines:

Code:

  #none /var/lib/lxc/guest6/rootfs/proc    proc   defaults 0 0
  #none /var/lib/lxc/guest6/rootfs/sys     sysfs  defaults 0 0

The other significant change I implemented was to stop using cgmanager (deprecated) and start using libcgroup services. I modified rc.cgmanager to non-executable, modified rc.config to executable & added the code below to my rc.local. Chris Willing's guide to unprivileged LXC was a great reference for me.

Code:

# LXC: http://www.darlo.tv/lxc/setup-unpriv-slackware.html
if [ -x /etc/rc.d/rc.cgconfig -a -x /etc/rc.d/rc.cgred -a -d /sys/fs/cgroup ]; then
  echo "Starting libcgroup services"
  /etc/rc.d/rc.cgconfig start
  /etc/rc.d/rc.cgred start
fi

Hope that helps!

Well, it didn't seem to have much (any) affect. Here is the output from the error log (test1b.log)...

Thanks for your thoughts! I will implement your "cgmanager to libcgroup" conversion on my 2 working machines!

Best,
Lumpy

Chuck56 · 08-09-2017, 07:05 AM

Quote:

Originally Posted by Uncle Lumpy

Well, it didn't seem to have much (any) affect. Here is the output from the error log (test1b.log)...

Any chance we could see a copy of the offending config file? The startup log file indicates trouble with a mount command.

Uncle Lumpy · 08-09-2017, 06:19 PM

Quote:

Originally Posted by Chuck56

Any chance we could see a copy of the offending config file? The startup log file indicates trouble with a mount command.

Chuck, I've attached the config file. This one has "lxc.mount = /var/lib/lxc/test1/rootfs/etc/fstab" commented out. I just tried running the container without the line being commented out but the error messages look the same to me whether that line is being read or not being read.

Thanks for looking at this! I'm about ready to throw in the towel as what's happening just doesn't make sense to me.

Best,
Kel

chris.willing · 08-09-2017, 07:12 PM

Looking at your config.txt, I see an entry "lxc.include = /usr/share/lxc/config/slackware.userns.conf" which I don't have in any of my containers - not sure if that's significant but that's the first thing I'd try removing.

In your log files, the first thing that looks wrong in both cases is the mounting of the rootfs at /var/lib/rootfs-lxc. The full message there is

Code:

lxc_conf - conf.c:setup_rootfs:1215 - mounted '/var/lib/lxc/test1/rootfs' on '/var/lib/rootfs-lxc'

so looks like something to do with lxc.conf. Looking at man lxc.conf, the files in which something could be wrongly set are /etc/lxc/default.conf (or ~/.config/lxc/default.conf). It also mentions /etc/lxc/lxc.conf (or ~/.config/lxc/lxc.conf). In particular, check /etc/lxc/lxc.conf. I don't actually have one, just /etc/lxc/lxc.conf.sample which I presume isn't used. However it contains 'lxcpath = /var/lib/lxc' so I would worry that this may be incorrectly set somewhere to /var/lib instead of /var/lib/lxc.

chris

Uncle Lumpy · 08-13-2017, 12:31 PM

Quote:

Originally Posted by chris.willing

Looking at your config.txt, I see an entry "lxc.include = /usr/share/lxc/config/slackware.userns.conf" which I don't have in any of my containers - not sure if that's significant but that's the first thing I'd try removing.

In your log files, the first thing that looks wrong in both cases is the mounting of the rootfs at /var/lib/rootfs-lxc. The full message there is

Code:

lxc_conf - conf.c:setup_rootfs:1215 - mounted '/var/lib/lxc/test1/rootfs' on '/var/lib/rootfs-lxc'

so looks like something to do with lxc.conf. Looking at man lxc.conf, the files in which something could be wrongly set are /etc/lxc/default.conf (or ~/.config/lxc/default.conf). It also mentions /etc/lxc/lxc.conf (or ~/.config/lxc/lxc.conf). In particular, check /etc/lxc/lxc.conf. I don't actually have one, just /etc/lxc/lxc.conf.sample which I presume isn't used. However it contains 'lxcpath = /var/lib/lxc' so I would worry that this may be incorrectly set somewhere to /var/lib instead of /var/lib/lxc.

chris

Thanks for the suggestions, Chris. The "lxc.include = /usr/share/lxc/config/slackware.userns.conf" came from a couple of online sources for creating unprivileged root containers. Since it works (or at least doesn't prevent the successful creation of unprivileged root containers) in two other Slackware machines I'm running I don't think this is the problem. I commented it out and had the same error.

I looked at the various lxc configuration files across the three machines...they are identical. Two machines work fine, nothing unprivileged container related works on the third. Could the problem somehow be hardware related?

At this point, I feel like marking this thread [Abandoned] rahter than solved. I've wasted way too much of my (and everyone else's) time chasing this--I'm moving on.

Thanks for everyone's help,
Lumpy