What's the correct way to specify a swap file for an lxc container

PeterSteele · 07-15-2015, 01:01 PM

I'm creating an lxc container under CentOS 7.1 using virt-install. Everything works fine, except there is no mechanism provided in virt-install for specifying how much swap the container uses, just memory.

What I thought I'd do is just add a swap file in my container and create the appropriate entry in /etc/fstab to load the swap file when the container starts. My container fstab looks something like this:

Code:

/dev/root / rootfs defaults 0 0
none /dev/shm tmpfs defaults 0 0
/var/tmp/swap  swap  swap  defaults  0 0

Unfortunately the swap file does not get added when the container boots. The reason (I assume) is because the swap file lives under the root fs (/), and this file system isn't ready when the fstab is processed. If I connect to my container immediately after it boots, I can run

swapon -a

and my swap file is then added as expected. I need this to happen automatically though, and if I can't put it in /etc/fstab, what's the correct way to automatically start a swap file when an lxc container starts?

smallpond · 07-15-2015, 01:11 PM

swapping to a file is ugly and doesn't work with all filesystems. Can you provide an lvm or disk partition instead?

PeterSteele · 07-15-2015, 02:59 PM

Unfortunately no. Our containers live in an isolated environment without access to the raw disk devices of the host, just the space that has been given to them for their root fs. We're using ext4 and swap files are working fine. The issue is just how to enable the swap file in an automated manner.

syg00 · 07-15-2015, 09:06 PM

Create the file at the root level (/swapfile) rather than at /var/tmp.

PeterSteele · 07-16-2015, 07:01 AM

That doesn't solve the problem--everything is under one mount point and moving the swapfile to /swap instead of /var/tmp/swap doesn't resolve the root cause that / is not ready by the time the system wants to start the swapfile. The startup sequence looks like this:

Code:

# virsh console vm-02
Connected to domain vm-02
Escape character is ^]
systemd 208 running in system mode. (+PAM +LIBWRAP +AUDIT +SELINUX +IMA +SYSVINIT +LIBCRYPTSETUP +GCRYPT +ACL +XZ)
Detected virtualization 'lxc-libvirt'.

Welcome to CentOS Linux 7 (Core)!

[  OK  ] Reached target Remote File Systems.
[  OK  ] Listening on Delayed Shutdown Socket.
[  OK  ] Listening on /dev/initctl Compatibility Named Pipe.
[  OK  ] Reached target Encrypted Volumes.
[  OK  ] Listening on Journal Socket.
         Mounting POSIX Message Queue File System...
         Starting Journal Service...
[  OK  ] Started Journal Service.
         Mounting Huge Pages File System...
         Mounting Debug File System...
         Mounting FUSE Control File System...
         Starting Show Plymouth Boot Screen...
         Mounting Configuration File System...
[  OK  ] Listening on LVM2 metadata daemon socket.
[  OK  ] Listening on Device-mapper event daemon FIFOs.
         Starting Monitoring of LVM2 mirrors, snapshots etc. ...ress polling...
         Starting Remount Root and Kernel File Systems...
[  OK  ] Created slice Root Slice.
[  OK  ] Created slice User and Session Slice.
[  OK  ] Created slice System Slice.
[  OK  ] Reached target Slices.
[  OK  ] Created slice system-getty.slice.
[  OK  ] Mounted POSIX Message Queue File System.
[  OK  ] Mounted Huge Pages File System.
[  OK  ] Mounted Debug File System.
[  OK  ] Mounted FUSE Control File System.
[  OK  ] Mounted Configuration File System.
         Starting LVM2 metadata daemon...
[  OK  ] Started LVM2 metadata daemon.
[  OK  ] Started Show Plymouth Boot Screen.
[  OK  ] Started Monitoring of LVM2 mirrors, snapshots etc. u...ogress polling.
[  OK  ] Started Remount Root and Kernel File Systems.
         Starting Configure read-only root support...
         Activating swap /swap...
[FAILED] Failed to activate swap /swap.
See 'systemctl status swap.swap' for details.
...

The command it suggests to enter gives me this:

Code:

# systemctl status swap.swap -l
swap.swap - /swap
   Loaded: loaded (/etc/fstab)
   Active: inactive (dead)
     What: /swap

Jul 16 04:50:06 pws systemd[1]: Activating swap /swap...
Jul 16 04:50:06 pws systemd[1]: Failed to activate swap /swap.

This doesn't really give me anything useful but I'm pretty sure the reason it fails is because the root file system isn't ready when the swap file is started. After the boot is complete of course the root file system is clearly up and running and if I run "swapon -a", the swapfile loads as expected. So the question is how can I delay the activation of the swap file that's listed in /etc/fstab until the root file system mount has completed?

PeterSteele · 07-16-2015, 08:44 AM

Based on some reading I've done, I thought I'd be able to add the line

After=local-fs.target

to /usr/lib/systemd/system/swap.target in my container, making the swap target start after the local-fs target starts, but that didn't work--the swap file is still activated too soon, before the local file system is ready. I must be missing something here, I was sure this was the solution...

PeterSteele · 07-20-2015, 07:50 AM

Turns out to be not as simple as I was expecting, but for different reasons. With containers, there is only one kernel. If I do add swap space to a container, e.g., using the swapon command, the swap space actually gets added to the host, not the container. The lesson to learn here is that LXC containers are not VMs, and although they can be treated like a VM in many ways, in some cases they behave quite differently.

syg00 · 07-20-2015, 07:54 AM

Thanks for the postback.

PeterSteele · 07-22-2015, 10:26 AM

In the end the solution was to allocate the swap files themselves in the host, not the containers. Swap space for a container is controlled by the virsh memtune command:

virsh memtune <container> --hard-limit=<mem> --config
virsh memtune <container> --swap-hard-limit=<mem+swap> --config

So, if for example I wanted to give a container called "test" access to 6G of memory plus another 2G of swap, I'd run the commands

virsh memtune test --hard-limit=6G --config
virsh memtune test --swap-hard-limit=8G --config

Note that the swap limit is in fact the sum of the physical memory plus how much swap is wanted. On the host itself, you need to allocate at least 2G of swap space, either using a partition or using swapfiles. You will probably also want to set vm.swappiness to 1 on the host so that the containers will only swap when they absolutely have to.