[SOLVED] fstab doesn't mount /tmp on local partition, but mount -a does... (CentOS7, EBS volume on AWS)

applebag · 12-31-2015, 10:17 AM

I'm trying to mount /tmp to a dedicated partition, but fstab won't do it. I have to "mount -a" after boot and that works perfectly. I've tried blowing away the partition, reformatted with xfs, commented out the /var/tmp bind mount, etc... No luck. I'm new to Linux, but have read a great deal about this topic. Can anyone assist?

Thanks!

/etc/fstab

Code:

UUID=0f790447-ebef-4ca0-b229-d0aa1985d57f / xfs defaults 1 1
UUID=df00ed07-4a22-4015-a76a-3b5af575bd1c /home ext4 defaults,nodev 0 0
UUID=101138e8-3947-4577-832e-fdc5d4ceed08 /var ext4 defaults 0 0
UUID=d48fee84-72cf-4fa1-b0ba-df1430e9d4ef /var/log ext4 defaults 0 0
UUID=59d5036d-2960-41c9-9f8d-163282ab1512 /var/log/audit ext4 defaults 0 0
UUID=8361e31a-525a-4ce5-8360-e0ebfd0a7721 /tmp ext4 defaults,nodev,nosuid 0 0
/tmp /var/tmp none bind 0 0
none /dev/shm tmpfs defaults,nosuid,noexec,nodev 0 0

blkid

Code:

/dev/xvda1: UUID="0f790447-ebef-4ca0-b229-d0aa1985d57f" TYPE="xfs"
/dev/xvdf1: UUID="8361e31a-525a-4ce5-8360-e0ebfd0a7721" TYPE="ext4"
/dev/xvde1: UUID="59d5036d-2960-41c9-9f8d-163282ab1512" TYPE="ext4"
/dev/xvdd1: UUID="d48fee84-72cf-4fa1-b0ba-df1430e9d4ef" TYPE="ext4"
/dev/xvdc1: UUID="101138e8-3947-4577-832e-fdc5d4ceed08" TYPE="ext4"
/dev/xvdb1: UUID="df00ed07-4a22-4015-a76a-3b5af575bd1c" TYPE="ext4"

df -h

Code:

Filesystem      Size  Used Avail Use% Mounted on
/dev/xvda1       10G  1.2G  8.9G  12% /
devtmpfs        478M     0  478M   0% /dev
tmpfs           496M     0  496M   0% /dev/shm
tmpfs           496M   13M  484M   3% /run
tmpfs           496M     0  496M   0% /sys/fs/cgroup
/dev/xvdc1      5.8G  113M  5.4G   3% /var
/dev/xvdd1      991M  4.1M  920M   1% /var/log
/dev/xvdb1      5.8G   25M  5.5G   1% /home
/dev/xvde1      991M  3.3M  921M   1% /var/log/audit
tmpfs           100M     0  100M   0% /run/user/1001

mount -l

Code:

sysfs on /sys type sysfs (rw,nosuid,nodev,noexec,relatime,seclabel)
proc on /proc type proc (rw,nosuid,nodev,noexec,relatime)
devtmpfs on /dev type devtmpfs (rw,nosuid,seclabel,size=489136k,nr_inodes=122284,mode=755)
securityfs on /sys/kernel/security type securityfs (rw,nosuid,nodev,noexec,relatime)
tmpfs on /dev/shm type tmpfs (rw,nosuid,nodev,noexec,relatime,seclabel)
devpts on /dev/pts type devpts (rw,nosuid,noexec,relatime,seclabel,gid=5,mode=620,ptmxmode=000)
tmpfs on /run type tmpfs (rw,nosuid,nodev,seclabel,mode=755)
tmpfs on /sys/fs/cgroup type tmpfs (ro,nosuid,nodev,noexec,seclabel,mode=755)
cgroup on /sys/fs/cgroup/systemd type cgroup (rw,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd)
pstore on /sys/fs/pstore type pstore (rw,nosuid,nodev,noexec,relatime)
cgroup on /sys/fs/cgroup/cpu,cpuacct type cgroup (rw,nosuid,nodev,noexec,relatime,cpuacct,cpu)
cgroup on /sys/fs/cgroup/perf_event type cgroup (rw,nosuid,nodev,noexec,relatime,perf_event)
cgroup on /sys/fs/cgroup/memory type cgroup (rw,nosuid,nodev,noexec,relatime,memory)
cgroup on /sys/fs/cgroup/net_cls type cgroup (rw,nosuid,nodev,noexec,relatime,net_cls)
cgroup on /sys/fs/cgroup/hugetlb type cgroup (rw,nosuid,nodev,noexec,relatime,hugetlb)
cgroup on /sys/fs/cgroup/freezer type cgroup (rw,nosuid,nodev,noexec,relatime,freezer)
cgroup on /sys/fs/cgroup/cpuset type cgroup (rw,nosuid,nodev,noexec,relatime,cpuset)
cgroup on /sys/fs/cgroup/blkio type cgroup (rw,nosuid,nodev,noexec,relatime,blkio)
cgroup on /sys/fs/cgroup/devices type cgroup (rw,nosuid,nodev,noexec,relatime,devices)
configfs on /sys/kernel/config type configfs (rw,relatime)
/dev/xvda1 on / type xfs (rw,relatime,seclabel,attr2,inode64,noquota)
selinuxfs on /sys/fs/selinux type selinuxfs (rw,relatime)
debugfs on /sys/kernel/debug type debugfs (rw,relatime)
hugetlbfs on /dev/hugepages type hugetlbfs (rw,relatime,seclabel)
nfsd on /proc/fs/nfsd type nfsd (rw,relatime)
mqueue on /dev/mqueue type mqueue (rw,relatime,seclabel)
systemd-1 on /proc/sys/fs/binfmt_misc type autofs (rw,relatime,fd=31,pgrp=1,timeout=300,minproto=5,maxproto=5,direct)
/dev/xvdc1 on /var type ext4 (rw,relatime,seclabel,data=ordered)
/dev/xvdd1 on /var/log type ext4 (rw,relatime,seclabel,data=ordered)
sunrpc on /var/lib/nfs/rpc_pipefs type rpc_pipefs (rw,relatime)
/dev/xvdb1 on /home type ext4 (rw,nodev,relatime,seclabel,data=ordered)
/dev/xvde1 on /var/log/audit type ext4 (rw,relatime,seclabel,data=ordered)
tmpfs on /run/user/1001 type tmpfs (rw,nosuid,nodev,relatime,seclabel,size=101548k,mode=700,uid=1001,gid=1001)

fdisk -l

Code:

Disk /dev/xvda: 10.7 GB, 10737418240 bytes, 20971520 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk label type: dos
Disk identifier: 0x000aec37

    Device Boot      Start         End      Blocks   Id  System
/dev/xvda1   *        2048    20964824    10481388+  83  Linux

Disk /dev/xvdf: 1073 MB, 1073741824 bytes, 2097152 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk label type: dos
Disk identifier: 0x39e97852

    Device Boot      Start         End      Blocks   Id  System
/dev/xvdf1            4096     2097151     1046528   83  Linux

Disk /dev/xvde: 1073 MB, 1073741824 bytes, 2097152 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk label type: dos
Disk identifier: 0x7fac483e

    Device Boot      Start         End      Blocks   Id  System
/dev/xvde1            2048     2097151     1047552   83  Linux

Disk /dev/xvdd: 1073 MB, 1073741824 bytes, 2097152 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk label type: dos
Disk identifier: 0x7679bb73

    Device Boot      Start         End      Blocks   Id  System
/dev/xvdd1            2048     2097151     1047552   83  Linux

Disk /dev/xvdc: 6442 MB, 6442450944 bytes, 12582912 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk label type: dos
Disk identifier: 0x44ed0ac1

    Device Boot      Start         End      Blocks   Id  System
/dev/xvdc1            2048    12582911     6290432   83  Linux

Disk /dev/xvdb: 6442 MB, 6442450944 bytes, 12582912 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk label type: dos
Disk identifier: 0x550d3380

    Device Boot      Start         End      Blocks   Id  System
/dev/xvdb1            2048    12582911     6290432   83  Linux

Gsidious · 12-31-2015, 01:37 PM

I can't say that I can see anything wrong with your /etc/fstab file or configuration, looks like it should work as it is. . .
You might try changing the /etc/fstab entry for /tmp and use the /dev/xvdf1 instead of the UUID and see if that works, shouldn't make a difference, but just might.

Code:

 /dev/xvdf1 /tmp ext4 defaults,nodev,nosuid 0 0

Make sure /dev/xvdf1 is still formatted with ext4.

applebag · 12-31-2015, 01:48 PM

Thanks, Gsidious. I've tried removing the bind mount entry with no luck. I've used the /dev/xvdf1 name instead of UUID with no luck. I've blown away the xvdf1 partition and recreated it, making sure fstab matches everything.

The only thing I can think of is that CentOS, by default, wants to put /tmp on the same volume as /. I might have to carve off a little space from the / partition and see if /tmp minds being mounted there.

There must be some issue with the order in which things are being enabled or executed since fstab won't mount it, but mount -a will. I'm puzzled and have a big shout out for anyone who can resolve this.

Emerson · 12-31-2015, 01:51 PM

Try auto option.

applebag · 12-31-2015, 01:57 PM

I've tried with auto and that didn't work after a reboot.

Emerson · 12-31-2015, 02:42 PM

Probably has something to do with systemd, can't help, I do not use systemd.

jpollard · 01-01-2016, 07:56 AM

One thing you might have to do is disable the tmp.mount entry (/lib/systemd/system/tmp.mount) which always mounts /tmp as tmpfs.

After that, the bind mount should be OK.

DarrenDrapkin · 01-01-2016, 08:47 AM

It looks to me as if "mount -a" is mounting /tmp as its own parition, then re-mounting it as /var/tmp. Also you are running tmpfs which stores your temp files in ram.
These lines from /etc/fstab seem to be the guilty parties

Code:

UUID=8361e31a-525a-4ce5-8360-e0ebfd0a7721 /tmp ext4 defaults,nodev,nosuid 0 0
/tmp /var/tmp none bind 0 0
none /dev/shm tmpfs defaults,nosuid,noexec,nodev 0 0

jpollard · 01-01-2016, 08:56 AM

Quote:

Originally Posted by DarrenDrapkin

It looks to me as if "mount -a" is mounting /tmp as its own parition, then re-mounting it as /var/tmp. Also you are running tmpfs which stores your temp files in ram.
These lines from /etc/fstab seem to be the guilty parties

Code:

UUID=8361e31a-525a-4ce5-8360-e0ebfd0a7721 /tmp ext4 defaults,nodev,nosuid 0 0
/tmp /var/tmp none bind 0 0
none /dev/shm tmpfs defaults,nosuid,noexec,nodev 0 0

I believe that is the result of the bind mount, and why I suggested to disable the systemd tmp.mount

applebag · 01-04-2016, 09:38 AM

Quote:

Originally Posted by jpollard

One thing you might have to do is disable the tmp.mount entry (/lib/systemd/system/tmp.mount) which always mounts /tmp as tmpfs.

After that, the bind mount should be OK.

I masked tmp.mount (systemctl mask tmp.mount), rebooted and confirmed it was masked, but my /tmp still isn't mounting on that partition.

Quote:

Originally Posted by DarrenDrapkin

It looks to me as if "mount -a" is mounting /tmp as its own parition, then re-mounting it as /var/tmp. Also you are running tmpfs which stores your temp files in ram.
These lines from /etc/fstab seem to be the guilty parties

Code:

UUID=8361e31a-525a-4ce5-8360-e0ebfd0a7721 /tmp ext4 defaults,nodev,nosuid 0 0
/tmp /var/tmp none bind 0 0
none /dev/shm tmpfs defaults,nosuid,noexec,nodev 0 0

The /var/tmp entry should just be a bind mount - that doesn't destroy any previous mounts, correct? It's more like a sym link, as I understand it.

...

Code:

$ df -T /tmp
Filesystem     Type 1K-blocks    Used Available Use Mounted on
/dev/xvda1     xfs   10471148 1200652   9270496  12% /
$
$ sudo umount /tmp
umount: /tmp: not mounted

It appears that /tmp is off of tmpfs.
It appears that /tmp isn't being mounted anywhere specifically, but is being thrown in on the root.
I'm assuming it's thrown in on / because it's not allowed on tmpfs and needs to go somewhere.
/tmp doesn't get mounted properly with or without the bind mount entry in fstab.

So, my question is... what's keeping it from mounting on the partition I've created?
Thanks in advance for your expertise!

applebag · 01-05-2016, 05:39 AM

Well, I'm not sure why it worked, but I got it to work.

I ran "systemctl mask tmp.mount" and confirmed it was masked after a reboot.
I ran "systemctl unmask tmp.mount" and confirmed it was enabled after a reboot.

For some reason, doing these two things was enough to get /tmp working properly. Thanks for all your help!

jpollard · 01-05-2016, 06:46 AM

There is supposed to be some internal data that systemd maintains from boot to boot (otherwise every boot would require the entire dependency net to be analysed again - and that is a slow process). It is possible that masking/unmasking caused it to recheck what was supposed to be used for /tmp - and this time it found the entry in the fstab file.

The advantage of using a disk based filesystem is that you can apply quota controls on users, as well as have a large space for scratch use without taking away memory for applications. The disadvantage is having to occasionally clean up the filesystem by deleting old files...

applebag · 01-05-2016, 12:10 PM

Quote:

Originally Posted by jpollard

There is supposed to be some internal data that systemd maintains from boot to boot (otherwise every boot would require the entire dependency net to be analysed again - and that is a slow process). It is possible that masking/unmasking caused it to recheck what was supposed to be used for /tmp - and this time it found the entry in the fstab file.

The advantage of using a disk based filesystem is that you can apply quota controls on users, as well as have a large space for scratch use without taking away memory for applications. The disadvantage is having to occasionally clean up the filesystem by deleting old files...

I hear you, man. I see you're in the DC area, so I'm guessing you're familiar with STIGs and CIS benchmarking. I can't say that using a disk for /tmp makes a lot of sense when more memory is readily available with a reboot, but compliance is the name of the game with the folks that sign my paycheck!

jpollard · 01-05-2016, 03:40 PM

Quote:

Originally Posted by applebag

I hear you, man. I see you're in the DC area, so I'm guessing you're familiar with STIGs and CIS benchmarking. I can't say that using a disk for /tmp makes a lot of sense when more memory is readily available with a reboot, but compliance is the name of the game with the folks that sign my paycheck!

It makes sense when any user can suck up 50% of your memory causing an OOM condition.

And you don't seem to realize that some servers cannot reboot just to make "more memory is readily available with a reboot". Rebooting just might lose critical information.

With a disk, you have control. With tmpfs you don't.

tmpfs works just fine with a single user workstation.

applebag · 01-05-2016, 03:52 PM

I have had the luxury of working in a lot of Dev environments that are friendly to reboots. It must suck to be on the other side of that! Counting my blessings. I'll keep the tmpfs points in mind - thanks to all for the tutelage!