Files on tmpfs surviving reboot

amirgol · 08-02-2011, 09:23 AM

I had to shutdown my home server (Debian Wheezy x64) for a minor hardware tweak and when I powered it back up one of the daemons failed because its pid file already was in /var/run. I've checked and indeed it was there, and its date was that of the previous reboot. a few other files and directories there also had dates predating the current reboot. But I've mounted /var/run on tmpfs! How could that be?
I've erased the pid file and started the service. As expected, its pid file was created in /var/run. I then powered the computer off, waited a few minutes and powered it back up. Again the daemon failed and the pid file had the same date and time it had before the reboot!

Code:

$ uptime
 17:03:53 up 1 day, 19:00,  1 user,  load average: 0.00, 0.01, 0.05
$ date
Tue Aug  2 17:04:02 IDT 2011

So the last reboot was at at about 22:02 Aug 1st.

Code:

$ mount | grep run
tmpfs on /var/run/lock type tmpfs (rw,noexec,nosuid,nodev,size=5242880,mode=1777,size=5242880,mode=1777)
tmpfs on /var/run/shm type tmpfs (rw,nosuid,nodev,size=20%,mode=1777,size=20%,mode=1777)
tmpfs on /var/run type tmpfs (rw,noexec,nosuid,nodev,relatime,size=400976k,mode=755,size=10M,mode=0755)

Everything under /var/run is on tmpfs.

Code:

$ ls -lt /var/run | tail
drwxr-xr-x 2 root        root        4096 Jul 31 22:03 mdadm
drwxr-xr-x 2 root        root        4096 Jul 31 22:03 sendsigs.omit.d
-rw-r--r-- 1 root        root           5 Jul 31 22:03 rsyslogd.pid
-rw-r--r-- 1 root        root         356 Jul 31 22:03 motd
drwxrwx--- 2 root        nut         4096 Jul 22 12:57 nut
drwxr-xr-x 3 root        root        4096 Jul 15 22:02 vsftpd
drwxr-x--- 2 www-data    www-data    4096 Jul  8 12:40 lighttpd
---------- 1 root        root           0 Jul  8 11:57 crond.reboot
drwxr-xr-x 2 root        root        4096 Jul  8 11:57 sshd
lrwxrwxrwx 1 root        root           9 Jul  8 10:59 lock -> /var/lock

Most files and directories were created right after the computer last booted, but some are older.

How can something like that happen?

MensaWater · 08-02-2011, 10:19 AM

What does "df -h /var/run" show?

amirgol · 08-02-2011, 10:42 AM

tmpfs 1.9G 318M 1.5G 18% /var/run

MensaWater · 08-03-2011, 10:51 AM

I wonder if there were already files in the /var/run subdirectory of the parent filesystem (i.e. /var or /) before you created you tmpfs mount and maybe it is finding those on startup?

Can you "umount /var/run" then do "ls -l /var/run" to see if that is the case? If so deleting the files seen with ls -l then remounting /var/run from the tmpfs might make it behave properly on subsequent boots.

I'm not sure putting /var/run on a tmpfs is a good idea anyway - many init scripts check for files here and clean them up on the way down or expect them to be there on the way up. It may be you've confused the scripts by not having the mount there at critical points. (I've always seen /var/run as simply a subdirectory and not a separate mount.)

amirgol · 08-04-2011, 10:49 AM

I thought that when mounting a partition as a certain directory, you'd only see the files in that partition and not those originally on that directory. Is that not the case?

Anyway, that can't be the case here. As I described earlier, I manually erased the pid file and then restarted the daemon, which created a new pid file - and it was still there even after a power cycle. That's not an old file.

As for mounting /var/run in RAM, that's been suggested on many websites. I've been doing this on an Openfiler server for three years and for two years on an Ubuntu based HTPC with no ill-effect.

MensaWater · 08-04-2011, 12:55 PM

Yes - once mounted you only see files in the filesystem not in the underlying directory. I was thinking perhaps you were having an issue wherein a script was finding a /var/run file from the underlying directory before the tmpfs was mounted on it.

tmpfs should be erased on reboot since it is in memory. However, I found comments suggesting setting up processes to preserve tmpfs files periodically to disk so it may be that is setup in your environment. In fact I found one link talking about an rc file dealing with things such as /var/run but it seemed that was a hack - however it was for Debian based systems. You might want to check your init scripts and see if any of them deal with saving and restoring /var/run files (or tmpfs in general). Also might want to check cron to see if there is any sort of routine there (including anacron if running) that does this.

amirgol · 08-12-2011, 05:16 PM

Hmm. That's probably what's going on - I can't think of any other explanation.

The question now is what is doing that. I couldn't find anything, but it's probably because I wasn't looking at the right place.

MensaWater · 08-15-2011, 10:59 AM

Good places to check:

/etc/init.d/*
(This would show if you have an init script that is recreating the stuff on startup and/or saving it on shutdown.)

Cron for scheduled items:
/etc/cron*/*
crontab -l (as root)

If it were me I'd simply grep for tmpfs and/or /var/run from the above locations e.g.

egrep -i "tmpfs|/var/run" /etc/init.d/*
grep -i "tmpfs|/var/run" /etc/cron*/*
crontab -l |grep -i "tmpfs|/var/run"

Of course the above assumes that there is an entry that is literal about this as opposed to a script that does it called from one of the above locations. The init scripts are likely to have /var/run in multiple files since many things create a /var/run/pid at start and delete it at stop so you might want to check init.d/* for tmpfs first. Also doing an "ls -l /etc/init.d/* |more" will show you a list of scripts and it may be that something there will stick out (e.g. if you saw an /etc/init.d/tmpfs you might want to see what it did.)