NFS "permission denied" after reboot
 

Yesterday, I needed to power down my machines for a few hours. When I turned them back on, my NFS-exported music library was inaccessible. All attempts to mount it (even locally) fail:

# mount -t nfs localhost:/local/music /mnt/hd
mount: localhost:/local/music failed, reason given by server: Permission denied

The same "permission denied" message appears in /var/log/messages on my laptop, and my mom's laptop, whenever automount tries to mount the music library. Much to my frustration, I can find no logged message on the desktop machine to explain why permission was denied. For that matter, I can't even find a log message which acknowledges that there was a denial.

I don't recall having made any configuration changes since last time I rebooted, so I'm baffled as to why I seem to be having permissions problems. I have my home directory NFS-mounted on my desktop machine, (it is exported on my laptop), so I know that NFS is still operational, but somehow it doesn't want to allow anything to be exported.

I'm digging for clues right now. My primary question is:

How can I enable more verbose logging from the NFS system?

All the NFS logging-related pages that Google turns up seem to be related to Solaris, and they don't seem applicable. I notice that there is an empty directory at /var/log/nfsd, which was created by the installation of the nfs-utils-1.0.7-i486-1 package. Is there a way to get some useful logging information dumped there?

Any other information/ideas/suggestions/questions would, of course, be welcomed as well.

$ cat /etc/slackware-version
10.2.0

$ uname -r
2.6.14

|)
|)enji

It's been a while since I used NFS, but could your /etc/exports file have changed? Also, do you have an /etc/xtab file and what does it now contain?

Thanks for the reply! :)

According to the output of:

# grep restart /var/log/messages*

the last restart prior to NFS export failure was about 12:30am on the 10th of December. The modification date of /etc/exports is about 12 hours after that. I 'diff'ed the file against a previous version modified on the 12th of October, however, and the only difference was that I had changed the /music export line to read /local/music - which is quite correct (/music on the desktop machine is just a symlink to /local/music).

I made a similar examination of /etc/hosts.allow - which has a modification timestamp on Christmas Eve - but 'diff'ing the file against my October version showed no differences (I probably tried, and subsequently reverted, some minor change, when the NFS failure started).

I can see the /local/music export listed when I run 'showmount -e hostname' on any of the machines on the LAN, and this produces a "rpc.mountd: export request from hostname" message in /var/log/messages. The export also shows up when I run 'exportfs' on the desktop machine.

I do not have an /etc/xtab file on my desktop machine, or my laptop machine (on which /local/benji is being exported without trouble.

As far as I can see, the NFS settings on my laptop machine and my desktop machine are identical, except for the desktop exporting /local/music, while the laptop exports /local/benji.

I'm quite mystified. :scratch:

|)
|)enji

What user are you logging into NFS as from the client session? If it's root and you're not using the no_root_squash setting you'll get the permission denied error.

I meant to post these links as well. This first link talks about syncing of userids as well as some other (hopefully) useful stuff:
<http://www.faqs.org/docs/Linux-HOWTO/NFS-HOWTO.html#SYMPTOM4>

These are general stuff that you've probably already seen:
<http://www.die.net/doc/linux/man/man5/exports.5.html>
<http://userpages.umbc.edu/~jack/ifsm498d/nfs.html>

It was my understanding that root_squash simply mapped UID and GID to 'nobody', which prevented direct access to files and directories that were not world-readable. Naturally, this has prevented me from accessing certain files and directories as root, but I don't quite follow how this would prevent me from mounting an NFS share, as long as I don't try to access it. It seems to me that this would make it impossible to mount NFS shares at boot time - since the init scripts are run as root. Am I misunderstanding something?

I suppose it's a moot point anyway, since I've mostly tried to mount the share as a normal user (via automount).

Thanks for the links. As you say, I've probably already seen them (either in the last few days, or back when I originally set up NFS), but I'll look through them nonetheless, just to make sure I'm not overlooking something.

The frustrating thing is, I'm starting to have intermittent mounting success. A few hours ago, I logged into my mom's laptop via SSH, and attempted to cd into the shared music directory. The automounter accessed the share without problem, and she than played music on her machine for a couple of hours. After she stopped, the automounter timeout was reached, and the NFS shared was unmounted. Since then, I've been unable to remount the share - either manually, or via automount.

To make matters worse, I'm getting strange messages in /var/log/messages:

automount[3165]: attempting to mount entry /home/tls
automount[5589]: lookup(file): lookup for tls failed
automount[3165]: attempting to mount entry /home/i686
automount[5590]: lookup(file): lookup for i686 failed
automount[3165]: attempting to mount entry /home/librt.so.1
automount[5591]: lookup(file): lookup for librt.so.1 failed
automount[3165]: attempting to mount entry /home/libc.so.6
automount[5592]: lookup(file): lookup for libc.so.6 failed
automount[3165]: attempting to mount entry /home/libpthread.so.0
automount[5593]: lookup(file): lookup for libpthread.so.0 failed
automount[3165]: attempting to mount entry /home/libnss_compat.so.2
automount[5594]: lookup(file): lookup for libnss_compat.so.2 failed
automount[3165]: attempting to mount entry /home/libnsl.so.1
automount[5595]: lookup(file): lookup for libnsl.so.1 failed
automount[3165]: attempting to mount entry /home/libnss_nis.so.2
automount[5596]: lookup(file): lookup for libnss_nis.so.2 failed
automount[3165]: attempting to mount entry /home/libnss_files.so.2
automount[5597]: lookup(file): lookup for libnss_files.so.2 failed

These messages have shown up in the logs on my laptop, my mom's laptop, and my desktop machine. At first glance, they seem to follow the first failed attempt to mount a share after a reboot. I'm starting to worry that I'm running into some obscure NFS or automount bug(s), either in the kernel, or in the user-space tools.

I'll try booting some older kernels, and see whether there is any effect.

No, you are correct. root_squash maps the root user to nobody. Apparently problems can occur when the uid of the nobody user is different on the client and server. A similar problem can occur if the uids of normal user accounts are different on the client and server. Neither of which really explains the output from your automounter.

Do you use the no_subtree_check option? From the man page:
If a subdirectory of a filesystem is exported, but the whole filesystem isn't then whenever a NFS request arrives, the server must check not only that the accessed file is in the appropriate filesystem (which is easy) but also that it is in the exported tree (which is harder). This check is called the subtree_check.
[snip]
As a general guide, a home directory filesystem, which is normally exported at the root and may see lots of file renames, should be exported with subtree checking disabled. A filesystem which is mostly readonly, and at least doesn't see many file renames (e.g. /usr or /var) and for which subdirectories may be exported, should probably be exported with subtree checks enabled.

If it isn't already obvious, I'm guessing wildly while trying not to waste your time... ;-)

I think that's done it!
 

I think I've found/solved the problem. I did a Google search for "intermittent nfs mount failure", and found this page:

http://www.troubleshooters.com/linux/nfs.htm

I noticed, in particular, the following line (which is talking about the syntax of the /etc/exports file):

(Emphasis mine.)

I changed the export line from:

/local/music x.x.x.*(rw,sync)

to:

/local/music x.x.x.x/255.255.255.0(rw,sync)

and ran:

# /etc/rc.d/rc.nfsd restart

and I was immediately able to connect from my other machines! :cool:

Thanks so much for your time! :)

|)
|)enji

No problem - thank you for the URL. I hadn't seen troubleshooters.com before...