[SOLVED] Stale NFS handle, can't restart service. Maybe a different issue?

sledwrecker · 08-21-2013, 11:13 AM

I'm an IT manager who just got tossed into the deep end of the pool in regards to Linux. Purely windows background so it's sink or swim for me.

I have an engineer trying to run "gedit" on RH5.8 I know what you are thinking but we're not upgrading yet.

Depending on which machine he is ssh'd into he can run gedit just fine. However, on one of our blades when he tries to run it nothing happens.

I've been able to see the following:
Aug 19 09:54:59 bla02 sshd[6124]: Authentication refused: bad ownership or modes for file /home/user/.ssh/authorized_keys
Aug 19 09:55:04 bla02 sshd[6124]: error: Could not get shadow information for user
Aug 19 09:55:04 bla02 sshd[6124]: Accepted password for werikson from 11.112.13.62 port 11591 ssh2
Aug 19 09:55:09 bla02 gconfd (user-6241): starting (version 2.14.0), pid 6241 user 'user'
Aug 19 09:55:09 bla02 gconfd (user-6241): Resolved address "xml:readonly:/etc/gconf/gconf.xml.mandatory" to a read-only configuration source at position 0
Aug 19 09:55:09 bla02 gconfd (user-6241): Resolved address "xml:readwrite:/home/werikson/.gconf" to a writable configuration source at position 1
Aug 19 09:55:09 bla02 gconfd (user-6241): Resolved address "xml:readonly:/etc/gconf/gconf.xml.defaults" to a read-only configuration source at position 2
Aug 19 09:55:39 bla02 gconfd (user-6241): Failed to close gconfd logfile; data may not have been properly saved (Stale NFS file handle)
Aug 19 09:55:39 bla02 gconfd (user-6241): GConf server is not in use, shutting down.
Aug 19 09:55:39 bla02 gconfd (user-6241): Exiting

When I try to restart the NFS service on bla02 it will stop all but the NFS services:
Shutting down NFS mountd: [ OK ]
Shutting down NFS daemon: [ OK ]
Shutting down NFS quotas: [ OK ]
Shutting down NFS services: [FAILED]
I am at root level but there are multiple users using this machine. My engineer reported this was working until a couple of days ago. Any ideas or direction?

I should also add that I fixed the authentication problem on the first couple lines. Now it's just the bad NFS handle that needs to be resolved.. I think.. :-)

Linux MR · 08-21-2013, 05:19 PM

There are quite a few unknowns at this moment
-does some person/user currently using the share?
-does the system need to be recycled?
-what has changed from then to now?

Most of our stale handles happen when a resource is abruptly removed while still in use. We normally have to bounce the box or stop the networking somehow.

Hope that helps...

unSpawn · 08-21-2013, 05:46 PM

Any chance of using 'lsof' to see which resources are in use?

sledwrecker · 08-22-2013, 09:58 AM

Quote:

Originally Posted by Linux MR

There are quite a few unknowns at this moment
-does some person/user currently using the share?
-does the system need to be recycled?
-what has changed from then to now?

Most of our stale handles happen when a resource is abruptly removed while still in use. We normally have to bounce the box or stop the networking somehow.

Hope that helps...

-Currently the user is not using the share. He has been directed to use another server that gedit work on. On a side note, I can open gedit from my session without any issues. This definitly appears to be user related not global.
- I cannot recycle (assume you mean reboot) the machine. It is a server in our cluster farm. Rebooting is an absolute last resort with very high level of impact.
- Nothing has really changed. I think he possibly had a failure closing out gedit in his last session and now it's failing to open as a result.

Still stumped.

sledwrecker · 08-22-2013, 10:01 AM

Quote:

Originally Posted by unSpawn

Any chance of using 'lsof' to see which resources are in use?

Thanks for this, I found on redhat I have to launch it from /usr/sbin but it's there. Unfortunetly it does not list anything being used by this particular user. Very useful however I will be using this command a lot!

Linux MR · 08-22-2013, 03:53 PM

-You could always try to refresh the export table, or
-forcibly unmount(i don't think it will go down willingly) and then remount the file handle in question, or
-restart the network (which could have huge impacts)

sledwrecker · 08-22-2013, 03:55 PM

Quote:

Originally Posted by Linux MR

-You could always try to refresh the export table, or
-forcibly unmount(i don't think it will go down willingly) and then remount the file handle in question, or
-restart the network (which could have huge impacts)

- How would I forecebly unmount the file handle?

Linux MR · 08-22-2013, 05:10 PM

For example

nfs on /my/nfs_export

Code:

# umount -f /my/nfs_export
# ###-->COMMENT:now checking to see if it's still mounted
# df -h
# mount -t nfs nfs_ip:/path/to/share /my/nfs_export

Hope that helps...

sledwrecker · 08-26-2013, 10:41 AM

Unfortunetly nothing I've tried has worked. I tried Linux MR's recommendation of unmounting the directory and re-mounting it. That also did not resolve the problem. I'm now waiting for downtime to reboot the blade.

sledwrecker · 08-28-2013, 02:39 PM

Got this resolved, one of my engineers is a linux genius

user@bla02 /home/user: ps -ef | grep user
root 10937 30061 0 14:19 ? 00:00:00 sshd: user [priv]
user 10942 10937 0 14:19 ? 00:00:00 sshd: user@pts/61
user 10943 10942 4 14:19 pts/61 00:00:00 -csh
user 11078 10943 0 14:20 pts/61 00:00:00 ps -ef
user 11079 10943 0 14:20 pts/61 00:00:00 grep user
user 13344 1 0 Jun24 ? 00:00:00 /usr/libexec/gam_server

user@bla02 /home/user: kill -9 13344

user@bla02 /home/user: ps -ef | grep user
root 10937 30061 0 14:19 ? 00:00:00 sshd: user [priv]
user 10942 10937 0 14:19 ? 00:00:00 sshd: user@pts/61
user 10943 10942 2 14:19 pts/61 00:00:00 -csh
user 11082 10943 0 14:20 pts/61 00:00:00 ps -ef
user 11083 10943 0 14:20 pts/61 00:00:00 grep user
user@bla02 /home/user: gedit

gedit now starts !