[SOLVED] Stale NFS handle on embedded system that does not have NFS running

I have a little box running Voyager Linux that monitors my power system. Its disk (16GB CF, actually) is getting full, so I deleted a lot of older files (and some I should *not* have). I was logged in via ssh, and was in the directory containing the files.

Now 'ls' gives a clean listing, but 'ls -al' gives masses of
cannot access xxxxx: stale NFS handle
There do not appear to be any nfs related processes running.

How do I clean up the directory?

Thanks
Jim

Are you in /net by any chance? If so you don't need to clean up anything there and should leave it alone - it is a quick reference to all nfs mounts available whether you've actually mounted them or not.

If you run "cat /etc/mtab" what does it show? Any nfs entries?

Do /etc/fstab or /etc/auto* files have any nfs mount entries (other than /net)?

You can try remounting whatever filesystem is giving the stale NFS mounts then cleanly unmount it. If the source hosts of the nfs mount is not available the only way I know to clear stale NFS mounts would be to reboot the system complaining about them.

Prior to doing a reboot you'd want to be sure you didn't delete any key files needed for boot. (Since you said you removed some files you shouldn't have.)

The files are in a subdirectory of /opt/greenMonitor/Systems. (/dev/hda3)
No trace of NFS. (unless I don't know what to look for)
apache2 is running. Could it be hiding something?

greenpc:/etc# cat /etc/mtab
rootfs / rootfs rw 0 0
none /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0
none /proc proc rw,nosuid,nodev,noexec,relatime 0 0
udev /dev tmpfs rw,relatime,size=10240k,mode=755 0 0
/dev/disk/by-label/ROOT_FS / ext2 ro,noatime 0 0
tmpfs /lib/init/rw tmpfs rw,nosuid,relatime,mode=755 0 0
varrun /var/run tmpfs rw,nosuid,relatime,mode=755 0 0
varlock /var/lock tmpfs rw,nosuid,nodev,noexec,relatime 0 0
usbfs /proc/bus/usb usbfs rw,nosuid,nodev,noexec,relatime 0 0
tmpfs /dev/shm tmpfs rw,nosuid,nodev,relatime 0 0
devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620 0 0
tmpfs /tmp tmpfs rw,nosuid,nodev,relatime 0 0
/dev/hda3 /opt/greenMonitor/Systems ext2 rw,noatime 0 0
tmpfs /var/log tmpfs rw,nosuid,relatime,mode=755 0 0
tmpfs /var/tmp tmpfs rw,nosuid,relatime,mode=755 0 0

greenpc:/etc# cat /etc/fstab
/dev/hda1       /       ext2    defaults,noatime,rw     0       0
proc            /proc   proc    defaults                0       0
tmpfs           /tmp    tmpfs   nosuid,nodev                    0               0
#tmpfs           /rw     tmpfs   defaults,size=32M        0       0
LABEL=SWAP      none    swap    pri=5                   0       0
LABEL=DATA /opt/greenMonitor/Systems ext2 defaults,noatime,rw 0 0

And your /etc/auto* files have what?

When you do the ls -al when it says cannot access "xxxxxx" is that something literal or did you hide the name of a system there? Does it show you the name of the file? Is the file or the subdirectory perhaps a symbolic link? For automatount stuff it often will be.

There are no /etc/auto* files.

xxxxx represents 17-20 apparently random decimal digits, and yes, specific filenames are used.

Here is what 'file' tells me, first for a file I deleted and then one not deleted:

greenpc:/opt/greenMonitor/Systems/mySystem# file gmStatus1000197456500531718gms                                                                   
gmStatus1000197456500531718gms: ERROR: cannot open `gmStatus1000197456500531718gms' (Stale NFS file handle)                                       

greenpc:/opt/greenMonitor/Systems/mySystem# file gmStatus1003656865053168417gms                                                                   
gmStatus1003656865053168417gms: XML  document text

Interestingly what appears to be your PS1 prompt "greenpc:/opt/greenMonitor/Systems/mySystem#" is in the same form as an NFS mount. The commands you're typing are while logged into system, greenpc, itself correct?

I wonder if your system is somehow interpreting the PS1 prompt or other variable you've set with that syntax and thinking it is trying to do an NFS traversal of server greenpc even though you're already on greenpc?

What happens if you type "pwd" in the above directory?

What is the output of:

ls -ld /opt
ls -ld /opt/greenMonitor
ls -ld /opt/greenMonitor/Systems
ls -ld /opt/greenMonitor/mySystem (The # is the end of your PS1 prompt rather than part of the directory name, correct?)

Are the results different if you type:

ls -ld greenpc:/opt
etc...

It is rather curious...
I'm logged in via ssh, and the output of 'pwd -P' is

greenpc:/opt/greenMonitor/Systems/mySystem/Logs/2012/2012-03# pwd -P
/opt/greenMonitor/Systems/mySystem/Logs/2012/2012-03

I stopped all processes that might use files on /dev/hda3, unmounted it, and ran fsck (several times).
The result was tens of thousands of

DATA: Entry 'gmStatus2692987059952628010gms' in /mySystem (105665) has deleted/unused inode 164434.  CLEARED.

It appears that there was one for each stale NFS handle. There were a few other problems fixed, like files being links to directories, and then tens of thousands of inodes with count 2 that should be 1.

After holding the Enter key down for several hours all problems appear to be fixed.

Something in the boot process is mounting /dev/hda3 under /opt/greenMonitor/Systems. While I was working on it I mounted it in a temporary directory, and the only contents were lost+found and mySystem.

Deleting files from lost+found worked without any problems. I suspect a bug in the Compact Flash driver is the cause of the problems. I will check with the manufacturer to see if others have had the problem.

Thanks for your help.

Jim