RHEL4 & NFS caching
 

Yes, I know, RHEL4 is ancient and unsupported. Still, it's what I have to work with. :)

I have a RHEL4 machine running the most recent 2.6.9 kernel. It mounts a NFS3 filesystem, and runs an application from this share. The application reads a number of large data files, each about 1Gb in size.

After a reboot, the machine runs as anyone would expect. My application reads in the required files and processes them in a timely fashion. However, randomly, after days, or weeks, the performance goes through the floor.

When the problem occurs ( and before hand ) I can observe that the kernel has plenty of stuff in its cache, with the cache usually at least twice the size of the data being read from NFS. However iostat shows that NFS is constantly being hit for *small* reads while the application is running. To me, it appears the kernel did not cache the entire input data file.

While my program was running, I verified that it had not cached the entire input through timing a read of the entire data set to /dev/null. First read takes a long time ( as not all the data is cached ). Subsequent reads are quick, as expected as all the data is then fully cached.

For whatever reason, once the problem occurs, it appears that this old kernel is flushing, or perhaps not caching portions of the input data files from NFS. Again, a reboot will restore performance, and restore proper caching of the input data files.

I note more recent kernels ( and NFS client/server versions ) have additional options for controlling caching. There appears to be limited room to move with RHEL4 though.

Can anyone make any suggestions, or comment on specific bugs that may be causing this behavior?

What nfs mounting options are you using? Can you post the relevant line from your fstab file?

I know some applications that users I support require the nfs mount to be mounted with the nolock option.

That said, the nolock option is to fix an issue that arises when using an nfs mount from a NAS appliance that only supports up to NFSv3

Sorry for the delay in getting back with the information.

The NFS share has the following options: rw,hard,intr,rsize=32768,wsize=32768,addr=x.x.x.x

From the NFS documentation on the machine ( and my limited understanding ), none of those options appear to impact caching of NFS, but please, am open to any and all suggestions :)