LinuxQuestions.org
Review your favorite Linux distribution.
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - Networking
User Name
Password
Linux - Networking This forum is for any issue related to networks or networking.
Routing, network cards, OSI, etc. Anything is fair game.

Notices


Reply
  Search this Thread
Old 05-04-2016, 07:54 PM   #1
cwwanner
LQ Newbie
 
Registered: Dec 2015
Posts: 11

Rep: Reputation: Disabled
NFS Server Non Responsive due to time changing


We have an issue with a Linux NFS Server in a volatile embedded environment. The NFS Server becomes becoming unresponsive for one or more NFS Connections. When an NFS connection becomes non-responsive, the NFS Server’s network stack still sends TCP acknowledgements. But the server never sends a NFS Response to the NFS Request. Eventually the TCP Window size goes to zero indicating that the socket is valid but the rpc/nfs application is no longer reading from the socket. We believe the problem is related to the NFS Server’s time.

The NFS Server will be powered up and down. The NFS Server does not have a battery, so time will start at 1970-01-01 00:00:01 at each power cycle. The NFS Server is a black box. The server does not permit us to log into the server. We do not have the ability to change the settings. The NFS Server does provide us the ability to set the time and/or configure the NFS Server’s NTP Client via a proprietary interface. The embedded environment has a laptop that will configure the NFS Server’s time after the NFS Server starts. The laptop is not on a network. So the laptop’s time is not synchronized with a NTP Server. The laptop will configure the NFS Server’s time to laptop’s current time. We discovered that the laptop’s time was approximately 30 to 40 seconds fast. When the laptop sets the time, the NFS Server approximately has 17 NFS Connections. The NFS Server will close all the NFS connections with the NFS Clients. The only exception is to the NFS Client’s connection that eventually becomes non-responsive. The NFS Client’s connection that was not closed out does not respond to a single NFS request. But the NFS Server responds after the NFS Client retransmits the NFS Request. The connection still appears to be functioning normally.

A Linux computer with a GPS Receiver will boot from the NFS Server. The Linux computer will be the NTP Server for the embedded environment. After the Linux computer starts and has a valid time from the GPS Receiver, the Linux computer will configure the NFS Server’s NTP Client to use the NTP server. The NFS Server’s time will jump backwards 30 to 40 seconds to the correct time. When the time jumps backwards, the NFS Server closes out all the NFS Client connections. The NFS Client’s connection that was not closed at the previous jump in time is not closed out again and becomes not responsive permanently.

The NFS Server's configuration: Kernel Version: 2.6.18-238.9.1, 64 Bit OS.

We are currently reviewing the NFS Daemon and Sun RPC source for the kernel. In kernel source tree, linux-2.6.18/net/sunrpc/svcsock.c at line 1194 will terminate connections due to the jump in time:

1185 spin_lock_bh(&serv->sv_lock);
1186 if (!list_empty(&serv->sv_tempsocks)) {
1187 svsk = list_entry(serv->sv_tempsocks.next,
1188 struct svc_sock, sk_list);
1189 /* apparently the "standard" is that clients close
1190 * idle connections after 5 minutes, servers after
1191 * 6 minutes
1192 * http://www.connectathon.org/talks96/nfstcp.pdf
1193 */
1194 if (get_seconds() - svsk->sk_lastrecv < 6*60
1195 || test_bit(SK_BUSY, &svsk->sk_flags))
1196 svsk = NULL;
1197 }

The kernel’s Sun RPC code will close the socket when time jumps forward by more than 360 seconds or goes backwards in time by 1 second. The function get_seconds returns an unsigned long and the sk_lastrecv is of type time_t (long). The result will be treated as an unsigned long. When time goes backward the result will be huge, so the value will be greater than 360.

The NFS Connections being closed out because of time jumping is not a problem in itself. The NFS Clients will recover without issue when the NFS Server closes the connection. But the problem appears that the RPC does not close the socket but only discards the socket (guessing).

We were able to reproduce the NFS Clients closing the our lab. But not the NFS Connection becoming non-responsive. We will continue to try to reproduce the problem in our lab.

I have the following questions:
1) The Linux Kernel 2.6.18-238.9.1 is fairly old at this point. How do I determine if this a known problem with the kernel and fixed in a future released?
2) If it is a known problem, how do I determine if there is a patch for this problem?
3) Has anyone experienced similar problems and have a fix available?

Any help will be greatly appreciated.
Thank You

Last edited by cwwanner; 05-04-2016 at 07:56 PM.
 
Old 05-06-2016, 07:10 AM   #2
keefaz
LQ Guru
 
Registered: Mar 2004
Distribution: Slackware
Posts: 6,552

Rep: Reputation: 872Reputation: 872Reputation: 872Reputation: 872Reputation: 872Reputation: 872Reputation: 872
What did you use for mount options on client side?
http://wiki.linux-nfs.org/wiki/index.php/NewNfsManPage

Last edited by keefaz; 05-06-2016 at 07:12 AM.
 
Old 05-06-2016, 02:09 PM   #3
cwwanner
LQ Newbie
 
Registered: Dec 2015
Posts: 11

Original Poster
Rep: Reputation: Disabled
The NFS Mount is the root filesystem of a embedded Linux computer.

The NFS options are nfsver=3, tcp, rsize=8192, and wsize=8192. These options are configured in U-Boot and passed to the kernel at startup.
 
Old 05-19-2016, 08:13 PM   #4
cwwanner
LQ Newbie
 
Registered: Dec 2015
Posts: 11

Original Poster
Rep: Reputation: Disabled
I think I determined why the NFS Connections are locking up. Here is my current theory:

The NFS Server needs to keep an authenticated list of IP addresses to allow that host access to the NFS Server. The NFS Server needs to communicate with Linux Mount Daemon to determine if the IP address is valid. The way the NFS Server communicates to that Mount Daemon is by the RPC Cache mechanism. The RPC Cache mechanism is basically a file in the Linuxís proc ram disk. A new host (IP Address) makes a connection to the NFS Server and issues a request. The NFS Server needs to authenticate that host with the Mount Daemon before it executes the NFS Request. The NFS Server writes an authentication request to the file and defers the NFS Request. The Mount Daemon responds back if the host was or was not authenticated. If the host was authenticated, the NFS Server processes the NFS Request and sends the NFS Response. If host was rejected, the NFS Server responds to the NFS Client with Bad Credential Response. Once the host has been authenticated, the NFS Server does not have to authenticate the host again until the authentication expires. Each IP address in the RPC Cache has an expiration time. The expiration time is based on the wall clock time. The authentication is good for 30 minutes before the NFS Server has to check again with the Mount Daemon.

The RPC Cache has logic to flush invalid entries from the cache. When a cache entry is updated from the Mount Daemon, the authentication cache will invoke the cache flush logic. All the entries will be deleted with an expired expiration time. The deletion of the cache entry does not handle the deferred NFS Request. The NFS Server at this point will have an outstanding request that has not been processed associated with a host. The NFS Server will not close the socket to the NFS Client until it receives notification of the deferred NFS Request.

I believe the following is happening when the NFS connection lockup condition:
1) The NFS Server has the 9 Hosts Authenticated and is processing NFS Requests normally.
2) The NFS Serverís Wall Clock time is updated from 1/1/1970 to x/x/2016. The user updates the NFS Serverís time from the laptop. The laptopís time is 30 to 40
seconds ahead of the actual time.
a. If the NFS Server is not processing an NFS Request from an NFS Client, the NFS Server will terminate the NFS Clientís connection due to idle detection logic.
3) If the NFS Server was processing two or more NFS Requests but has not performed the authentication check when the wall clock time has been updated, the LWF state could
occur. The following logic that leads to the connection lockup is based on two NFS Requests being processed when time jumps:
a. The NFS Server checks the RPC Cache if the host is authenticated. Each host is valid but since the wall clock time jumped 46 years, the expiration time has
been exceeded.
b. The NFS Server will queue two Authentication Requests to the Mount Daemon.
c. The NFS Server defers each NFS Request by storing the NFS Request in a hash table.
d. The Mount Daemon processes the first authentication request and sends a response back to the NFS Server. The mechanism to send the response is a Linux system
call. The Mount Daemon will not process the second authentication request until the first request is completely processed.
e. The NFS Server will parse the authentication response and update the hostís cache entry. The deferred NFS Request for the host will be removed from the hash
table and placed on the NFS Serverís work queue to be processed.
f. The final step of the processing of the authentication response is the cache flush logic is invoked. When the cache flush logic finds a cache entry with an
expired time, the following happens:
i. If the cache entry has an authentication request pending, the request is deleted from the queue.
ii. The cache entry is deleted from the cache.
iii. The cache flush logic does not handle the deferred NFS Request. If the cache entry had a deferred NFS Request, the NFS Request is lost without
notification to the NFS Server.
g. The NFS Server no longer has authentication request and has lost the NFS Request for the 2nd NFS Client.
i. The NFS Client has a 60 second timeout before a retransmission of the NFS Request will occur.
h. The host that encountered the dropped NFS Request will have a 6o second delay during startup.
i. After the 60 second delay the Hostís NFS Client will retransmit the NFS Request. The NFS Server will process that NFS Request and future NFS Requests normally.
j. After the host has completed startup, the host will start an application.
k. The application will configure the NFS Serverís NTP Client
l. The NFS Serverís wall clock time is now adjusted backwards 30 to 40 seconds.
i. The RPC connection logic will think that 6 minutes of idle time have passed, if the wall clock time goes back in time by 1 or more seconds.
ii. The RPC Cache mechanism is not affected by wall clock time going backwards in time, only forward in time.
m. If the NFS Server is not processing a NFS Request from this NFS Client, the NFS Server will mark the NFS Clientís connection for termination.
n. The NFS Server will stop processing NFS Requests from that NFS Client.
o. The NFS Server will see an outstanding request that still needs to be processed. The termination of the connection is deferred.
p. The outstanding request will never be processed.
q. The NFS Server will never close the NFS Clientís socket and ignore all new NFS Requests from that NFS Client.
r. The host encounters the NFS Connection lockup at this point.

I am thinking of making the following changes to prevent the NFS Connection lockup:
a. Patch the RPC Socket logic to use jiffies instead of wall clock time to determine the connection idle time.
b. Patch the RPC Cache flush logic to delete the deferred NFS Request and notify the NFS Server of the dropped request, when cache entry is being deleted.

Any thoughts about patching the kernel?
 
  


Reply

Tags
nfs, rpc, time


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off



Similar Threads
Thread Thread Starter Forum Replies Last Post
System becomes un-responsive after a certain time BHABANIPRASADPATI Ubuntu 8 08-18-2009 06:43 AM
Time on Server keeps changing whether I'm using NTP or not. custangro Linux - Server 15 05-12-2009 03:57 PM
Server Messed up due to Glibc_Private error & Time ref. Hammad101 Linux - Server 0 02-23-2008 04:01 PM
LXer: Linux down time may be due to missing documentation or changing ... LXer Syndicated Linux News 2 06-13-2006 02:01 PM
NFS, client to server, time too large hendrikus Linux - Networking 2 07-29-2004 05:16 PM

LinuxQuestions.org > Forums > Linux Forums > Linux - Networking

All times are GMT -5. The time now is 04:53 PM.

Main Menu
Advertisement
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
Open Source Consulting | Domain Registration