DNS entries killed my NFS

Tyler_H72 · 04-16-2009, 02:33 PM

I am running a file-server on SLES10. I have had absolutely no problems with it until earlier this week. It has a number of NFS mounts that it serves up to other servers (all running OpenSuSE10.3 or 11), and those mounts have been in place for quite some time now without issue.
On Monday, I showed up at work and found that there had been (apparantly) a brief power outage on Sunday night (I say apparantly because I have no monitoring in place for this sort of thing, but judging by the log messages I received, the server had restarted in the middle of the night). I noticed that none of my NFS mounts were working, although all of the mounted servers listed the mounts as being active. After some time, I remembered that the only recent change had been my adding an entry in the /etc/hosts file for one of the mounted servers. I removed that entry, and everything started to work properly. The next day, the same thing happened- the logs showed what I believe to be a restart, and the NFS mounts were down. I added the DNS entry (which I had removed the day before to fix the system) and everything started working again. The next day, same scenario, and I found that changing the time to a few days later than it actually was fixed the problem.
After playing with it more, I have discovered that changing either the time or the /etc/hosts file switches the state of the mounts, either from working to not working or vice-versa. I cannot find any other things that affect this state, and I have tried multiple times restarting the NFS server without making changes to either of these, with no effect. I have checked the logs, and interestingly enough, altough the server states that the mount requests were authenticated, the servers that are trying to mount the directories return an error stating that the server denied permission.
If anyone has any ideas whatsoever as to what is happening here, or if you need more information to figure out what is wrong, please write me back. This issue is wreaking havoc on our backup process, our SVN server, and various other aspects of our daily work, and has consumed the majority of my week.

kentyler · 04-16-2009, 02:49 PM

Try to use ip addresses in your fstab. It may be due to the fact the linux machine comes back before the nfs exported machine does so when the drive tries to mount it fails but the mount shows mounted but not authenticated.

Post your logfiles.

Tyler_H72 · 04-16-2009, 03:35 PM

Actually, the /etc/fstab file is written using only IP addresses, and has worked fine for a number of months prior. The log files (/var/log/messages) that are relevant are as follows:

Code:

Apr 16 17:51:04 gvl-file-srv kernel: Bridge firewalling registered
Apr 16 17:53:26 file-srv ifstatus:     eth2      device: Linksys NC100 Network Everywhere Fast Ethernet 10/100 (rev 11)
Apr 16 17:53:26 file-srv ifstatus:     eth2      configuration: eth-id-xx:xx:xx:xx:xx:xx
Apr 16 17:53:26 file-srv ifstatus-dhcp:     eth2      DHCP client NOT running
Apr 16 17:53:26 file-srv ifstatus:     eth2      is down
Apr 16 17:53:26 file-srv ifstatus:     eth3      device: Broadcom Corporation NetXtreme II BCM5708 Gigabit Ethernet (rev 11)
Apr 16 17:53:26 file-srv ifstatus:               No configuration found for eth3
Apr 16 17:53:26 file-srv ifstatus:     eth4      device: Broadcom Corporation NetXtreme II BCM5708 Gigabit Ethernet (rev 11)
Apr 16 17:53:26 file-srv ifstatus:     eth4      configuration: eth-id-xx:xx:xx:xx:xx:xx
Apr 16 17:53:26 file-srv ifstatus:     eth4      IP address: 192.168.xx.xx/xx
Apr 16 17:53:26 file-srv ifstatus:     lo        
Apr 16 17:53:26 file-srv ifstatus:               IP address: 127.0.0.1/8
Apr 16 17:53:26 file-srv ifstatus:     eth2      device: Linksys NC100 Network Everywhere Fast Ethernet 10/100 (rev 11)
Apr 16 17:53:26 file-srv ifstatus:     eth2      configuration: eth-id-xx:xx:xx:xx:xx:xx
Apr 16 17:53:26 file-srv ifstatus-dhcp:     eth2      DHCP client NOT running
Apr 16 17:53:26 file-srv ifstatus:     eth2      is down
Apr 16 17:53:26 file-srv ifstatus:     eth3      device: Broadcom Corporation NetXtreme II BCM5708 Gigabit Ethernet (rev 11)
Apr 16 17:53:26 file-srv ifstatus:               No configuration found for eth3
Apr 16 17:53:26 file-srv ifstatus:     eth4      device: Broadcom Corporation NetXtreme II BCM5708 Gigabit Ethernet (rev 11)
Apr 16 17:53:26 file-srv ifstatus:     eth4      configuration: eth-id-xx:xx:xx:xx:xx:xx
Apr 16 17:53:26 file-srv ifstatus:     eth4      IP address: 192.168.xx.xx/xx
Apr 16 17:53:27 file-srv ifstatus:     lo        
Apr 16 17:53:27 file-srv ifstatus:               IP address: 127.0.0.1/8
Apr 16 17:53:27 file-srv ifprobe:     eth2      still using old config:  --> restart interface!
Apr 16 17:53:27 file-srv ifdown:     eth2      device: Linksys NC100 Network Everywhere Fast Ethernet 10/100 (rev 11)
Apr 16 17:53:27 file-srv ifdown:     eth2      configuration: eth-id-xx:xx:xx:xx:xx:xx
Apr 16 17:53:27 file-srv ifstatus:     lo        
Apr 16 17:53:27 file-srv ifstatus: lo is up
Apr 16 17:53:27 file-srv ifstatus: 1: lo: <LOOPBACK,UP> mtu 16436 qdisc noqueue 
Apr 16 17:53:27 file-srv ifstatus:     link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
Apr 16 17:53:27 file-srv ifstatus:     inet 127.0.0.1/8 scope host lo
Apr 16 17:53:27 file-srv ifstatus:     inet6 ::1/128 scope host 
Apr 16 17:53:27 file-srv ifstatus:        valid_lft forever preferred_lft forever
Apr 16 17:53:27 file-srv ifstatus:               IP address: 127.0.0.1/8
Apr 16 17:53:27 file-srv ifstatus-route: Configured routes for interface lo:
Apr 16 17:53:27 file-srv ifstatus-route:   127/8 - - lo  
Apr 16 17:53:27 file-srv ifstatus-route: Active routes for interface lo:
Apr 16 17:53:27 file-srv ifstatus-route:   127.0.0.0/8  scope link
Apr 16 17:53:27 file-srv ifstatus-route: 1 of 1 configured routes for interface lo up
Apr 16 17:53:27 file-srv ifup:     eth2      device: Linksys NC100 Network Everywhere Fast Ethernet 10/100 (rev 11)
Apr 16 17:53:27 file-srv ifup:     eth2      configuration: eth-id-xx:xx:xx:xx:xx:xx
Apr 16 17:53:27 file-srv ifup:     eth2      Startmode is 'off'
Apr 16 17:54:15 file-srv monit[3563]: 'log.txt' timestamp test failed for /root/bin/TapeBackup/
Apr 16 17:54:29 file-srv ifstatus:     eth2      device: Linksys NC100 Network Everywhere Fast Ethernet 10/100 (rev 11)
Apr 16 17:54:29 file-srv ifstatus:     eth2      configuration: eth-id-xx:xx:xx:xx:xx:xx
Apr 16 17:54:29 file-srv ifstatus-dhcp:     eth2      DHCP client NOT running
Apr 16 17:54:29 file-srv ifstatus:     eth2      is down
Apr 16 17:54:29 file-srv ifstatus:     eth3      device: Broadcom Corporation NetXtreme II BCM5708 Gigabit Ethernet (rev 11)
Apr 16 17:54:29 file-srv ifstatus:               No configuration found for eth3
Apr 16 17:54:29 file-srv ifstatus:     eth4      device: Broadcom Corporation NetXtreme II BCM5708 Gigabit Ethernet (rev 11)
Apr 16 17:54:29 file-srv ifstatus:     eth4      configuration: eth-id-xx:xx:xx:xx:xx:xx
Apr 16 17:54:29 file-srv ifstatus:     eth4      IP address: 192.168.xx.xx/xx
Apr 16 17:54:29 file-srv ifstatus:     lo        
Apr 16 17:54:29 file-srv ifstatus:               IP address: 127.0.0.1/8
Apr 16 17:54:30 file-srv ifstatus:     eth2      device: Linksys NC100 Network Everywhere Fast Ethernet 10/100 (rev 11)
Apr 16 17:54:30 file-srv ifstatus:     eth2      configuration: eth-id-xx:xx:xx:xx:xx:xx
Apr 16 17:54:30 file-srv ifstatus-dhcp:     eth2      DHCP client NOT running
Apr 16 17:54:30 file-srv ifstatus:     eth2      is down
Apr 16 17:54:30 file-srv ifstatus:     eth3      device: Broadcom Corporation NetXtreme II BCM5708 Gigabit Ethernet (rev 11)
Apr 16 17:54:30 file-srv ifstatus:               No configuration found for eth3
Apr 16 17:54:30 file-srv ifstatus:     eth4      device: Broadcom Corporation NetXtreme II BCM5708 Gigabit Ethernet (rev 11)
Apr 16 17:54:30 file-srv ifstatus:     eth4      configuration: eth-id-xx:xx:xx:xx:xx:xx
Apr 16 17:54:30 file-srv ifstatus:     eth4      IP address: 192.168.xx.xx/xx
Apr 16 17:54:30 file-srv ifstatus:     lo        
Apr 16 17:54:30 file-srv ifstatus:               IP address: 127.0.0.1/8
Apr 16 17:54:30 file-srv ifprobe:     eth2      still using old config:  --> restart interface!
Apr 16 17:54:30 file-srv ifdown:     eth2      device: Linksys NC100 Network Everywhere Fast Ethernet 10/100 (rev 11)
Apr 16 17:54:30 file-srv ifdown:     eth2      configuration: eth-id-xx:xx:xx:xx:xx:xx
Apr 16 17:54:30 file-srv ifstatus:     lo        
Apr 16 17:54:30 file-srv ifstatus: lo is up
Apr 16 17:54:30 file-srv ifstatus: 1: lo: <LOOPBACK,UP> mtu 16436 qdisc noqueue 
Apr 16 17:54:30 file-srv ifstatus:     link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
Apr 16 17:54:30 file-srv ifstatus:     inet 127.0.0.1/8 scope host lo
Apr 16 17:54:30 file-srv ifstatus:     inet6 ::1/128 scope host 
Apr 16 17:54:30 file-srv ifstatus:        valid_lft forever preferred_lft forever
Apr 16 17:54:30 file-srv ifstatus:               IP address: 127.0.0.1/8
Apr 16 17:54:30 file-srv ifstatus-route: Configured routes for interface lo:
Apr 16 17:54:30 file-srv ifstatus-route:   127/8 - - lo
Apr 16 17:54:30 file-srv ifstatus-route: Active routes for interface lo:
Apr 16 17:54:30 file-srv ifstatus-route:   127.0.0.0/8  scope link
Apr 16 17:54:30 file-srv ifstatus-route: 1 of 1 configured routes for interface lo up
Apr 16 17:54:30 file-srv ifup:     eth2      device: Linksys NC100 Network Everywhere Fast Ethernet 10/100 (rev 11)
Apr 16 17:54:30 file-srv ifup:     eth2      configuration: eth-id-xx:xx:xx:xx:xx:xx
Apr 16 17:54:30 file-srv ifup:     eth2      Startmode is 'off'





Apr 16 18:21:33 file-srv exportfs[14105]: duplicated export entries:
Apr 16 18:21:33 file-srv exportfs[14105]: 	192.168.xxx.*:/root/bin/report
Apr 16 18:21:33 file-srv exportfs[14105]: 	192.168.xxx.*:/root/bin/report


Apr 16 18:22:11 file-srv rpc.mountd: export request from 192.168.xxx.xxx

Apr 16 19:39:23 file-srv smbd[8366]: [2009/04/16 19:39:23, 0] lib/util_sock.c:read_data(529)
Apr 16 19:39:23 file-srv smbd[8366]:   read_data: read failure for 4 bytes to client 192.168.xxx.xxx. Error = No route to host

Apr 16 20:06:43 file-srv smbd[6520]: [2009/04/16 20:06:43, 0] lib/util_sock.c:read_data(529)
Apr 16 20:06:43 file-srv smbd[6520]:   read_data: read failure for 4 bytes to client 192.168.xxx.xxx. Error = Connection timed 
out

Apr 17 04:22:01 file-srv zmd: NetworkManagerModule (WARN): Failed to connect to NetworkManager

Apr 17 04:24:52 file-srv zmd: Daemon (WARN): Not starting remote web server

Apr 17 05:21:58 file-srv zmd: ShutdownManager (WARN): Preparing to sleep...
Apr 17 05:21:58 file-srv zmd: ShutdownManager (WARN): Going to sleep, waking up at 04/18/2009 04:11:58

Apr 17 10:30:11 file-srv smbd[17849]: [2009/04/17 10:30:11, 0] smbd/nttrans.c:call_nt_transact_ioctl(2350)
Apr 17 10:30:11 file-srv smbd[17849]:   call_nt_transact_ioctl(0x90078): Currently not implemented.
Apr 17 13:33:00 file-srv kernel: NFSD: Using /var/lib/nfs/v4recovery as the NFSv4 state recovery directory
Apr 17 13:33:00 file-srv kernel: NFSD: starting 90-second grace period


Apr 17 12:45:10 file-srv rpc.mountd: export request from 192.168.xxx.xxx


Apr 16 13:15:25 file-srv ctl_mboxlist[19635]: DBERROR: reading /var/lib/imap/db/skipstamp, assuming the worst: No such file or directory
Apr 16 13:15:26 file-srv ctl_mboxlist[19635]: skiplist: recovered /var/lib/imap/mailboxes.db (0 records, 144 bytes) in 1 second

The above log had various monit alerts and other such unrelated messages, but other than that this is all of the relevant messages that I found. The following lines are represented of each time I tried to mount the directories- this was found in the above logs at various spots, as well as the NFS restart shown at the bottom. By the way, I realize the times and dates in the above section are inconsistent, but as I said I did some toying with the date and time when I was trying to fix this issue.

Code:

Apr 13 14:34:15 file-srv rpc.mountd: refused mount request from srv.domain.com for /home/daisy (/): not exported
Apr 13 14:37:17 file-srv rpc.mountd: refused mount request from srv.domain.com for /home/daisy (/): not exported
Apr 13 14:38:04 file-srv rpc.mountd: refused mount request from srv.domain.com for /home/daisy (/): not exported
Apr 13 14:38:15 file-srv rpc.mountd: refused mount request from srv.domain.com for /home/daisy (/): not exported
Apr 13 14:38:16 file-srv rpc.mountd: refused mount request from srv.domain.com for /home/daisy (/): not exported
Apr 13 14:38:45 file-srv rpc.mountd: authenticated mount request from srv.domain.com:946 for /srv/svn (/srv/svn)
Apr 13 14:38:48 file-srv rpc.mountd: authenticated mount request from srv.domain.com:947 for /srv/svn (/srv/svn)
Apr 13 14:38:50 file-srv rpc.mountd: authenticated mount request from srv.domain.com:948 for /srv/svn (/srv/svn)
Apr 13 14:39:00 file-srv kernel: nfsd: last server has exited
Apr 13 14:39:00 file-srv kernel: nfsd: unexporting all filesystems
Apr 13 14:39:00 file-srv kernel: RPC: failed to contact portmap (errno -5).
Apr 13 14:39:00 file-srv rpc.mountd: Caught signal 15, un-registering and exiting.
Apr 13 14:39:00 file-srv kernel: NFSD: Using /var/lib/nfs/v4recovery as the NFSv4 state recovery directory
Apr 13 14:39:00 file-srv kernel: NFSD: starting 90-second grace period
Apr 13 14:39:02 file-srv rpc.mountd: authenticated mount request from srv.domain.com:949 for /srv/svn (/srv/svn)
Apr 13 14:39:04 file-srv rpc.mountd: authenticated mount request from srv.domain.com:950 for /srv/svn (/srv/svn)
Apr 13 14:39:14 file-srv rpc.mountd: authenticated mount request from srv.domain.com:953 for /srv/svn (/srv/svn)