Hi,

i'm new to linux so bear with me.

we have two servers running in a cluster mode. i disable the cluster so that we could install a new os kernel that was provided by our vendor tumbleweed.

So the active server which does not have the upgraded os kernel the version is
Linux version 2.6.5-7.283-smp (geeko@buildhost) (gcc version 3.3.3 (SuSE Linux)) #1 SMP Wed Nov 29 16:55:53 UTC 2006

on the passive server which has got the upgraded os kernel, the verion is
Linux version 2.6.16.60-0.39.3-smp (geeko@buildhost) (gcc version 4.1.2 20070115 (SUSE Linux)) #1 SMP Mon May 11 11:46:34 UTC 2009

for the server which does not have the kernel upgrade, i can mount to netapp mount. however from the server which has it's kernel upgraded, i get a rpc timeout error when i try to mount the drive.

"rpcinfo -p" on the server which has not been upgraded
stdevint01:~ # rpcinfo -p
program vers proto port
100000 2 tcp 111 portmapper
100000 2 udp 111 portmapper
100024 1 udp 32840 status
100021 1 udp 32840 nlockmgr
100021 3 udp 32840 nlockmgr
100021 4 udp 32840 nlockmgr
100024 1 tcp 45094 status
100021 1 tcp 45094 nlockmgr
100021 3 tcp 45094 nlockmgr
100021 4 tcp 45094 nlockmgr

"rpcinfo -p" on the server which has been upgraded
stdevint02:~ # rpcinfo -p
program vers proto port
100000 2 tcp 111 portmapper
100000 2 udp 111 portmapper

rpcinfo -p on the netapp server which we are trying to mount
stdevint02:~ # rpcinfo -p 10.10.12.169
program vers proto port
100024 1 tcp 4047 status
100011 1 udp 4049 rquotad
100024 1 udp 4047 status
100021 4 tcp 4045 nlockmgr
100021 3 tcp 4045 nlockmgr
100021 1 tcp 4045 nlockmgr
100021 4 udp 4045 nlockmgr
100021 3 udp 4045 nlockmgr
100021 1 udp 4045 nlockmgr
100005 3 tcp 4046 mountd
100005 2 tcp 4046 mountd
100005 1 tcp 4046 mountd
100005 3 udp 4046 mountd
100005 2 udp 4046 mountd
100005 1 udp 4046 mountd
100003 3 tcp 2049 nfs
100003 2 tcp 2049 nfs
100003 3 udp 2049 nfs
100003 2 udp 2049 nfs
100000 2 tcp 111 portmapper
100000 2 udp 111 portmapper

tcpdump from the server which has not been upgraded
14:48:30.195369 IP 10.0.63.48.43879 > 10.10.12.169.111: S 2405958562:2405958562(0) win 5840 <mss 1460,sackOK,timestamp 1138878866 0,nop,wscale 0>
14:48:30.198792 IP 10.10.12.169.111 > 10.0.63.48.43879: S 1747572012:1747572012(0) ack 2405958563 win 8760 <mss 1460,nop,nop,sackOK,nop,wscale 0,nop,nop,timestamp 1056463355 1138878866>
14:48:30.198815 IP 10.0.63.48.43879 > 10.10.12.169.111: . ack 1 win 5840 <nop,nop,timestamp 1138878869 1056463355>
14:48:30.198859 IP 10.0.63.48.43879 > 10.10.12.169.111: P 1:45(44) ack 1 win 5840 <nop,nop,timestamp 1138878869 1056463355>
14:48:30.200240 IP 10.10.12.169.111 > 10.0.63.48.43879: P 1:453(452) ack 45 win 8760 <nop,nop,timestamp 1056463355 1138878869>
14:48:30.200251 IP 10.0.63.48.43879 > 10.10.12.169.111: . ack 453 win 6432 <nop,nop,timestamp 1138878871 1056463355>
14:48:30.200282 IP 10.0.63.48.43879 > 10.10.12.169.111: F 45:45(0) ack 453 win 6432 <nop,nop,timestamp 1138878871 1056463355>
14:48:30.201143 IP 10.10.12.169.111 > 10.0.63.48.43879: . ack 46 win 8760 <nop,nop,timestamp 1056463355 1138878871>
14:48:30.201292 IP 10.10.12.169.111 > 10.0.63.48.43879: F 453:453(0) ack 46 win 8760 <nop,nop,timestamp 1056463355 1138878871>
14:48:30.201305 IP 10.0.63.48.43879 > 10.10.12.169.111: . ack 454 win 6432 <nop,nop,timestamp 1138878872 1056463355>
14:48:30.205557 IP 10.0.63.48.1000 > 10.10.12.169.4046: S 2405926255:2405926255(0) win 5840 <mss 1460,sackOK,timestamp 1138878876 0,nop,wscale 0>
14:48:30.208645 IP 10.10.12.169.4046 > 10.0.63.48.1000: S 2868713730:2868713730(0) ack 2405926256 win 8760 <mss 1460,nop,nop,sackOK,nop,wscale 0,nop,nop,timestamp 1056463356 1138878876>
14:48:30.208656 IP 10.0.63.48.1000 > 10.10.12.169.4046: . ack 1 win 5840 <nop,nop,timestamp 1138878879 1056463356>
14:48:30.208689 IP 10.0.63.48.1000 > 10.10.12.169.4046: P 1:109(108) ack 1 win 5840 <nop,nop,timestamp 1138878879 1056463356>
14:48:30.209900 IP 10.10.12.169.4046 > 10.0.63.48.1000: P 1:77(76) ack 109 win 8760 <nop,nop,timestamp 1056463356 1138878879>
14:48:30.209910 IP 10.0.63.48.1000 > 10.10.12.169.4046: . ack 77 win 5840 <nop,nop,timestamp 1138878880 1056463356>
14:48:30.209955 IP 10.0.63.48.43880 > 10.10.12.169.111: S 2403140452:2403140452(0) win 5840 <mss 1460,sackOK,timestamp 1138878880 0,nop,wscale 0>
14:48:30.212949 IP 10.10.12.169.111 > 10.0.63.48.43880: S 3027489341:3027489341(0) ack 2403140453 win 8760 <mss 1460,nop,nop,sackOK,nop,wscale 0,nop,nop,timestamp 1056463357 1138878880>
14:48:30.212960 IP 10.0.63.48.43880 > 10.10.12.169.111: . ack 1 win 5840 <nop,nop,timestamp 1138878883 1056463357>
14:48:30.212978 IP 10.0.63.48.43880 > 10.10.12.169.111: P 1:61(60) ack 1 win 5840 <nop,nop,timestamp 1138878883 1056463357>
14:48:30.214054 IP 10.10.12.169.111 > 10.0.63.48.43880: P 1:33(32) ack 61 win 8760 <nop,nop,timestamp 1056463357 1138878883>
14:48:30.214063 IP 10.0.63.48.43880 > 10.10.12.169.111: . ack 33 win 5840 <nop,nop,timestamp 1138878884 1056463357>
14:48:30.214085 IP 10.0.63.48.43880 > 10.10.12.169.111: F 61:61(0) ack 33 win 5840 <nop,nop,timestamp 1138878885 1056463357>
14:48:30.214108 IP 10.0.63.48.1000 > 10.10.12.169.4046: F 109:109(0) ack 77 win 5840 <nop,nop,timestamp 1138878885 1056463356>
14:48:30.214212 IP 10.0.63.48.0 > 10.10.12.169.2049: 0 null
14:48:30.214957 IP 10.10.12.169.111 > 10.0.63.48.43880: . ack 62 win 8760 <nop,nop,timestamp 1056463357 1138878885>
14:48:30.215073 IP 10.10.12.169.111 > 10.0.63.48.43880: F 33:33(0) ack 62 win 8760 <nop,nop,timestamp 1056463357 1138878885>
14:48:30.215082 IP 10.0.63.48.43880 > 10.10.12.169.111: . ack 34 win 5840 <nop,nop,timestamp 1138878886 1056463357>
14:48:30.215173 IP 10.10.12.169.4046 > 10.0.63.48.1000: . ack 110 win 8760 <nop,nop,timestamp 1056463357 1138878885>
14:48:30.215294 IP 10.10.12.169.4046 > 10.0.63.48.1000: F 77:77(0) ack 110 win 8760 <nop,nop,timestamp 1056463357 1138878885>
14:48:30.215302 IP 10.0.63.48.1000 > 10.10.12.169.4046: . ack 78 win 5840 <nop,nop,timestamp 1138878886 1056463357>
14:48:30.217218 IP 10.10.12.169.2049 > 10.0.63.48.0: reply ok 0 null
14:48:30.217230 IP 10.0.63.48.758 > 10.10.12.169.2049: . ack 3915002460 win 5840 <nop,nop,timestamp 1138878888 1056463357>
14:48:30.217249 IP 10.0.63.48.1317324766 > 10.10.12.169.2049: 116 fsinfo [|nfs]
14:48:30.218450 IP 10.10.12.169.2049 > 10.0.63.48.1317324766: reply ok 168 fsinfo [|nfs]
14:48:30.218460 IP 10.0.63.48.758 > 10.10.12.169.2049: . ack 169 win 5840 <nop,nop,timestamp 1138878889 1056463357>

tcpdump on the server which has got the kernel upgrade
09:18:07.657072 IP 10.0.63.50.48708 > 10.10.12.169.111: S 3867538137:3867538137(0) win 5840 <mss 1460,sackOK,timestamp 85103020 0,nop,wscale 2>
09:18:07.661471 IP 10.10.12.169.111 > 10.0.63.50.48708: S 720357792:720357792(0) ack 3867538138 win 8760 <mss 1460,nop,nop,sackOK,nop,wscale 0,nop,nop,timestamp 1045835584 85103020>
09:18:07.661483 IP 10.0.63.50.48708 > 10.10.12.169.111: . ack 1 win 1460 <nop,nop,timestamp 85103022 1045835584>
09:18:07.661522 IP 10.0.63.50.48708 > 10.10.12.169.111: P 1:45(44) ack 1 win 1460 <nop,nop,timestamp 85103022 1045835584>
09:18:07.662972 IP 10.10.12.169.111 > 10.0.63.50.48708: P 1:453(452) ack 45 win 8760 <nop,nop,timestamp 1045835585 85103022>
09:18:07.662978 IP 10.0.63.50.48708 > 10.10.12.169.111: . ack 453 win 1728 <nop,nop,timestamp 85103022 1045835585>
09:18:07.662998 IP 10.0.63.50.48708 > 10.10.12.169.111: F 45:45(0) ack 453 win 1728 <nop,nop,timestamp 85103022 1045835585>
09:18:07.663062 IP 10.0.63.50.642 > 10.10.12.169.4046: UDP, length 104
09:18:07.663919 IP 10.10.12.169.111 > 10.0.63.50.48708: . ack 46 win 8760 <nop,nop,timestamp 1045835585 85103022>
09:18:07.664027 IP 10.10.12.169.111 > 10.0.63.50.48708: F 453:453(0) ack 46 win 8760 <nop,nop,timestamp 1045835585 85103022>
09:18:07.664034 IP 10.0.63.50.48708 > 10.10.12.169.111: . ack 454 win 1728 <nop,nop,timestamp 85103022 1045835585>
09:18:10.664005 IP 10.0.63.50.642 > 10.10.12.169.4046: UDP, length 104
09:18:13.663932 IP 10.0.63.50.642 > 10.10.12.169.4046: UDP, length 104
09:18:16.663881 IP 10.0.63.50.642 > 10.10.12.169.4046: UDP, length 104
09:18:19.663830 IP 10.0.63.50.642 > 10.10.12.169.4046: UDP, length 104
09:18:22.663780 IP 10.0.63.50.642 > 10.10.12.169.4046: UDP, length 104
09:18:25.663730 IP 10.0.63.50.642 > 10.10.12.169.4046: UDP, length 104

somehow the server which has got the kernel upgrade is trying to use udp. the server goes through a firewall before getting to the netapp server and upd is not allowed.

in the fstab entries, i specified tcp,proto=tcp but no matter what values i put in, it's trying to use upd.

could anyone please assist.

Thanks,
Brian.

Just an idea, but is lockd module loaded on upgraded machine ?

Do lsmod on each server and post plz.

hi,

thank you for u're response.

lsmod on the machine which is not upgraded
stdevint01:~ # lsmod
Module Size Used by
udf 102792 0
iptable_mangle 19584 0
iptable_nat 42540 0
ip_conntrack 55228 1 iptable_nat
iptable_filter 19584 0
ip_tables 35456 3 iptable_mangle,iptable_nat,iptable_filter
ip6table_filter 19328 0
ip6_tables 36752 1 ip6table_filter
ipv6 349816 258
edd 27552 0
usb_storage 86720 1
hid 60672 0
joydev 27392 0
sg 58936 0
st 59172 0
sr_mod 34596 0
ehci_hcd 47492 0
uhci_hcd 49184 0
hw_random 22440 0
evdev 26880 0
af_packet 41612 0
usbcore 140408 6 usb_storage,hid,ehci_hcd,uhci_hcd
bnx2 159568 0
subfs 25600 3
dm_mod 77024 0
ext3 141168 2
jbd 90824 1 ext3
megaraid_sas 51128 3
sd_mod 38144 4
scsi_mod 151424 6 usb_storage,sg,st,sr_mod,megaraid_sas,sd_mod


lsmod on the machine which is upgraded
stdevint02:~ # lsmod
Module Size Used by
ip6table_mangle 19072 0
iptable_mangle 19584 0
iptable_nat 25348 0
ip_nat 36268 1 iptable_nat
ip_conntrack 74524 2 iptable_nat,ip_nat
nfnetlink 23368 2 ip_nat,ip_conntrack
ip6table_filter 19584 0
ip6_tables 32712 2 ip6table_mangle,ip6table_filter
nfs 237272 0
lockd 89200 1 nfs
nfs_acl 20224 1 nfs
sunrpc 175432 3 nfs,lockd,nfs_acl
iptable_filter 19712 0
ip_tables 31048 3 iptable_mangle,iptable_nat,iptable_filter
x_tables 31112 3 iptable_nat,ip6_tables,ip_tables
joydev 27520 0
ipv6 340000 84
sr_mod 33060 0
af_packet 56460 0
usb_storage 101440 0
usbhid 61088 0
loop 33040 0
dm_mod 80528 0
hw_random 22568 0
shpchp 61984 0
uhci_hcd 49440 0
ide_cd 57888 0
pci_hotplug 44800 1 shpchp
cdrom 52520 2 sr_mod,ide_cd
ehci_hcd 49800 0
usbcore 150440 5 usb_storage,usbhid,uhci_hcd,ehci_hcd
bnx2 196104 0
ext3 153872 2
jbd 84672 1 ext3
ata_piix 34308 0
ahci 46472 0
libata 146600 2 ata_piix,ahci
edd 26760 0
fan 21896 0
thermal 32528 0
processor 53228 1 thermal
sg 53304 0
megaraid_sas 57148 5
piix 28164 0 [permanent]
sd_mod 37888 4
scsi_mod 171192 7 sr_mod,usb_storage,ahci,libata,sg,megaraid_sas,sd_mod
ide_disk 32768 0
ide_core 166148 4 usb_storage,ide_cd,piix,ide_disk


as mentioned earlier, i'm new to linux. secondly the update to the kernel was provided by a vendor whose application we are upgrading. so i don't know how to check if lockd is loaded on the server. however the lsmod output showed me that lockd is being used by nfs so i would assume that it's loaded on the upgrade server.

Brian.

I though the lsmod were inversed ie.: lockd on old server and not on upgraded... but it's not the case.

Try to manually lauch rpc.portmap and rpc.statd on upgraded machine, and see rpcinfo output.

hi,

as mentioned i'm new to linux...did a search on google...

i did a "/etc/init.d/portmap stop" and then i did a "/etc/init.d/portmap start" in order to stop and restart rpc.portmap

i could'nt find any information on how to manually start rpc.statd. most of the google tell me that rpc.statd is located in /sbin but i can't find it in there.

to reiterate we are on a SuSe box and running the nfs client so that we can connect to a nfs server.

Brian.

So it's probably an udp problem. Maybe the new kernel is not compiled with nfs over tcp.
Look in /boot to see any "config" file, or in /usr/src/linux , looking for a ".config". Inside, search nfs options.

I suppose you tried to manually mount the nfs share with the tcp option ?
There's also an option to specify the lockd port, ie : lockd.tcpport=48000

i looked in /boot. there were'nt any config files in there...

looked in /usr/src/linux-2.6.5-7.283-obj/x86_64 directory. there is a smp directory. in that directory there is a makedumpfile.config file and it does not contain any nfs entries.


how do you check if the new kernel is complied with nfs over tcp.

yes i did try to manually mount the nfs share with the tcp option and got a rpc timeout error.

i tried using the lockd.tcpport while manually trying to mount the share and got the below error. google also did'nt show me any mount options to use lockd

stdevint02:~ # mount -t nfs 10.12.170:/vol/NFS_TST_VOL001/TRANSDATA /share/TRANSDATA -o tcp,proto=tcp,lockd.tcpport=48000
mount: unknown nfs mount parameter: lockd.tcpport=48000

Take a look at kernel compiling, in order to see which options are used.
Locate the sources, and launch "make xconfig" as root in it.

Strange that lockd gives error.

You should consider contacting your kernel provider...