rpc: remote program is not registered
 

client side trace (printed by program in shell)

In clnt_create_timed...
trying netid tcp
hostname: 10.98.228.147 progNum: 6874321 ver: 1
net id:tcp nc_protofmly: inet nc_proto:tcp
nc_device_path:-
tp.sec: 2
--RPC errorNum: 15--

1> Note that errorNum 15 means "remote program not registered"

while here's the server side trace

[root@10.92.228.147]# /usr/bin/rpcinfo
program version netid address service owner
100000 4 tcp6 ::.0.111 portmapper superuser
100000 3 tcp6 ::.0.111 portmapper superuser
100000 4 udp6 ::.0.111 portmapper superuser
100000 3 udp6 ::.0.111 portmapper superuser
100000 4 local /var/run/rpcbind.sock portmapper superuser
100000 3 local /var/run/rpcbind.sock portmapper superuser
6874321 1 tcp 0.0.0.0.30.151 - superuser
...

2> Note that the service for programNum 6874321 is '-'. Basically I've installed rpcbind on server side and assume that server will register service via rpcbind and client will talk to rpcbind first.

[root@10.92.228.147]# rpcinfo -p server_host_name
program vers proto port service
100000 2 tcp 111 portmapper
100000 2 udp 111 portmapper
100024 1 udp 756 status
100024 1 tcp 759 status
100021 1 udp 53281 nlockmgr
100021 3 udp 53281 nlockmgr
100021 4 udp 53281 nlockmgr
100021 1 tcp 44463 nlockmgr
100021 3 tcp 44463 nlockmgr
100021 4 tcp 44463 nlockmgr
100007 2 udp 980 ypbind
100007 1 udp 980 ypbind
100007 2 tcp 983 ypbind
100007 1 tcp 983 ypbind


2.1> Curiously, when rpcinfo with -p(probe portmapper) hostname option, rpcbind is not shown.

3> 0.0.0.0.30.151 means 30*256+151 = 7831, which is exactly what the program should behave (so it's a correct port).

[root@10.92.228.147]# netstat -l | grep 7831
tcp 0 0 *:7831 *:* LISTEN

4> Note that Recv/Send is 0 and local port is truly 7831, listening..

So symptom is:
server does seem to have registered the program but client not able to rpc to it because client think the program is not registered on server.

Grateful for any thoughts! :)

any idea?