Kerberized NFS with IPA Mount Issue

maccas17 · 02-15-2014, 06:43 PM

I've configured IPA on CentOS 6.5 and have an NFS server and an NFS client (both also CentOS 6.5) configured and working as IPA clients, e.g. can login as an IPA LDAP user.

I have tested plain NFSv4 and that works fine:

Code:

Testing Non-Kerberized NFS v4:
#####
#####
Client:
[root@nfs-client ~]# mount -v -t nfs4 -o rw,sec=sys nfs-server.example.local:/ /mnt
mount.nfs4: timeout set for Sat Feb 15 23:58:23 2014
mount.nfs4: trying text-based options 'sec=sys,addr=10.50.0.18,clientaddr=10.50.0.11'
nfs-server.example.local:/ on /mnt type nfs4 (rw,sec=sys)
[root@nfs-client ~]# df -h /mnt
Filesystem            Size  Used Avail Use% Mounted on
nfs-server.example.local:/   50G   14G   33G  30% /mnt
[root@nfs-client ~]# mount|grep nfs
sunrpc on /var/lib/nfs/rpc_pipefs type rpc_pipefs (rw)
nfsd on /proc/fs/nfsd type nfsd (rw)
nfs-server.example.local:/ on /mnt type nfs4 (rw,sec=sys,addr=10.50.0.18,clientaddr=10.50.0.11)

#####
#####
Server:
[root@nfs-server ~]# cat /etc/exports
/pmtest    10.50.0.0/24(rw,sec=sys,fsid=0)

[root@nfs-server ~]# exportfs -v
/pmtest         10.50.0.0/24(rw,wdelay,root_squash,no_subtree_check,fsid=0,sec=sys,rw,root_squash,no_all_squash)

When I try to mount using kerberos it fails. I've searched for a number of days and tried many things, but am still stuck. The key error I think is in the NFS server syslog:

Code:

Feb 15 23:43:24 nfs-server rpc.svcgssd[6446]: ERROR: GSS-API: error in handle_nullreq: gss_accept_sec_context(): GSS_S_FAILURE (Unspecified GSS failure.  Minor code may provide more information) - Wrong principal in request
Feb 15 23:43:24 nfs-server rpc.svcgssd[6446]: ERROR: GSS-API: error in handle_nullreq: gss_accept_sec_context(): GSS_S_FAILURE (Unspecified GSS failure.  Minor code may provide more information) - Wrong principal in request

I don't understand how I have the wrong principal in the krb5.keytab. The various guides I've seen all have a similar keytab config as me. Does anyone have any suggestions or ideas as I have little experience with kerberos and I'm not finding it easy to get my head round this

###########################################################
Output and Config Info From Failed Kerberized NFS mount:

Both client and server have secure NFS set to yes and name resolution is fine:

Code:

[root@nfs-client ~]# nslookup nfs-server
Server:         10.50.0.20
Address:        10.50.0.20#53

Name:   nfs-server.example.local
Address: 10.50.0.18

[root@nfs-client ~]# nslookup nfs-client
Server:         10.50.0.20
Address:        10.50.0.20#53

Name:   nfs-client.example.local
Address: 10.50.0.11


[root@nfs-server ~]# nslookup nfs-server
Server:         10.50.0.20
Address:        10.50.0.20#53

Name:   nfs-server.example.local
Address: 10.50.0.18

[root@nfs-server ~]# nslookup nfs-client
Server:         10.50.0.20
Address:        10.50.0.20#53

Name:   nfs-client.example.local
Address: 10.50.0.11

Code:

#####
#####
Client:
[root@nfs-client ~]# service iptables status;getenforce
iptables: Firewall is not running.
Disabled

Attempted mount:
[root@nfs-client ~]# mount -v -t nfs4 -o rw,sec=krb5 nfs-server.example.local:/ /mnt
mount.nfs4: timeout set for Sat Feb 15 23:45:23 2014
mount.nfs4: trying text-based options 'sec=krb5,addr=10.50.0.18,clientaddr=10.50.0.11'
mount.nfs4: mount(2): Permission denied
mount.nfs4: access denied by server while mounting nfs-server.example.local:/

/var/log/messages:
Feb 15 23:43:23 nfs-client rpc.gssd[1123]: dir_notify_handler: sig 37 si 0x7fffaf4fac70 data 0x7fffaf4fab40
Feb 15 23:43:23 nfs-client rpc.gssd[1123]: dir_notify_handler: sig 37 si 0x7fffaf4fac70 data 0x7fffaf4fab40
Feb 15 23:43:23 nfs-client rpc.gssd[1123]: dir_notify_handler: sig 37 si 0x7fffaf4fac70 data 0x7fffaf4fab40
Feb 15 23:43:23 nfs-client rpc.gssd[1123]: dir_notify_handler: sig 37 si 0x7fffaf4fac70 data 0x7fffaf4fab40
Feb 15 23:43:23 nfs-client rpc.gssd[1123]: dir_notify_handler: sig 37 si 0x7fffaf4fac70 data 0x7fffaf4fab40
Feb 15 23:43:23 nfs-client rpc.gssd[1123]: handling gssd upcall (/var/lib/nfs/rpc_pipefs/nfs/clnt0)
Feb 15 23:43:23 nfs-client rpc.gssd[1123]: handle_gssd_upcall: 'mech=krb5 uid=0 enctypes=18,17,16,23,3,1,2 '
Feb 15 23:43:23 nfs-client rpc.gssd[1123]: handling krb5 upcall (/var/lib/nfs/rpc_pipefs/nfs/clnt0)
Feb 15 23:43:23 nfs-client rpc.gssd[1123]: process_krb5_upcall: service is '<null>'
Feb 15 23:43:23 nfs-client rpc.gssd[1123]: Full hostname for 'nfs-server.example.local' is 'nfs-server.example.local'
Feb 15 23:43:23 nfs-client rpc.gssd[1123]: Full hostname for 'nfs-client.example.local' is 'nfs-client.example.local'
Feb 15 23:43:23 nfs-client rpc.gssd[1123]: No key table entry found for NFS-CLIENT.EXAMPLE.LOCAL$@EXAMPLE.LOCAL while getting keytab entry for 'NFS-CLIENT.EXAMPLE.LOCAL$@EXAMPLE.LOCAL'
Feb 15 23:43:23 nfs-client rpc.gssd[1123]: No key table entry found for root/nfs-client.example.local@EXAMPLE.LOCAL while getting keytab entry for 'root/nfs-client.example.local@EXAMPLE.LOCAL'
Feb 15 23:43:23 nfs-client rpc.gssd[1123]: Success getting keytab entry for 'nfs/nfs-client.example.local@EXAMPLE.LOCAL'
Feb 15 23:43:23 nfs-client rpc.gssd[1123]: Successfully obtained machine credentials for principal 'nfs/nfs-client.example.local@EXAMPLE.LOCAL' stored in ccache 'FILE:/tmp/krb5cc_machine_EXAMPLE.LOCAL'
Feb 15 23:43:23 nfs-client rpc.gssd[1123]: INFO: Credentials in CC 'FILE:/tmp/krb5cc_machine_EXAMPLE.LOCAL' are good until 1392594203
Feb 15 23:43:23 nfs-client rpc.gssd[1123]: using FILE:/tmp/krb5cc_machine_EXAMPLE.LOCAL as credentials cache for machine creds
Feb 15 23:43:23 nfs-client rpc.gssd[1123]: using environment variable to select krb5 ccache FILE:/tmp/krb5cc_machine_EXAMPLE.LOCAL
Feb 15 23:43:23 nfs-client rpc.gssd[1123]: creating context using fsuid 0 (save_uid 0)
Feb 15 23:43:23 nfs-client rpc.gssd[1123]: creating tcp client for server nfs-server.example.local
Feb 15 23:43:23 nfs-client rpc.gssd[1123]: DEBUG: port already set to 2049
Feb 15 23:43:23 nfs-client rpc.gssd[1123]: creating context with server nfs@nfs-server.example.local
Feb 15 23:43:23 nfs-client rpc.gssd[1123]: WARNING: Failed to create krb5 context for user with uid 0 for server nfs-server.example.local
Feb 15 23:43:23 nfs-client rpc.gssd[1123]: WARNING: Failed to create machine krb5 context with credentials cache FILE:/tmp/krb5cc_machine_EXAMPLE.LOCAL for server nfs-server.example.local
Feb 15 23:43:23 nfs-client rpc.gssd[1123]: WARNING: Machine cache is prematurely expired or corrupted trying to recreate cache for server nfs-server.example.local
Feb 15 23:43:23 nfs-client rpc.gssd[1123]: Full hostname for 'nfs-server.example.local' is 'nfs-server.example.local'
Feb 15 23:43:23 nfs-client rpc.gssd[1123]: Full hostname for 'nfs-client.example.local' is 'nfs-client.example.local'
Feb 15 23:43:23 nfs-client rpc.gssd[1123]: No key table entry found for NFS-CLIENT.EXAMPLE.LOCAL$@EXAMPLE.LOCAL while getting keytab entry for 'NFS-CLIENT.EXAMPLE.LOCAL$@EXAMPLE.LOCAL'
Feb 15 23:43:23 nfs-client rpc.gssd[1123]: No key table entry found for root/nfs-client.example.local@EXAMPLE.LOCAL while getting keytab entry for 'root/nfs-client.example.local@EXAMPLE.LOCAL'
Feb 15 23:43:23 nfs-client rpc.gssd[1123]: Success getting keytab entry for 'nfs/nfs-client.example.local@EXAMPLE.LOCAL'
Feb 15 23:43:23 nfs-client rpc.gssd[1123]: INFO: Credentials in CC 'FILE:/tmp/krb5cc_machine_EXAMPLE.LOCAL' are good until 1392594203
Feb 15 23:43:23 nfs-client rpc.gssd[1123]: INFO: Credentials in CC 'FILE:/tmp/krb5cc_machine_EXAMPLE.LOCAL' are good until 1392594203
Feb 15 23:43:23 nfs-client rpc.gssd[1123]: using FILE:/tmp/krb5cc_machine_EXAMPLE.LOCAL as credentials cache for machine creds
Feb 15 23:43:23 nfs-client rpc.gssd[1123]: using environment variable to select krb5 ccache FILE:/tmp/krb5cc_machine_EXAMPLE.LOCAL
Feb 15 23:43:23 nfs-client rpc.gssd[1123]: creating context using fsuid 0 (save_uid 0)
Feb 15 23:43:23 nfs-client rpc.gssd[1123]: creating tcp client for server nfs-server.example.local
Feb 15 23:43:23 nfs-client rpc.gssd[1123]: DEBUG: port already set to 2049
Feb 15 23:43:23 nfs-client rpc.gssd[1123]: creating context with server nfs@nfs-server.example.local
Feb 15 23:43:23 nfs-client rpc.gssd[1123]: WARNING: Failed to create krb5 context for user with uid 0 for server nfs-server.example.local
Feb 15 23:43:23 nfs-client rpc.gssd[1123]: WARNING: Failed to create machine krb5 context with credentials cache FILE:/tmp/krb5cc_machine_EXAMPLE.LOCAL for server nfs-server.example.local
Feb 15 23:43:23 nfs-client rpc.gssd[1123]: WARNING: Failed to create machine krb5 context with any credentials cache for server nfs-server.example.local
Feb 15 23:43:23 nfs-client rpc.gssd[1123]: doing error downcall
Feb 15 23:43:23 nfs-client rpc.gssd[1123]: dir_notify_handler: sig 37 si 0x7fffaf4fa770 data 0x7fffaf4fa640
Feb 15 23:43:23 nfs-client rpc.gssd[1123]: dir_notify_handler: sig 37 si 0x7fffaf4fa770 data 0x7fffaf4fa640
Feb 15 23:43:23 nfs-client rpc.gssd[1123]: dir_notify_handler: sig 37 si 0x7fffaf4fa770 data 0x7fffaf4fa640
Feb 15 23:43:23 nfs-client rpc.gssd[1123]: dir_notify_handler: sig 37 si 0x7fffaf4fa770 data 0x7fffaf4fa640
Feb 15 23:43:23 nfs-client rpc.gssd[1123]: dir_notify_handler: sig 37 si 0x7fffaf4fa770 data 0x7fffaf4fa640
Feb 15 23:43:23 nfs-client rpc.gssd[1123]: dir_notify_handler: sig 37 si 0x7fffaf4fa770 data 0x7fffaf4fa640
Feb 15 23:43:23 nfs-client rpc.gssd[1123]: destroying client /var/lib/nfs/rpc_pipefs/nfs/clnt0

/etc/krb5.conf
includedir /var/lib/sss/pubconf/krb5.include.d/

[libdefaults]
  default_realm = EXAMPLE.LOCAL
  dns_lookup_realm = false
  dns_lookup_kdc = true
  rdns = false
  ticket_lifetime = 24h
  forwardable = yes
  allow_weak_crypto = true
  permitted_enctypes = des3-cbc-sha1

[realms]
  EXAMPLE.LOCAL = {
  kdc = ipa-server.example.local:88
  master_kdc = ipa-server.example.local:88
  admin_server = ipa-server.example.local:749
  default_domain = example.local
  pkinit_anchors = FILE:/etc/ipa/ca.crt
  }

[domain_realm]
  .example.local = EXAMPLE.LOCAL
  example.local = EXAMPLE.LOCAL

/etc/krb5.keytab entries:
[root@nfs-client ~]# klist -kte
Keytab name: FILE:/etc/krb5.keytab
KVNO Timestamp         Principal
---- ----------------- --------------------------------------------------------
   4 02/15/14 23:27:51 host/nfs-client.example.local@EXAMPLE.LOCAL (des3-cbc-sha1)
   3 02/15/14 23:27:58 nfs/nfs-client.example.local@EXAMPLE.LOCAL (des3-cbc-sha1)


#####
#####
Server:
[root@nfs-server ~]# cat /etc/exports
/pmtest    10.50.0.0/24(rw,sec=krb5,fsid=0)

[root@nfs-server ~]# exportfs -v
/pmtest         10.50.0.0/24(rw,wdelay,root_squash,no_subtree_check,fsid=0,sec=krb5,rw,root_squash,no_all_squash)

[root@nfs-server ~]# service iptables status;getenforce
iptables: Firewall is not running.
Disabled


/var/log/messages:
Feb 15 23:43:24 nfs-server rpc.svcgssd[6446]: ERROR: GSS-API: error in handle_nullreq: gss_accept_sec_context(): GSS_S_FAILURE (Unspecified GSS failure.  Minor code may provide more information) - Wrong principal in request
Feb 15 23:43:24 nfs-server rpc.svcgssd[6446]: ERROR: GSS-API: error in handle_nullreq: gss_accept_sec_context(): GSS_S_FAILURE (Unspecified GSS failure.  Minor code may provide more information) - Wrong principal in request


/etc/krb5.conf
includedir /var/lib/sss/pubconf/krb5.include.d/

[libdefaults]
  default_realm = EXAMPLE.LOCAL
  dns_lookup_realm = true
  dns_lookup_kdc = true
  rdns = false
  ticket_lifetime = 24h
  forwardable = yes
  allow_weak_crypto = true
  permitted_enctypes = des3-cbc-sha1

[realms]
  EXAMPLE.LOCAL = {
  kdc = ipa-server.example.local:88
  master_kdc = ipa-server.example.local:88
  admin_server = ipa-server.example.local:749
  default_domain = example.local
  pkinit_anchors = FILE:/etc/ipa/ca.crt
  }

[domain_realm]
  .example.local = EXAMPLE.LOCAL
  example.local = EXAMPLE.LOCAL


/etc/krb5.keytab entries:
[root@nfs-server ~]# klist -kte
Keytab name: FILE:/etc/krb5.keytab
KVNO Timestamp         Principal
---- ----------------- --------------------------------------------------------
   2 02/15/14 23:09:43 host/nfs-server.example.local@EXAMPLE.LOCAL (des3-cbc-sha1)
   3 02/15/14 23:09:51 nfs/nfs-server.example.local@EXAMPLE.LOCAL (des3-cbc-sha1)