[SOLVED] NFS-mounted $HOME directories does not work on local graphical login, but works on ssh/tty

someshpr · 08-14-2017, 06:23 PM

Hello,

I am trying to set up a small lab netowrk of three clients and a file server via NFS. All machines are Ubuntu 16.04. My users all use the organization's active directory to authenticate. And I am trying to host their home directories on the server and mount them on clients via NFS so that everyone has access to their stuff irrespective the client he/she sits on.

I exported the home directory from my server, and the NFS shares mounts fine on the clients. This is my server exportfs

Code:

/export/home       xx.yy.0.0/255.255.0.0(rw,sync,no_subtree_check,no_root_squash)

And this is my client fstab

Code:

ip.of.the.server:/export/home   /nfs/home   nfs rw,async,hard,intr 0 0

As per the AD, the home directory path is supposed to be at /home/USERS/

Code:

$ wbinfo -i somesh
somesh:*:91000:10500:R, S:/home/USERS/somesh:/bin/bash

So, in cleints, I created a structure

Code:

somesh@client1:/home$ ls -al
total 28374772
drwxr-xr-x  6 root     root            4096 Aug 11 14:45 ./
drwxr-xr-x 28 root     root            4096 Aug 12 06:38 ../
lrwxrwxrwx  1 root     root              17 Aug 11 12:24 USERS -> /nfs/home/USERS/

and I kept the individual home directories in server

Code:

somesh@server:/export/home/USERS$ ls -al
total 32
drwxr-xr-x  8 root        root         4096 Aug 14 15:42 ./
drwxr-xr-x  3 root        root         4096 Aug 10 19:41 ../
drwxr-xr-x 16 somesh      domain users 4096 Aug 14 16:54 somesh/

I checked via ssh and everything works fine. I could log on to any client, see my stuff from server, as expected. But the problem is when try to login locally via GUI, after accepting my username and password, it gets stuck! Nothing happens for a long time. When I was first testing this, a couple of times a desktop environment was partially loading, i.e., I could see the icons, and click on them but no panels, no right-click, etc. But for last few times, nothing loads for at least 10 minutes, before I restart the client.
All the while, I can login via ssh and tty.

If I change the home directories to point to a local directory,

Code:

somesh@client1:/home$ ls -al
total 28374772
drwxr-xr-x  6 root     root            4096 Aug 11 14:45 ./
drwxr-xr-x 28 root     root            4096 Aug 12 06:38 ../
lrwxrwxrwx  1 root     root              17 Aug 11 12:24 USERS -> /localhome/USERS/

I get back my GUI login capability.

I am not sure if this is problem with NFS mounts, or with X/lightdm, or graphics card driver, or all of the above!

Any suggestions how can I find out the reason of the problem would be very much appreciated.

AwesomeMachine · 08-14-2017, 06:45 PM

If these are actually /home directories, they must be mounted on /home on the clients, not /nfs/home. If the are not actually /home directories, then they should have some other name to avoid confusion.

Are there three separate remote directories? You can't use the same /home directory for everyone. It probably doesn't work because /nfs/home is the wrong mount point. The GUI login won't work without /home in the right place.

someshpr · 08-14-2017, 08:01 PM

Quote:

Originally Posted by AwesomeMachine

If these are actually /home directories, they must be mounted on /home on the clients, not /nfs/home. If the are not actually /home directories, then they should have some other name to avoid confusion.

Are there three separate remote directories? You can't use the same /home directory for everyone. It probably doesn't work because /nfs/home is the wrong mount point. The GUI login won't work without /home in the right place.

Thanks AwesomeMachine for the quick response.

Sorry if my description of the problem was confusing. Each user has a separate home directory, which as per the Active Directory, are to be located at /home/USERS/$USERNAME on any client. What I am trying to do is to point the /home/USERS directory to a NFS mount (i.e., /nfs/home/USERS) via a symbolic link. This strategy works as long as /home/USERS points to any local location (i.e., /localhome/USERS) via symlink, but seemingly only breaks down when the target is a NFS mount (and when I try to login via GUI).

Are you suggesting that I may need to mount individual user's home directories separately? I.e., export from the server as

Code:

/export/home/user1       xx.yy.0.0/255.255.0.0(rw,sync,no_subtree_check,no_root_squash)
/export/home/user2       xx.yy.0.0/255.255.0.0(rw,sync,no_subtree_check,no_root_squash)

and mount on client as

Code:

ip.of.the.server:/export/home/user1   /home/USERS/user1   nfs rw,async,hard,intr 0 0
ip.of.the.server:/export/home/user2   /home/USERS/user2   nfs rw,async,hard,intr 0 0

Or this problem crops up because the Active Directory wants the home to be located at /home/USERS/$USERNAME and GUI by default wants the home directory at /home/$USERNAME?

I'll experiment with a local user without Active Directory authentication, so that I can test if the problem is due to some conflict due to AD.

AwesomeMachine · 08-14-2017, 08:36 PM

That's exactly correct. Linux will not work unless the users /home directory is directly under /home, not /home/users. Each directory requires a separate export. Each /home directory requires a separate fstab entry; a unique entry.

Linux looks for configuration files unique to each user in /home/username. So, that must be a link to the nfs share. If you put configs in other directories than where they're supposed to be it's chaos.

Then you have to change every script on the system to reflect the change. That could take 2x forever. It's not like executable files where you can just change the path.

I don't know why AD should care where you put the /home/username link. Then other thing is, if you have the system setup so it doesn't know which share to mount until it receives login credentials from the GUI login, the config files necessary to login might not sync properly with the nfs mount.

Until the nfs share is mounted, the GUI login cannot continue. That's a tricky one. So, you would probably need to mount all three /home directories on each boot. Then, the system would use the one that matches the particular login credentials.

someshpr · 08-15-2017, 08:32 AM

Thanks AwesomeMachines for the detailed reply. There are quite few things that I am still confused about:

Quote:

Originally Posted by AwesomeMachine

That's exactly correct. Linux will not work unless the users /home directory is directly under /home, not /home/users.

But I can always specify a non-default home directory, e.g.,

Code:

useradd -d /non/default/home/user user
usermod -m -d /non/default/new-home/user user

And that works perfectly in both GUI and ssh.

Quote:

Originally Posted by AwesomeMachine

I don't know why AD should care where you put the /home/username link.

I guess that is because my users authenticate via AD so their profile configurations are loaded from AD

Code:

$ wbinfo -i somesh
somesh:*:91000:10500:R, S:/home/USERS/somesh:/bin/bash

But this is besides the point, as it seems its not a problem with AD or non-standard location of home directory. As my next test seem to suggest.

Quote:

Originally Posted by AwesomeMachine

Each directory requires a separate export. Each /home directory requires a separate fstab entry; a unique entry.

To test if the issues are with mountpoints or with AD, I created three local users on a client:

Code:

$ useradd -d /localhome/USERS/user1 user1
$ useradd -d /home/USERS/user2 user2
$ useradd -d /home/user3 user3

where /localhome/USERS/user1 is a local directory, /home/USERS is a symlink to a NFS mount (/nfs/home) and hence /home/USERS/user2 is a directory loaded from the NFS server, and /home/user3 is a direct NFS mount of a home directory from the NFS server

Code:

$ df -h
Filesystem                 Size  Used Avail Use% Mounted on
ip.of.the.server:/export/home  917G  168G  703G  20% /nfs/home  <--- user2 is a directory in this mount
ip.of.the.server:/export/home/user3  917G  168G  703G  20% /home/user3

$ readlink -f /home/USERS
/nfs/home/USERS

ssh/tty log in works in all three cases but GUI login only works for user1.

Quote:

Originally Posted by AwesomeMachine

Then other thing is, if you have the system setup so it doesn't know which share to mount until it receives login credentials from the GUI login, the config files necessary to login might not sync properly with the nfs mount.

Until the nfs share is mounted, the GUI login cannot continue. That's a tricky one. So, you would probably need to mount all three /home directories on each boot. Then, the system would use the one that matches the particular login credentials.

All NFS shares are mounted at boot via static fstab. And the mounts are available when I tried to test GUI logins.

One more thing: I noticed when I try to use GUI login for any user whose home directory is on the NFS server (and hence cannot log in), it creates a .NFSxxyy file inside that user's home directory. Does this .NFS file indicate that something is not proper with the mounts?

DavidMcCann · 08-15-2017, 10:24 AM

Any display manager for a GUI log-in is going to check the permissions of certain critical directories, like ~/.gconf, and it won't work if they aren't correct. Could this be the source of your problem?

IsaacKuo · 08-15-2017, 02:57 PM

Hey, one potential gotcha...but it's old, like from back when I was still using GNOME2 I think. Once upon a time, I tried to use a symlink for my home directory. Something like

Code:

cd /home
ln -s /mnt/hda5/home/ikuo  <---don't do this!

And GNOME2 didn't like it at all. It just wouldn't log in graphically properly at all. So don't use symlinks in /home.

Anyway, have you considered the solution of mounting the shared folder directly onto /home, and then using a symlink for /home/USERS/? The basic setup would be something like:

/etc/fstab entry to mount the shared /export/home directly onto /home:

Code:

ip.of.the.server:/export/home /home nfs rw,async,hard,intr 0 0

The commands to create the desired symlink on the server:

Code:

cd /export/home
ln -s ./ USERS

This way, you have honest to goodness real home directories for each /home/userid

I don't really understand why AD wants them to be /home/USERS/userid instead of /home/userid, but I'm hoping that the symlink for USERS will be okay.

someshpr · 08-15-2017, 06:08 PM

Thanks for the pointers DavidMcKann and IsaacKuo.

Quote:

Originally Posted by DavidMcCann

Any display manager for a GUI log-in is going to check the permissions of certain critical directories, like ~/.gconf, and it won't work if they aren't correct. Could this be the source of your problem?

Could be... Unfortunately I have not found any such directory yet. GUI login seems to create a ~/.gconf with correct permissions. I need to explore a little bit more...

Quote:

Originally Posted by IsaacKuo

Hey, one potential gotcha...but it's old, like from back when I was still using GNOME2 I think. Once upon a time, I tried to use a symlink for my home directory. Something like

Code:

cd /home
ln -s /mnt/hda5/home/ikuo  <---don't do this!

And GNOME2 didn't like it at all. It just wouldn't log in graphically properly at all. So don't use symlinks in /home.

Anyway, have you considered the solution of mounting the shared folder directly onto /home, and then using a symlink for /home/USERS/? The basic setup would be something like:

/etc/fstab entry to mount the shared /export/home directly onto /home:

Code:

ip.of.the.server:/export/home /home nfs rw,async,hard,intr 0 0

The commands to create the desired symlink on the server:

Code:

cd /export/home
ln -s ./ USERS

This way, you have honest to goodness real home directories for each /home/userid

I don't really understand why AD wants them to be /home/USERS/userid instead of /home/userid, but I'm hoping that the symlink for USERS will be okay.

Unfortunately the problem persists even after this symlink/mount point manipulation.

I have no idea why the home directories are set to be /home/USERS/$username in the AD, but I guess that is the prerogative of the AD admin and some IT policy of the organization! And I do not think that is really the issue (as my three-local-user experiment suggests). It is not an issue with symlink either. I have other home directories which are symlink to a different location (in fact the local account from which I sudo everything is like that).

More and more I am inclined to believe that the problem is with NFS and display manager setup.

AwesomeMachine · 08-15-2017, 08:14 PM

Given the discussion thus far, I'm inclined to agree with DavidMcCann, that the permissions on the /home directories are being changed somehow in the whole process. Everything in the /home directories must be owned by the user:user, especially .Xauthority. Who is the owner on the server?

The ownership permissions may need to be corrected in fstab on the clients.

someshpr · 08-16-2017, 07:52 AM

Quote:

Originally Posted by AwesomeMachine

Given the discussion thus far, I'm inclined to agree with DavidMcCann, that the permissions on the /home directories are being changed somehow in the whole process. Everything in the /home directories must be owned by the user:user, especially .Xauthority. Who is the owner on the server?

The ownership permissions may need to be corrected in fstab on the clients.

The ownership of .Xauthority seems consistent. Both on the server and on the client this is what I get.

Code:

$ stat .Xauthority 
  File: '.Xauthority'
  Size: 115       	Blocks: 8          IO Block: 1048576 regular file
Device: 2bh/43d	Inode: 25955032    Links: 1
Access: (0600/-rw-------)  Uid: (910000/    somesh)   Gid: (10500/domain users)
Access: 2017-08-15 17:45:31.527820290 -0500
Modify: 2017-08-16 07:29:10.767182432 -0500
Change: 2017-08-16 07:29:10.767182432 -0500
 Birth: -

$ wbinfo -i somesh
somesh:*:910000:10500:Roy, Somesh:/home/MARQNET/somesh:/bin/bash

$ id somesh
uid=910000(somesh) gid=10500(domain users) groups=10500(domain users),910000(somesh),...

Now, last night before leaving the office I attempted one last GUI log in, it was stuck and I kept it so. Today morning I see I am logged-in. It seems it took overnight to login! I never waited for more than 10 minutes before this. Could be something funny going on with my NFS setup that is causing a really slow response when it comes to interacting with the display manager, may be? But any other file operation is being synced with NFS instantly.

AwesomeMachine · 08-16-2017, 08:27 AM

I see .Xauthority is not owned by user:user. It is owned by user:domain users. In the mount options in fstab you can specify: umask=0600,uid=910000,gid=910000.

If you need execute permissions, make umask=700

someshpr · 08-16-2017, 11:08 PM

I really appreciate the time and effort all of you have put in helping me resolve this issue. Thank you.

Quote:

Originally Posted by AwesomeMachine

I see .Xauthority is not owned by user:user. It is owned by user:domain users. In the mount options in fstab you can specify: umask=0600,uid=910000,gid=910000.

If you need execute permissions, make umask=700

Well, why does the .Xauthority need to be owned by user:user and not by uid:gid? Currently it is owned by uid:gid.

The wrong-permission hypothesis also does not explain why user3 cannot login via GUI. This user has same uid and gid (i.e., user:user) for all files including .Xauthority.

Quote:

Originally Posted by someshpr

To test if the issues are with mountpoints or with AD, I created three local users on a client:

Code:

$ useradd -d /localhome/USERS/user1 user1
$ useradd -d /home/USERS/user2 user2
$ useradd -d /home/user3 user3

where /localhome/USERS/user1 is a local directory, /home/USERS is a symlink to a NFS mount (/nfs/home) and hence /home/USERS/user2 is a directory loaded from the NFS server, and /home/user3 is a direct NFS mount of a home directory from the NFS server

Code:

$ df -h
Filesystem                 Size  Used Avail Use% Mounted on
ip.of.the.server:/export/home  917G  168G  703G  20% /nfs/home  <--- user2 is a directory in this mount
ip.of.the.server:/export/home/user3  917G  168G  703G  20% /home/user3

$ readlink -f /home/USERS
/nfs/home/USERS

ssh/tty log in works in all three cases but GUI login only works for user1.

Also, as I mentioned in the last post, I was finally able to log in following IsaacKuo's strategy but the login process takes more than 30 mins!

Quote:

Originally Posted by someshpr

Now, last night before leaving the office I attempted one last GUI log in, it was stuck and I kept it so. Today morning I see I am logged-in. It seems it took overnight to login! I never waited for more than 10 minutes before this. Could be something funny going on with my NFS setup that is causing a really slow response when it comes to interacting with the display manager, may be? But any other file operation is being synced with NFS instantly.

Also, nfs with umask seems to generate error in fstab. May be I am doing the sytax of fstab wrong... but before I spend time researching on adding umask correctly, I still have reservation accepting this as the cause of my problems.

Code:

$ cat /etc/fstab
ip.of.the.server:/export/home/somesh   /home/USERS/somesh   nfs umask=0700,uid=910000,gid=910000,rw,async,hard,intr 0 0

$ sudo mount -av
mount.nfs: timeout set for Wed Aug 16 22:29:14 2017
mount.nfs: trying text-based options 'umask=0700,uid=910000,gid=910000,hard,intr,vers=4,addr=ip.of.the.server,clientaddr=ip.of.the.client'
mount.nfs: mount(2): Invalid argument
mount.nfs: an incorrect mount option was specified

I also tried autofs and gdm3 instead of lightdm, just in case they affect anything, to no avail.

AwesomeMachine · 08-17-2017, 11:33 PM

Just use the ls -al command on the local /home and remote /home, and see if the ownerships are the same. For instance, on my laptop, every file in my /home directory is owned by user 1000 and group 1000.

someshpr · 08-18-2017, 10:06 PM

Thanks everyone for their contributions. I think I have found the problem.

It seems the problem was with the NFS lock. I changed my fstab scripts to load the NFS mount with "nolock" option and now everything is fine and dandy! It had nothing to do with AD, user permissions.

May be the display manager needs to perform some locking of some files that was not possible when NFS was mounted without the "nolock" option and hence it was waiting on that...

But this brings another question to mind. I thought "nolock" option makes the mount vulnerable to possible data corruption when mulitple clients try to access same file simultaneously. Is that true? Should I be concerned that I have to mount NFS with "nolock" option?

IsaacKuo · 08-19-2017, 03:22 AM

Quote:

Originally Posted by someshpr

Thanks everyone for their contributions. I think I have found the problem.

It seems the problem was with the NFS lock. I changed my fstab scripts to load the NFS mount with "nolock" option and now everything is fine and dandy! It had nothing to do with AD, user permissions.

May be the display manager needs to perform some locking of some files that was not possible when NFS was mounted without the "nolock" option and hence it was waiting on that...

But this brings another question to mind. I thought "nolock" option makes the mount vulnerable to possible data corruption when mulitple clients try to access same file simultaneously. Is that true? Should I be concerned that I have to mount NFS with "nolock" option?

This might indicate that there's something messed up with the nfs file server. I'm not sure how to troubleshoot this.

There is some risk with the "nolock" option if more than one client accesses the same file at the same time. This actually isn't a huge problem for a lot of software, such as vi, which create a small utility file when someone is editing a file (instead of holding a file lock for minutes/hours ... not recommended!). But if, say, two people have software appending stuff to a shared log file on two different computers, there's a good chance a log entry will get overwritten due to two simultaneous log appends at some point.

In your use case I kind of don't really see a serious risk. I mean, a user is only going to be accessing his files from one computer at a time. But the real risk is that because a problem is so incredibly unlikely, no one is going to be aware of the risk and they will not notice when the problem does cause some mysterious highly unlikely data loss in the middle of some random file somewhere.

See that's the thing. If it causes a failure, it's likely to cause a very obscure data loss that no one notices. And someday someone is trying to use log files to troubleshoot some issue and they just can't figure it out because a log file entry just isn't there.

I wouldn't be comfortable with it, even though the chances of a real practical problem are low. I'm more comfortable with failures that throw a noticeable error than silent failures.

So, even though I'm not really sure how to troubleshoot your nfs server for the "nolock" problem, I suggest you look around for how to try and figure it out.