elogind stops incrementing session ID after some uptime

opty · 03-08-2023, 06:05 AM

elogind 246.10 stops incrementing session ID after some uptime (check /var/log/messages*) and even creating new sessions for concurrent SSH logins. Daemon restart doesn't help.

I'm using Slackware 15.0 (32 bits), kernels 5.15.38-smp (uptime 200+ days) and 5.15.63-smp (uptime 128+ days).

0XBF · 03-08-2023, 08:54 AM

Haven't seen this problem ever, although I am using the 64-bit version of slackware 15.0.

Elogind's operation is pretty basic. The daemon runs at startup via /etc/rc.d/rc.elogind, it uses the system dbus to communicate and so depends on /etc/rc.d/rc.messagebus. As long as both are executable then elogind daemon should be running from boot.

The actual session registering and un-registering happens in the pam stack, with the pam_elogind.so module. As long as you have the stock configs in /etc/pam.d/ then the pam_elogind.so module should be getting used for most login methods (console, sddm, xdm, ssh, to name a few). If you've edited those configs and removed or made it possible to skip pam_elogind then it would fail to register a session with elogind.

How do you know it stops tracking sessions? Are there any logs of elogind-daemon crashing or the pam_elogind module failing in the pam logs?

You can use something like "loginctl list-sessions" to see what its tracking, and test if the daemon is running and/or can start at all. Btw, using 'loginctl' will auto activate the elogind-daemon if it was not running, so if it had crashed or otherwise stopped, it would just start another instance when using the loginctl command.

opty · 03-09-2023, 03:01 AM

D-Bus daemon runs, /etc/pam.d/* not modified, I know from /var/log/messages*, no crashes/failures and I'm quite familiar with loginctl, thanks.

Markus Wiesner · 03-09-2023, 10:24 AM

Quote:

Originally Posted by opty

elogind 246.10 stops incrementing session ID after some uptime (check /var/log/messages*) and even creating new sessions for concurrent SSH logins. Daemon restart doesn't help.

I noticed this happening after manually restarting sshd. The problem was that the restarted sshd inherited the elogin cgroup from the session I had when running /etc/rc.d/rc.sshd restart.

You can check the current cgroup of sshd with (note: for simplicity this and the following examples assume that "pgrep" finds exactly ONE process – if not, you will get errors):

Code:

$ grep elogind /proc/$(pgrep -f /usr/sbin/sshd)/cgroup
13:name=elogind:/

If you have a number at the end (ignore the "13:" at the beginning) like this:

Code:

13:name=elogind:/123

then the daemon is already assigned to that cgroup (123 in this case) and everyone logging in with ssh seems to inherit it.

You can (as root) remove a process from its elogind cgroup with:

Code:

pgrep -f /usr/sbin/sshd >/sys/fs/cgroup/elogind/cgroup.procs

For me I wrote a little wrapper script that I put in /usr/local/sbin/rc

Code:

#!/bin/bash
SCRIPT="/etc/rc.d/rc.$1"
shift
echo $$ >/sys/fs/cgroup/elogind/cgroup.procs
exec "${SCRIPT}" "$@"

so that i can restart services with

Code:

rc sshd restart

I also just found and tried this command which seems to do the same:

Code:

cgexec -g "*:/" /etc/rc.d/rc.sshd restart

but that's only from a short test! Hope this helps.

opty · 03-10-2023, 12:58 PM

That helped a bit, thanks!

Now elogind creates new sessions for concurrent SSH logins but using its own ID (e.g. c1) and complains to /var/log/syslog:

Code:

2023-03-10T19:39:38.000000+01:00 vodopnik elogind-daemon 339 - - Existing logind session ID 88 used by new audit session, ignoring.

First SSH login stays stuck at this ID.

Markus Wiesner · 03-10-2023, 02:15 PM

Quote:

Originally Posted by opty

That helped a bit, thanks!

Now elogind creates new sessions for concurrent SSH logins but using its own ID (e.g. c1) and complains to /var/log/syslog:

Did you also restart elogind or only sshd?

Maybe other services should be restarted without cgroup as well. You can check the cgroup of all processes with:

Code:

ps -e -o user,pid,cgroup,args

opty · 03-10-2023, 03:45 PM

I got rid of cgroups except the current SSH login but some services still show the stuck session ID in /proc/$PID/sessionid.

opty · 03-13-2023, 06:28 AM

I logged in using serial console and session ID jumped from 88 to 145. Then I rebooted 5.15.63-smp to 5.15.94-smp and ID incrementing works again but now I'm afraid of a potential service restart.

Code:

alias checksids='(cd /proc && for pid in [[:digit:]]*; do if [ -e $pid ]; then sid=`cat $pid/sessionid`; if [ ${sid:-0} -ne 4294967295 ]; then printf "%5i %-16s %4i %-24s %4i\n" $pid `cat $pid/comm` $sid `grep elogind $pid/cgroup` `cat $pid/loginuid`; fi; fi; done)'

on the 5.15.38-smp machine still shows e.g.

Code:

opty@zeryk:~$ checksids
15603 elogind-daemon    124 13:name=elogind:/        1000
15795 dnsmasq           124 13:name=elogind:/        1000
[...]
 6917 udevd             124 13:name=elogind:/        1000
 7139 ntpd              124 13:name=elogind:/        1000
 7146 sshd              124 13:name=elogind:/        1000

and no matter what I try (actually sudo -b bash -c '...'), I can't get rid of session 124. At least not alone, though.

Question of the day: How do I restart a service using sudo so it doesn't inherit /proc/self/sessionid of the shell I'm doing it from?

Update: I logged in using SSH as root and session ID jumped from 124 to 128. Then I restarted a service using sudo (due to /var/log/secure record), logged out, logged in again as root and ID incremented to 129 but I guess it worked just because of root as regular user login behaviour remains unchanged (ID stuck at 124).

Another update: I restarted SSH daemon as root using sudo and the issue seems fixed now (logged in as regular user and got incremented ID) but I want to restart as regular user using sudo, not as root.

Markus Wiesner · 03-13-2023, 09:40 AM

Quote:

Originally Posted by opty

Update: I logged in using SSH as root and session ID jumped from 124 to 128. Then I restarted a service using sudo (due to /var/log/secure record), logged out, logged in again as root and ID incremented to 129 but I guess it worked just because of root as regular user login behaviour remains unchanged (ID stuck at 124).

Looks like you only get a new session id if the sessionid of sshd belongs to a different user than the one who is currently logging in with ssh (restart as root = sessionid of root is stuck, restart as user with sudo = sessionid of sudo-user is stuck). I don't know if i missed that when i initially found my "cgroup" workaround or if something changed the behaviour afterwards.

So far I have only found another rather cumbersome workaround (which probably doesn't play well with sudo either): use /etc/inittab to start an "empty" tmux server (tmux -D) to which you can then connect to restart the services from there. As the server is started by init, it will have no cgroup and no sessionid and this inherits to all tmux-sessions and services restarted from those sessions. 🙈

But I would definitely prefer a way to reset the sessionid like the cgroup.

0XBF · 03-13-2023, 03:21 PM

Quote:

Originally Posted by Markus Wiesner

So far I have only found another rather cumbersome workaround (which probably doesn't play well with sudo either): use /etc/inittab to start an "empty" tmux server (tmux -D) to which you can then connect to restart the services from there. As the server is started by init, it will have no cgroup and no sessionid and this inherits to all tmux-sessions and services restarted from those sessions. 🙈

But I would definitely prefer a way to reset the sessionid like the cgroup.

Another option is to keep the root account from registering elogind sessions with a rule to skip the pam_elogind.so module.

E.g. If you have tty login access then you could modify /etc/pam.d/login to have the following line, right before the pam_elogind.so module is loaded:

Code:

...
session         sufficient      pam_succeed_if.so quiet uid = 0
-session        optional        pam_elogind.so

The 'sufficient' rule passes, as long as the user authenticating is uid 0. 'quiet' prevents this rule from getting logged in /var/log/secure, which would otherwise be logged on every tty login/logout when this rule tests.

With the 'sufficient' condition, pam exits the current config without running further modules, which should just be the remaining pam_elogind.so module. Order matters so pam_elogind needs to be last, with this rule second last. With that setup, root will skip loading and registering an elogind session on tty logins. Then you can login with a tty as root to start/stop rc.sshd while always keeping sshd unassociated with a elogind/cgroup session.

Kinda hacky but it works. Had some similar sessionid issues with tigervnc a while back which opty linked to. Didn't realize it was the same thing happening here. Not sure if there's an easier way to remove a process from a session id. I guess this is one of the cases where its nice to have an init started service manager that runs outside of elogind's scope.

opty · 03-14-2023, 04:19 AM

Quote:

Originally Posted by Markus Wiesner

Looks like you only get a new session id if the sessionid of sshd belongs to a different user than the one who is currently logging in with ssh (restart as root = sessionid of root is stuck, restart as user with sudo = sessionid of sudo-user is stuck).

Yes, session ID increments when login UID changes.

I will continue to monitor this issue.

opty · 03-17-2023, 10:51 AM

OpenSSH upgraded and I broke it again, fortunately just for root.

Markus Wiesner · 03-17-2023, 01:40 PM

Quote:

Originally Posted by opty

OpenSSH upgraded and I broke it again, fortunately just for root.

If I'm not missing something again, I've now found a solution that does not need the ugly inittab + tmux workaround and also works with sudo. You apparently can't directly change /proc/PID/sessionid, but it will be reset when you unset /proc/PID/loginuid.

Try this extended version of my (simplified) above /usr/local/sbin/rc script:

Code:

#!/bin/bash
SCRIPT="/etc/rc.d/rc.$1"
shift
echo $((0xffffffff)) >/proc/$$/loginuid
echo $$ >/sys/fs/cgroup/elogind/cgroup.procs
exec "${SCRIPT}" "$@"

(don't forget to chmod +x /usr/local/sbin/rc) and then restart sshd with:

Code:

rc sshd restart

opty · 03-18-2023, 03:31 PM

I already tried to unset in a shell and got -EPERM due to AUDIT_FEATURE_LOGINUID_IMMUTABLE set but I might conclude or do something wrong.

Markus Wiesner · 03-18-2023, 04:30 PM

Quote:

Originally Posted by opty

I already tried to unset in a shell and got -EPERM due to AUDIT_FEATURE_LOGINUID_IMMUTABLE set but I might conclude or do something wrong.

Did you try to unset the loginuid of the already running sshd? That doesn't work for me either:

Code:

root@...:~# echo $((0xffffffff)) > /proc/$(pgrep -f /usr/sbin/sshd)/loginuid
bash: echo: write error: Operation not permitted

But changing the loginuid (which is not 0 but 1000 here at first because of sudo in an xterm) of the current process works:

Code:

root@...:~# echo $(</proc/$$/loginuid)
1000
root@...:~# echo $(</proc/$$/sessionid)
1
root@...:~# echo $((0xffffffff)) > /proc/$$/loginuid
root@...:~# echo $(</proc/$$/loginuid)
4294967295
root@...:~# echo $(</proc/$$/sessionid)
4294967295

Note: I'm running slackware64-current, but still with kernel 5.19.16 (kernel-generic-5.19.16-x86_64-1). If it even with /proc/$$/loginuid does not work, then it seems to be due to the kernel version or its configuration?