ICE sockets accumulate then cause failure, possible Slackware fix is proposed

pan64 · 11-07-2023, 12:15 AM

Quote:

Originally Posted by _peter

this is from 2009-07-01 11:03:33 UTC
https://gitlab.freedesktop.org/xorg/...ice/-/issues/1

the interesting question: why is it not yet fixed? I don't really know the answer, there can be a few possibilities, like:
1. it is not that trivial at all
2. it is not that important at all
3. it is a bug, but not in the mentioned lib (cannot be fixed in that lib)
4. it is not a bug at all, but a misuse

rkelsen · 11-07-2023, 03:22 AM

Quote:

Originally Posted by selfprogrammed

*** Do we blame XFCE4, or Xorg
I run XFCE4.

At this point, I think it is pretty clear that its an XFCE problem.

Check out the FreeBSD solution from 2005: https://lists.freebsd.org/pipermail/...ch/082242.html

zeebra · 11-07-2023, 03:42 AM

Hmm.. I haven't personally run into this issue, but then again, I use Wayland (KDE), not Xorg. I do however have ICE sockets in my logs, more on that in a second.. I actually use Xorg too, on another computer and a virtual machine, both with Slackware.

May I propose using Tomoyo-Linux (kernel module+userland tools) to get a better overview of this situation, and see who makes the various system calls, set up sockets, when and how, and request to have them removed, when and how. I see ICE sockets all the time in my Tomoyo audit logs, and I have a general idea of how they work, from studying policies. I have not looked into this in detail though, because quite frankly, I don't care about it.

If you set up MAC audit (or audit in general), you can sort any calls that include ICE-Unix.

Tomoyo is useful for this for two reasons. The audid you can easily set up and use, plus the overview that ncurses based tomoyo policy tools provide (also easy to set up and use).

Petri Kaukasoina · 11-07-2023, 05:28 AM

Quote:

Originally Posted by selfprogrammed

*** Do we blame XFCE4, or Xorg
...
From my reading of the rc.6 script, I doubt that XFCE4 is being given the chance to shutdown cleanly.
...
I have some doubt that there is anywhere, that any of the scripts or services that started Xorg are ever given a clean shutdown.

I did verify that it is Xorg that creates the ICE socket. I do not know if there is anywhere in Xorg that would remove it.

It's not about shutdown. Let's keep the system running. If you are in runlevel 3 (linux console) and startxfce4, it creates a socket, owned by you, in /tmp/.ICE-unix. netstat -lpAunix shows

Code:

Active UNIX domain sockets (only servers)
Proto RefCnt Flags       Type       State         I-Node   PID/Program name     Path
unix  2      [ ACC ]     STREAM     LISTENING     2338337  16870/xfce4-session  /tmp/.ICE-unix/16870
unix  2      [ ACC ]     STREAM     LISTENING     2338336  16870/xfce4-session  @/tmp/.ICE-unix/16870

The server listening (xfce4-session) is run under your UID. When you "Log Out", xfce finishes and you go back to text console. And you still have the socket left.

Then kde. It again creates the socket, owned by you, and now the server listening is process ksmserver, running under your UID. When you "Log Out", kde finishes and you go back to text console. No sockets left.

But I use fvwm. It never creates anything in /tmp/.ICE-unix, at least with my fvwm configuration. 'ls -l /tmp/.ICE-unix' shows the directory has the same date as I originally installed Slackware-15.0, so nothing has ever been there. But it's the same Xorg server running with fvwm as with xfce or kde.

The only place where string /tmp/.ICE-unix is found hard coded (in addition to rc.S) is in /usr/lib64/libICE.so.6.3.0. And /usr/libexec/Xorg does not link to libICE, not even indirectly. So I can't understand how you verified that it was Xorg that created the ICE socket. Xorg runs as root and its log files are owned by root. The X11 socket /tmp/.X11-unix/X0 is owned by root and its server is Xorg, running as root (and the X11 socket is removed when the X server ends). But the ICE socket is owned by the user, just like xfce4-session and ksmserver run as user.

allend · 11-07-2023, 06:11 AM

If you look at the Inter-Client Exchange Library documentation, you will see that is deliberately designed to be non-blocking, and so does no state monitoring. It is expected that clients will handle connections.

Quote:

When it is known that the client on the other side of the ICE connection has terminated the connection without initiating shutdown negotiation, the IceSetShutdownNegotiation function should be called to turn off shutdown negotiation. This will prevent IceCloseConnection from writing to a broken connection.

Looking at ice-layer.c in XFCE, there is:

Quote:

static void
ice_error_handler (IceConn ice_conn)
{
/*
* The I/O error handlers does whatever is necessary to respond
* to the I/O error and then returns, but it does not call
* IceCloseConnection. The ICE connection is given a "bad IO"
* status, and all future reads and writes to the connection
* are ignored. The next time IceProcessMessages is called it
* will return a status of IceProcessMessagesIOError. At that
* time, the application should call IceCloseConnection.
*/
xfsm_verbose ("ICE connection fd = %d, ICE I/O error on connection\n",
IceConnectionNumber (ice_conn));
}

That is in accord with what is expected behaviour. From the first link:

Quote:

IceProcessMessagesIOError - an IO error occurred, and the caller must explicitly close the connection by calling IceCloseConnection

What libICE does not handle is the case when a client dies unexpectedly without running IceCloseConnection (e.g. glitch in software or immediate closure of X). The stale sockets are left behind. This might be called a bug in libICE, as it is responsible for the creation of the sockets, but does not perform connection maintenance.

So the user needs to deal with the accumulation of stale sockets. For people regularly rebooting, this is simply a matter of cleaning /tmp/.ICE-unix and /tmp/.X11-unix, which are automatically created when Slackware boots.
For users with long uptimes, a cron job to remove the stale sockets is the solution.

KDE Plasma is not affected as it uses the internal DBUS mechanism to perform the functions provided by libICE.

pan64 · 11-07-2023, 07:01 AM

Quote:

Originally Posted by allend

What libICE does not handle is the case when a client dies unexpectedly without running IceCloseConnection (e.g. glitch in software or immediate closure of X). The stale sockets are left behind. This might be called a bug in libICE, as it is responsible for the creation of the sockets, but does not perform connection maintenance.

No, libICE itself not, but the app, which uses this lib should close those sockets.
gdm also uses this lib (actually gdm-binary).

allend · 11-07-2023, 07:09 AM

Quote:

No, libICE itself not, but the app, which uses this lib should close those sockets.
gdm also uses this lib (actually gdm-binary).

Exactly, the app is expected to do it, but cannot because it has unexpectedly been crashed or closed.

0XBF · 11-07-2023, 07:44 AM

Quote:

Originally Posted by Petri Kaukasoina

The X11 socket /tmp/.X11-unix/X0 is owned by root and its server is Xorg, running as root (and the X11 socket is removed when the X server ends). But the ICE socket is owned by the user, just like xfce4-session and ksmserver run as user.

Is there any config file to tell libICE where to keep the user sockets? or is that hardcoded?

Seems to me like it would be beneficial to store them in the user's $XDG_RUNTIME_DIR so that they get purged on every session_close.

It looks like only the user sockets in /tmp/.ICE-unix get built up from unclean shutdowns on my machines, while the /tmp/.X11-unix directory is managed properly.

the3dfxdude · 11-07-2023, 08:09 AM

Quote:

Originally Posted by pan64

the interesting question: why is it not yet fixed? I don't really know the answer, there can be a few possibilities, like:
1. it is not that trivial at all
2. it is not that important at all
3. it is a bug, but not in the mentioned lib (cannot be fixed in that lib)
4. it is not a bug at all, but a misuse

Well, 5th possibility, but very similar to number 3, more generally the bug report filed to the wrong people.

Petri Kaukasoina · 11-07-2023, 09:20 AM

Quote:

Originally Posted by 0XBF

Is there any config file to tell libICE where to keep the user sockets? or is that hardcoded?

Seems to me like it would be beneficial to store them in the user's $XDG_RUNTIME_DIR so that they get purged on every session_close.

It's hard coded. In the source of libICE, libICE-1.1.1/src/icetrans.c includes c code from /usr/include/X11/Xtrans/Xtranssock.c which is installed by package xtrans:

Code:

PACKAGE DESCRIPTION:
xtrans: xtrans (X network transport library)
xtrans:
xtrans: xtrans is a library of code that is shared among various X packages
xtrans: to handle network protocol transport in a modular fashion, allowing a
xtrans: single place to add new transport types. It is used by the X server,
xtrans: libX11, libICE, the X font server, and related components.

The same c code is used when building other binaries using UNIX domain sockets in /tmp. For libICE, these are defined:

Code:

#if defined(ICE_t)
#define UNIX_PATH "/tmp/.ICE-unix/"
#define UNIX_DIR "/tmp/.ICE-unix"
#endif /* ICE_t */

Then futher down there are three places where UNIX_PATH is used. You could try to change those places (adding #if defined(ICE_t) in suitable places) to use a path read from $XDG_RUNTIME_DIR by getenv(), instead.

But maybe the socket needs to be available for everybody? They are srwxrwxrwx. I think sharing the cookie of file .ICEAuthority, programs running under different UIDs can talk to each other using the ICE socket. It would not be possible using $XDG_RUNTIME_DIR because of its mode bits.

GazL · 11-07-2023, 09:39 AM

If you're using a display manager such as xdm then an alternative to a cron job is to add the cleanup to the startup script (note: startup, not xsession) it runs.
Xstartup is run as root, so permissions aren't a problem.

'startx' is a bigger issue as there's no way to deal with other peoples left over sockets without getting into suid root or 'sudo' shenanigans.

henca · 11-07-2023, 01:50 PM

Quote:

Originally Posted by Petri Kaukasoina

On one machine here there were lots of sockets in /tmp/.ICE-unix. I created file /etc/cron.hourly/ice and made it executable chmod +x /etc/cron.hourly/ice.

Code:

#!/bin/sh
find /tmp/.ICE-unix -maxdepth 1 -type s ! -newerat now-30mins ! -exec fuser -s '{}' \; -delete

Cron runs it once an hour and it deletes sockets from /tmp/.ICE-unix older than 30 mins which are not accessed at the moment. It did remove all stale sockets and left the one xfce was using.

I have now done some testing on a number of virtual and physical machines running different versions of Slackware and ended up modifying your nice /etc/cron.hourly/ice script:

Code:

#!/bin/sh
# Clean up ICE sockets left which oterwise might prevent other users to start X.
find /tmp/.ICE-unix -maxdepth 1 -type s ! -newerat now-30mins ! -exec fuser -s '{}' \; -delete >& /dev/null

# If the .ICE-unix directory has been lost for some reason, restore it:
mkdir -p -m 1777 /tmp/.ICE-unix

My most important modification was to redirect the stderr output of find to /dev/null. On some machines the cron job did result in noise sent back to me as email. I was able to see at least two different kinds of error messages:

Some machines complained that there was no /tmp/.ICE-unix directory. This made me add the mkdir row to the script as that directory is supposed to be there.
Some machines with home directories on NFS servers made fuser spit out error messages like below:

Code:

Cannot stat file /proc/22009/fd/1: Stale NFS file handle
Cannot stat file /proc/22009/fd/2: Stale NFS file handle
Cannot stat file /proc/22117/fd/1: Stale NFS file handle
Cannot stat file /proc/22117/fd/2: Stale NFS file handle

This might happen on NFS drives, and mostly it happens this way:

A user logs in through kdm or tdm which sends all output of stdout and stderr to ~/.xsession-errors
The user then also logs in to another machine where a new .xsession-errors is created in the same NFS mounted home directory.
The first .xsession-errors will be deleted before the new one is created.
The deleted file, still being opened on another machine in the network will be renamed to .nfsXXXX
On the first machine, /proc/PID/fd/1 still points to .xsession-errors

Having fuser complaining in such a situation might be a little annoying, but redirecting those complaints to /dev/null works fine for this cron job.

The script seems to work fine. It does not only remove obsolete /tmp/.ICE-unix/PID-sockets owned by users, but also obsolete /tmp/.ICE-unix/dcop-sockets owned by root.

I have tested on Slackware 15.0, 14.2, 14.1 and some older versions of Slackware, the oldest one Slackware 12.0.

I also tried all different desktop environments and window managers in Slackware 15.0 and some more. KDE, XFCE and TDE (third party desktop environment not included in Slackware) produced those PID-sockets, but only XFCE allways left them behind after exit. None of the simple Window managers like twm, fvwm, windowmaker, fluxbox, blackbox or i3 (not included in Slackware) created those sockets.

regards Henrik

selfprogrammed · 11-09-2023, 07:31 AM

My original post had more information, but CloudFlare interfered, so that posting was lost. The current OP is a shorter version that I re-wrote while fighting with CloudFlare to get it posted.

I have been trying to track down the testing logs that verify what if remember.
I cannot verify how I determined that Xorg created the socket. I worked on this since mid Sept, and there are too many logs to find anything anymore.
I had instrumented my startx and startxfce4 so that it logged every step of starting.
At this time, I believe that the socket create error occurred after Xorg was starting. Because Xorg calls so many other programs, it becomes un-clear again.

Later xfce4 realized that the socket was not there, and aborted. I can see that in the error messages.

I did find a Xorg log from a failed startx, and that show Xorg starting up, and getting almost everything setup, and then aborting.

*** Source code
I went through various X-windows source tar files, looking for the source of the error messages that I got on the screen when startx failed.
This is where I found the /tmp/.ICE-unix directory.

I found that the ICE does not create the sockets itself. It uses Xtrans. Xtrans is setup in a strange manner. ICE sets a macro "TRANS", and the trans code uses that macro to append "IceTrans" to all its function names. That is why the error message names cannot be found by normal searching.

There are many ICE strings in /usr/bin/xfce4-session, BUT SocketCreateListener is not there. SocketUNIXCreateListener is not there.
It appears to be using functions that connect to an existing socket.

I did a grep of programs and libraries.
The ICE function that fails is IceTransSocketUNIXCreateListener (see OP).
Grep found SocketCreateListener, and SocketUNIXCreateListener in these files:
/usr/bin/Xdmx
/usr/bin/Xnest
/usr/bin/Xvfb
/usr/bin/Xwayland

The libraries that have SocketUNIXCreateListener are:
/usr/lib/libFS.so.6.0.0
/usr/lib/libICE.so.6.3.0
/usr/lib/libX11.so.6.4.0
/usr/lib/libXfont2.so.2.0.0

Notable that all the xfce4 binaries and libraries are not in the above. If it is invoking SocketUNIXCreateListener, it must be doing it indirectly.
Next would to be to untar that source code and look for it directly.

( Examination of libFS source reveals that it invokes Xtrans, with the prefix string "FSTrans". This would be creating a different socket.
There is also a X11Trans, a FontTrans, a XSERVTrans, and a XimTrans. All these would generate a SocketUNIXCreateListener function. )

*** dmseg log
I did find this log of a failed startx.

It shows that xfce4 had got screensaver started.

Code:

**** startx first time, will fail due to ICE
Oct 20 23:17:42 systemx acpid: client connected from 1116[0:1100]
Oct 20 23:17:42 systemx acpid: 1 client rule loaded
Oct 20 23:17:47 systemx dbus-daemon[1164]: [session uid=7xx pid=1162] Activating service name='org.a11y.Bus' requested by ':1.0' (uid=7xx pid=1121 comm="xfce4-session ")
Oct 20 23:17:47 systemx dbus-daemon[1164]: [session uid=7xx pid=1162] Successfully activated service 'org.a11y.Bus'
Oct 20 23:17:47 systemx dbus-daemon[1164]: [session uid=7xx pid=1162] Activating service name='org.xfce.Xfconf' requested by ':1.2' (uid=7xx pid=1121 comm="xfce4-session ")
Oct 20 23:17:47 systemx dbus-daemon[1164]: [session uid=7xx pid=1162] Successfully activated service 'org.xfce.Xfconf'
Oct 20 23:17:47 systemx dbus-daemon[913]: [system] Activating service name='org.freedesktop.PolicyKit1' requested by ':1.7' (uid=7xx pid=1121 comm="xfce4-session ") (using servicehelper)
Oct 20 23:17:47 systemx polkitd[1185]: Started polkitd version 0.120
Oct 20 23:17:47 systemx dbus-daemon[913]: [system] Successfully activated service 'org.freedesktop.PolicyKit1'
Oct 20 23:17:47 systemx dbus-daemon[1164]: [session uid=7xx pid=1162] Activating service name='org.xfce.ScreenSaver' requested by ':1.2' (uid=7xx pid=1121 comm="xfce4-session ")
Oct 20 23:17:48 systemx dbus-daemon[1164]: [session uid=7xx pid=1162] Activating service name='org.gtk.vfs.Daemon' requested by ':1.5' (uid=7xx pid=1198 comm="/usr/bin/xfce4-screensaver ")
Oct 20 23:17:48 systemx dbus-daemon[1164]: [session uid=7xx pid=1162] Successfully activated service 'org.gtk.vfs.Daemon'
Oct 20 23:17:48 systemx kernel: fuse: init (API version 7.34)
Oct 20 23:17:48 systemx dbus-daemon[1164]: [session uid=7xx pid=1162] Successfully activated service 'org.xfce.ScreenSaver'
**** Point of failure



**** startx second time, will be successful.
  The start time of Xorg.log is Oct 20 23:34:53
  pid=1185 is polkitd currently running
  pid=1337 is startx currently running, 
  pid=1377 is Xorg currently running
  pid=1382 is xfce4-session currently running
  pid=1423 is defunct
  pid=1425 is dbus-daemon currently running
  pid=1494 is xfdesktop
Oct 20 23:34:53 systemx acpid: client connected from 1377[0:1100]
Oct 20 23:34:53 systemx acpid: 1 client rule loaded
Oct 20 23:34:55 systemx dbus-daemon[1425]: [session uid=7xx pid=1423] Activating service name='org.a11y.Bus' requested by ':1.0' (uid=7xx pid=1382 comm="xfce4-session ")
Oct 20 23:34:55 systemx dbus-daemon[1425]: [session uid=7xx pid=1423] Successfully activated service 'org.a11y.Bus'
Oct 20 23:34:55 systemx dbus-daemon[1425]: [session uid=7xx pid=1423] Activating service name='org.xfce.Xfconf' requested by ':1.2' (uid=7xx pid=1382 comm="xfce4-session ")
Oct 20 23:34:55 systemx dbus-daemon[1425]: [session uid=7xx pid=1423] Successfully activated service 'org.xfce.Xfconf'
Oct 20 23:34:55 systemx dbus-daemon[1425]: [session uid=7xx pid=1423] Activating service name='org.xfce.ScreenSaver' requested by ':1.2' (uid=7xx pid=1382 comm="xfce4-session ")
Oct 20 23:34:55 systemx dbus-daemon[1425]: [session uid=7xx pid=1423] Activating service name='org.gtk.vfs.Daemon' requested by ':1.5' (uid=7xx pid=1448 comm="/usr/bin/xfce4-screensaver ")
Oct 20 23:34:55 systemx dbus-daemon[1425]: [session uid=7xx pid=1423] Successfully activated service 'org.gtk.vfs.Daemon'
Oct 20 23:34:55 systemx dbus-daemon[1425]: [session uid=7xx pid=1423] Successfully activated service 'org.xfce.ScreenSaver'
Oct 20 23:34:58 systemx dbus-daemon[913]: [system] Activating service name='org.freedesktop.Accounts' requested by ':1.14' (uid=7xx pid=1494 comm="xfdesktop --display :0.0 --sm-client-id 283744961-") (using servicehelper)
Oct 20 23:34:58 systemx dbus-daemon[1425]: [session uid=7xx pid=1423] Activating service name='org.freedesktop.Notifications' requested by ':1.15' (uid=7xx pid=1500 comm="xfce4-power-manager --restart --sm-client-id 25ffd")
Oct 20 23:34:59 systemx accounts-daemon[1511]: started daemon version 0.6.55
Oct 20 23:34:59 systemx dbus-daemon[913]: [system] Successfully activated service 'org.freedesktop.Accounts'

*** rc.6
I instrumented the rc.6 script to log which processes were still around to get killed.
I could NOT find the Xorg process, or any other X process in the log. They must have been terminated already.
Thus, I doubt we can blame rc.6 killing of processes either.

Petri Kaukasoina · 11-09-2023, 09:30 AM

Quote:

Originally Posted by selfprogrammed

Grep found SocketCreateListener, and SocketUNIXCreateListener in these files:
/usr/bin/Xdmx
/usr/bin/Xnest
/usr/bin/Xvfb
/usr/bin/Xwayland

And /usr/libexec/Xorg, /usr/bin/Xephyr and /usr/bin/Xvnc. All those different X servers create the X11 socket in /tmp/.X11-unix. Not in /tmp/.ICE-unix.

Quote:

The libraries that have SocketUNIXCreateListener are:
/usr/lib/libFS.so.6.0.0
/usr/lib/libICE.so.6.3.0
/usr/lib/libX11.so.6.4.0
/usr/lib/libXfont2.so.2.0.0

libFS.so and libXfont2.so create a socket in /tmp/.font-unix (a long time ago a fontserver was used) and libICE.so in /tmp/.ICE-unix. libX11.so refers to /tmp/.XIM-unix, whatever that is.

Quote:

Notable that all the xfce4 binaries and libraries are not in the above. If it is invoking SocketUNIXCreateListener, it must be doing it indirectly.

Yes, xfce4-session links to libICE.

Petri Kaukasoina · 11-09-2023, 10:09 AM

Quote:

Originally Posted by selfprogrammed

The libraries that have SocketUNIXCreateListener are:
/usr/lib/libICE.so.6.3.0

If you look at /usr/include/X11/Xtrans/Xtranssock.c, you will find this line:

Code:

        snprintf (tmpport, sizeof(tmpport), "%s%ld", UNIX_PATH, (long)getpid());

That line assembles the file name for the socket. libICE has defined ICE_t which results UNIX_PATH=/tmp/.ICE-unix/ and getpid() returns the process PID of the calling process. So you get /tmp/.ICE-unix/PID.

In SocketCreateListener, this line then assigns the name to the socket:

Code:

    while (bind (fd, (struct sockaddr *) sockname, namelen) < 0)

So, if you want to know which process created the socket, just look at the file name, the file name equals the PID. And that will be the PID of xfce4-session.