compat32 libs (multilib) not linking in xsession (run level 4)

This must certainly be an installation issue but it has been very difficult to debug/track down. I've done a fair amount of sleuthing with the linker. I'm soliciting ideas of things to try that I haven't already - i.e. before I throw in the towel and perform a complete reinstall of the OS.

I have a very old X-application that I have been running over the years with the help of multilib. In fact it is so old that it has to be built in Slackware 8 with libc version 5. The source will not build/compile on a modern systems - so I knew something like this would eventually crop up. I am pretty certain however this is an OS installation issue and not associated with the old code. In fact I know this as other "current" systems with essentially the same installation do not have this issue.

I normally call this application from a script - actually a .desktop file that then calls the script. The script basically sets some Xresouces before it calls the application. Because it is a script I was able to inject some debugging lines into it and see which libraries are/not linking via ldd.

One very relevant point is that this issue is only observed when X is started from the X desktop manager - i.e. run level 4. When X is started from the command line (run level 3) the application runs and links the compat32 libraries no issue. In fact it will run from my shell (via command line) in run level 4. Below are the all the libraries (from #ldd x-app) the application uses. I have enclosed the compat32 libs in the code-box which do not link (and thus the application fails to run) in run level 4:

linux-gate.so.1 (0xf7edc000)
libm.so.6 => /lib/libm.so.6 (0xf7d90000)
libXaw3d.so.6 => /usr/i386-linux-libc5/lib/libXaw3d.so.6 (0xf7d02000)
libXmu.so.6 => /usr/lib/libXmu.so.6 (0xf7ce7000)
libXt.so.6 => /usr/lib/libXt.so.6 (0xf7c85000)
libXext.so.6 => /usr/lib/libXext.so.6 (0xf7c71000)
libXpm.so.4 => /usr/lib/libXpm.so.4 (0xf7c5e000)
libSM.so.6 => /usr/lib/libSM.so.6 (0xf7c53000)
libICE.so.6 => /usr/lib/libICE.so.6 (0xf7c36000)
libX11.so.6 => /usr/lib/libX11.so.6 (0xf7af0000)
libresolv.so.2 => /lib/libresolv.so.2 (0xf7adb000)
libc.so.6 => /lib/libc.so.6 (0xf78dc000)
/lib/ld-linux.so.2 (0xf7ede000)

        libuuid.so.1 => /lib/libuuid.so.1 (0xf78d3000)
        libxcb.so.1 => /usr/lib/libxcb.so.1 (0xf78a7000)
        libXau.so.6 => /usr/lib/libXau.so.6 (0xf78a2000)
        libXdmcp.so.6 => /usr/lib/libXdmcp.so.6 (0xf789b000)

You should note this doesn't appear to be a library path issue as some libraries from /lib and /usr/lib are linking. I've checked the 4 missing libraries and confirmed they are from the compat32 packages of the multilib installation below. These of course have their 64-bit counterparts in /lib64 and /usr/lib64.

util-linux-compat32-2.38.1-x86_64-1compat32
libxcb-compat32-1.15-x86_64-1compat32
libXau-compat32-1.0.11-x86_64-1compat32
libXdmcp-compat32-1.1.4-x86_64-1compat32

I managed a work-around to force the linking (in an xsession) by setting the LD_LIBRARY_PATH variable inside my script (see below). This is an excerpt of /etc/profile.d/32dev.sh. It doesn't actually fix the underlying issue with run level 4. It is only a band-aid fix.

if [ ! "$LD_LIBRARY_PATH" = "" ]; then export LD_LIBRARY_PATH="/usr/local/lib:/lib:/usr/lib:$LD_LIBRARY_PATH" else export LD_LIBRARY_PATH="/usr/local/lib:/lib:/usr/lib" fi

I have reinstalled (via "upgradepkg --reinstall") both the sddm packages (sddm and sddm-kcm) and the multilib packages. Next in line is a complete reinstall of the X series packages. Hoping someone might point me to specific ones first. Thanks for any input.

What happens if you move this away (and run ldconfig):

/usr/i386-linux-libc5/lib/libXaw3d.so.6

As expected the application will not run without the Xaw3d library since it was compiled with it. Why would you ask this?

Well, the ldd output for your application did not list libc.so.5. There were only libc.so.6 and other familiar libraries which link to libc.so.6. Except /usr/i386-linux-libc5/lib/libXaw3d.so.6. The hypothesis was that it was a mistake, wrong link to a libc5 library, because you also have a libc6 version of it in /usr/lib/libXaw3d.so.6. When you set LD_LIBRARY_PATH=/usr/local/lib:/lib:/usr/lib, it told to find /usr/lib/libXaw3d.so.6 before /usr/i386-linux-libc5/lib/libXaw3d.so.6. And the libc6 version of /usr/lib/libXaw3d.so.6 loads libuuid.so.1, libxcb.so.1, libXau.so.6, libXdmcp.so.6, that you told did not link.

Anyway, you should not use ldd but this command to see what libraries the application binary really needs:

objdump -x /usr/bin/application|grep NEEDED

Does it need libc.so.6 or libc.so.5 ?

(I can't imagine how a binary would work linking to both libc.so.6 and a libc5 version of a shared library at the same time.)

It's possible that the dynamic linker can't figure out whether your libXaw3d.so.6 libraries are libc5 or libc6 versions. You can help it by giving it a hint. Something like this in /etc/ld.so.conf

/usr/lib=libc6
/usr/i386-linux-libc5/lib=libc5

Do you have libXaw3d-compat32-1.6.3-x86_64-3compat32 installed? I noticed that alien's collection of compat32 libraries does not have that. If not, build it with convertpkg-compat32. If you have libXaw3d-compat32 installed, you have /usr/lib/libXaw3d.so.6.

Petri, Thank you so very much for your keen insights into this. I might have mentioned that I do not have the libc6 (64-bit) version of the Athena 3d widgets library installed on my system. I removed it because it confuses the application (won't run) if this version precedes it in the library path. Only the carry-over (ported) version from libc5 is installed on my system for this reason. Also I probably should mention that this version of Xaw3d actually links to libc-2.2.3.so which although very old is actually a version of libc6 [libc.so.6 -> libc-2.2.3.so]. I am pretty sure the old application will not build with the compat32 version of Xaw3d library. But it might run with this version instead of my old one and so it is certainly worth converting and installing it to see if it will. When I run the objdump -x (filtering "NEEDED") on the application binary the output is as follows:

NEEDED libm.so.6
NEEDED libXaw3d.so.6
NEEDED libXmu.so.6
NEEDED libXt.so.6
NEEDED libXext.so.6
NEEDED libXpm.so.4
NEEDED libSM.so.6
NEEDED libICE.so.6
NEEDED libX11.so.6
NEEDED libresolv.so.2
NEEDED libc.so.6

It is very interesting that the compat32 versions above (libuuid,libxcb, libXau and libXdmcp) are apparently not used but that they do link through ldd. Any idea why?

On my system, running ldconfig -p | grep Xaw3 gives below:

libXaw3dxft.so.8 (libc6,x86-64) => /usr/lib64/libXaw3dxft.so.8
libXaw3dxft.so (libc6,x86-64) => /usr/lib64/libXaw3dxft.so
libXaw3d.so.6 (libc6) => /usr/i386-linux-libc5/lib/libXaw3d.so.6
libXaw3d.so (libc6) => /usr/i386-linux-libc5/lib/libXaw3d.so

So I was incorrect about libc5. I think the first versions of my application (xgospel) that ran were built on a libc5 version.

Originally Posted by wirelessmc NEEDED libm.so.6 NEEDED libXaw3d.so.6 NEEDED libXmu.so.6 NEEDED libXt.so.6 NEEDED libXext.so.6 NEEDED libXpm.so.4 NEEDED libSM.so.6 NEEDED libICE.so.6 NEEDED libX11.so.6 NEEDED libresolv.so.2 NEEDED libc.so.6 It is very interesting that the compat32 versions above (libuuid,libxcb, libXau and libXdmcp) are apparently not used but that they do link through ldd.

In your list there are libSM.so.6 and libX11.so.6. But libSM.so.6 loads libuuid.so.1, and libX11.so.6 loads libxcb.so.1. And libxcb.so.1 then loads both libXau.so.6 and libXdmcp.so.6. See for example 'objdump -x /usr/lib/libSM.so.6|grep NEEDED'

Ok that makes sense. I converted the 32-bit Xaw3d library to compat32 as you suggested and got the following when I tried to execute the old application. This is not surprising really since it was built with the older version of Xaw3d. I have tried building the application on relatively recent 32-bit systems. No luck here either. This code dates back to 1996 and the most recent versions are 1998.

Fatal glibc error: malloc.c:2593 (sysmalloc): assertion failed: (old_top == initial_top (av) && old_size == 0) || ((unsigned long) (old_size) >= MINSIZE && prev_inuse (old_top) && ((unsigned long) old_end & (pagesize - 1)) == 0)

BTW I have reinstalled the a, l, x and multilib/compat32 packages. Still no luck fixing this. I see other things that lead me to believe something got corrupt during the installation (from a USB stick). I have a local version of the slackware64-current repository on disk so I may just have to bite-the-bullet here and rebuild the system from scratch. This is more than a little irritating as many other things were built/packaged and installed subsequent to the installation. Thanks for all your help Petri.

Certainly this is for an audience of one (you) Petri. As even AlienBob said: "I cannot help you, this is quite an esoteric problem." After a complete re-installation of slackware64-current and still having the same issue, it dawned on me that this wasn't an installation issue after all (yikes!). However that said something changed in the slackware64-current base that made setting of library paths more strict. This was indeed a library path issue and the solution was pretty simple after all once I realized this. I just added the following lines in /etc/profile and /etc/csh.login and that resolved the issue. Somehow on older systems (even very recent systems) this was not needed. The Xsession shell must have inherited the library path from ld.so.conf in older systems. I think the Xsession shell can no longer do that. That may or may not be by design. I will bring it up in a different thread. Thanks again Petri!

--------added lines to /etc/profile-------

if [ ! "$LD_LIBRARY_PATH" = "" ]; then export LD_LIBRARY_PATH="/usr/local/lib:/lib:/usr/lib:/usr/i386-linux-libc5/lib:$LD_LIBRARY_PATH" else export LD_LIBRARY_PATH="/usr/local/lib:/lib:/usr/lib:/usr/i386-linux-libc5/lib" fi

------------------------------------------

-------added lines to /etc/csh.login------

if ($?LD_LIBRARY_PATH) then setenv LD_LIBRARY_PATH ${LD_LIBRARY_PATH}:/usr/local/lib:/usr/lib:/usr/i386-linux-libc5/lib else setenv LD_LIBRARY_PATH /usr/local/lib:/usr/lib:/usr/i386-linux-libc5/lib endif

------------------------------------------

Good that you solved it. Yes, quite an esoteric problem...