[SOLVED] 32 bit binaries do not run on 64 bit machine

Valery Reznic · 02-27-2011, 01:34 AM

Quote:

Originally Posted by frznchckn

[rfriesen@2 ~]$ ls -l /lib/libc.so.*
lrwxrwxrwx 1 root root 11 Feb 25 09:35 /lib/libc.so.6 -> libc-2.5.so
[rfriesen@~]$ ls -l /lib/ld-linux.so.*
lrwxrwxrwx 1 root root 9 Feb 25 09:35 /lib/ld-linux.so.2 -> ld-2.5.so

a "sudo find / -name linux-gate.so.*" turned up empty?

and what is output of

Code:

ls -l /lib/libc-2.5.so
ls -l /lib/ld-2.5.so

and

Code:

readelf -d hw32

frznchckn · 02-27-2011, 10:50 AM

[rfriesen@nsnjplpc2 c-test]$ gcc -v -m32 -o hw32 main.c
Using built-in specs.
Target: x86_64-redhat-linux
Configured with: ../configure --prefix=/usr --mandir=/usr/share/man --infodir=/usr/share/info --enable-shared --enable-threads=posix --enable-checking=release --with-system-zlib --enable-__cxa_atexit --disable-libunwind-exceptions --enable-libgcj-multifile --enable-languages=c,c++,objc,obj-c++,java,fortran,ada --enable-java-awt=gtk --disable-dssi --disable-plugin --with-java-home=/usr/lib/jvm/java-1.4.2-gcj-1.4.2.0/jre --with-cpu=generic --host=x86_64-redhat-linux
Thread model: posix
gcc version 4.1.2 20080704 (Red Hat 4.1.2-50)
/usr/libexec/gcc/x86_64-redhat-linux/4.1.2/cc1 -quiet -v main.c -quiet -dumpbase main.c -m32 -mtune=generic -auxbase main -version -o /tmp/ccwr3XTz.s
ignoring nonexistent directory "/usr/lib/gcc/x86_64-redhat-linux/4.1.2/../../../../x86_64-redhat-linux/include"
#include "..." search starts here:
#include <...> search starts here:
/usr/local/include
/usr/lib/gcc/x86_64-redhat-linux/4.1.2/include
/usr/include
End of search list.
GNU C version 4.1.2 20080704 (Red Hat 4.1.2-50) (x86_64-redhat-linux)
compiled by GNU C version 4.1.2 20080704 (Red Hat 4.1.2-50).
GGC heuristics: --param ggc-min-expand=100 --param ggc-min-heapsize=131072
Compiler executable checksum: ef754737661c9c384f73674bd4e06594
as -V -Qy --32 -o /tmp/ccK28NY1.o /tmp/ccwr3XTz.s
GNU assembler version 2.17.50.0.6-14.el5 (x86_64-redhat-linux) using BFD version 2.17.50.0.6-14.el5 20061020
/usr/libexec/gcc/x86_64-redhat-linux/4.1.2/collect2 --eh-frame-hdr -m elf_i386 --hash-style=gnu -dynamic-linker /lib/ld-linux.so.2 -o hw32 /usr/lib/gcc/x86_64-redhat-linux/4.1.2/../../../../lib/crt1.o /usr/lib/gcc/x86_64-redhat-linux/4.1.2/../../../../lib/crti.o /usr/lib/gcc/x86_64-redhat-linux/4.1.2/32/crtbegin.o -L/usr/lib/gcc/x86_64-redhat-linux/4.1.2/32 -L/usr/lib/gcc/x86_64-redhat-linux/4.1.2/32 -L/usr/lib/gcc/x86_64-redhat-linux/4.1.2/../../../../lib -L/lib/../lib -L/usr/lib/../lib /tmp/ccK28NY1.o -lgcc --as-needed -lgcc_s --no-as-needed -lc -lgcc --as-needed -lgcc_s --no-as-needed /usr/lib/gcc/x86_64-redhat-linux/4.1.2/32/crtend.o /usr/lib/gcc/x86_64-redhat-linux/4.1.2/../../../../lib/crtn.o

---------- Post added 2011-02-27 at 08:51 ----------

[rfriesen@nsnjplpc2 c-test]$ ls -l /lib/libc-2.5.so
-rwxr-xr-x 1 root root 1690380 Dec 14 09:26 /lib/libc-2.5.so
[rfriesen@nsnjplpc2 c-test]$ ls -l /lib/ld-2.5.so
-rwxr-xr-x 1 root root 128596 Dec 14 09:26 /lib/ld-2.5.so
[rfriesen@nsnjplpc2 c-test]$ readelf -d hw32

Dynamic section at offset 0x534 contains 20 entries:
Tag Type Name/Value
0x00000001 (NEEDED) Shared library: [libc.so.6]
0x0000000c (INIT) 0x804826c
0x0000000d (FINI) 0x8048478
0x6ffffef5 (GNU_HASH) 0x8048168
0x00000005 (STRTAB) 0x80481d8
0x00000006 (SYMTAB) 0x8048188
0x0000000a (STRSZ) 74 (bytes)
0x0000000b (SYMENT) 16 (bytes)
0x00000015 (DEBUG) 0x0
0x00000003 (PLTGOT) 0x8049600
0x00000002 (PLTRELSZ) 24 (bytes)
0x00000014 (PLTREL) REL
0x00000017 (JMPREL) 0x8048254
0x00000011 (REL) 0x804824c
0x00000012 (RELSZ) 8 (bytes)
0x00000013 (RELENT) 8 (bytes)
0x6ffffffe (VERNEED) 0x804822c
0x6fffffff (VERNEEDNUM) 1
0x6ffffff0 (VERSYM) 0x8048222
0x00000000 (NULL) 0x0
[rfriesen@nsnjplpc2 c-test]$

Valery Reznic · 02-27-2011, 11:06 AM

Could you post you hw32 here?

johnsfine · 02-27-2011, 02:42 PM

Quote:

Originally Posted by Valery Reznic

Could you post you hw32 here?

I assume you mean frznchckn should attach the binary to a post (I think that is done by the paperclip icon when posting, but I've never tried it in this forum).

We do seem to be failing to distinguish whether something is wrong at link time vs. something is wrong both at load time and when ldd is run. So letting us examine the executable on our own systems should allow us to at least find decide that boundary.

Everything posted in post #20 looks correct to me, so I no longer believe anything went wrong at link time.

I don't know how to even investigate what might have gone wrong at load time. The usual problem is looking for .so files that aren't there. But we have already seen both that the correct .so files are present and that ldd thinks incorrect .so files are requested. Further, post #20 shows us the correct .so files are requested at link time.

Just to check a few more obscure possibilities, you should use the file -L command on the files referenced by the link command. I've copied those file names out of your link command to make that easier

Code:

file -L /lib/ld-linux.so.2
file -L /usr/lib/gcc/x86_64-redhat-linux/4.1.2/../../../../lib/crt1.o
file -L /usr/lib/gcc/x86_64-redhat-linux/4.1.2/../../../../lib/crti.o
file -L /usr/lib/gcc/x86_64-redhat-linux/4.1.2/32/crtbegin.o
file -L /usr/lib/gcc/x86_64-redhat-linux/4.1.2/32/libgcc*

With a quick google search, the best overview of loading I found was here:
http://www.cs.virginia.edu/~dww4s/ar.../ld_linux.html

frznchckn · 02-27-2011, 02:43 PM

You wanted the source right?

#include <stdio.h>

int main() {
printf("Hello World\n");

return 0;
}

johnsfine · 02-27-2011, 02:55 PM

We saw source earlier. I think that request was to attach the binary.

Valery Reznic · 02-27-2011, 02:58 PM

Quote:

Originally Posted by frznchckn

You wanted the source right?

#include <stdio.h>

int main() {
printf("Hello World\n");

return 0;
}

No, I want binary (you already posted source in the post #2)

Also could you please run your executable under strace:

Code:

strace -i ./hw32

and post here output?

frznchckn · 02-27-2011, 03:22 PM

LQ only accepts certain file extensions, so the hw32.txt is really just the hw32 binary.

[rfriesen@nsnjplpc2 c-test]$ strace ./hw32
execve("./hw32", ["./hw32"], [/* 25 vars */]) = 0
brk(0) = 0xcce5000
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x2abdf3844000
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x2abdf3845000
access("/etc/ld.so.preload", R_OK) = -1 ENOENT (No such file or directory)
open("/etc/ld.so.cache", O_RDONLY) = 3
fstat(3, {st_mode=S_IFREG|0644, st_size=89144, ...}) = 0
mmap(NULL, 89144, PROT_READ, MAP_PRIVATE, 3, 0) = 0x2abdf3846000
close(3) = 0
open("/lib64/libc.so.6", O_RDONLY) = 3
read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\220\332\241:6\0\0\0"..., 832) = 832
fstat(3, {st_mode=S_IFREG|0755, st_size=1722304, ...}) = 0
mmap(0x363aa00000, 3502424, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x363aa00000
mprotect(0x363ab4e000, 2097152, PROT_NONE) = 0
mmap(0x363ad4e000, 20480, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x14e000) = 0x363ad4e000
mmap(0x363ad53000, 16728, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0x363ad53000
close(3) = 0
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x2abdf385c000
arch_prctl(ARCH_SET_FS, 0x2abdf385c6e0) = 0
mprotect(0x363ad4e000, 16384, PROT_READ) = 0
mprotect(0x363a81b000, 4096, PROT_READ) = 0
munmap(0x2abdf3846000, 89144) = 0
brk(0) = 0xcce5000
brk(0xcd06000) = 0xcd06000
open("/usr/lib/locale/locale-archive", O_RDONLY) = 3
fstat(3, {st_mode=S_IFREG|0644, st_size=56413888, ...}) = 0
mmap(NULL, 56413888, PROT_READ, MAP_PRIVATE, 3, 0) = 0x2abdf385d000
close(3) = 0
fstat(1, {st_mode=S_IFCHR|0620, st_rdev=makedev(136, 1), ...}) = 0
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x2abdf6e2a000
write(1, "./hw32\n", 7./hw32
) = 7
close(1) = 0
munmap(0x2abdf6e2a000, 4096) = 0
exit_group(0) = ?

Valery Reznic · 02-27-2011, 03:40 PM

I downloaded hw32 and run it on my (32-bit) box. It;s running just fine, and print 'Hello World', as it should.

Also I noted strace output shows that hw32 for some reason tryied to mmap 64-bit library - /lib64/libc.so.6

Very, very, strange. May be your ld.so.cache file messed up. Could you move /etc/ld.so.cache aside

Code:

mv /etc/ld.so.cache /etc/ld.so.cache.orig

and then run you hw32 again under strace

Code:

strace -i ./hw32

frznchckn · 02-27-2011, 03:56 PM

[rfriesen@nsnjplpc2 c-test]$ sudo mv /etc/ld.so.cache /etc/ld.so.cache.orig
[sudo] password for rfriesen:
[rfriesen@nsnjplpc2 c-test]$ ./hw32
./hw32
[rfriesen@nsnjplpc2 c-test]$ strace -i ./hw32
[ 363aa9a887] execve("./hw32", ["./hw32"], [/* 25 vars */]) = 0
[ 363a6139ea] brk(0) = 0x1eb9a000
[ 363a613ffa] mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x2b904b225000
[ 363a613ffa] mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x2b904b226000
[ 363a613f37] access("/etc/ld.so.preload", R_OK) = -1 ENOENT (No such file or directory)
[ 363a613e77] open("/etc/ld.so.cache", O_RDONLY) = -1 ENOENT (No such file or directory)
[ 363a613e77] open("/lib64/tls/x86_64/libc.so.6", O_RDONLY) = -1 ENOENT (No such file or directory)
[ 363a613de5] stat("/lib64/tls/x86_64", 0x7fffc007eb00) = -1 ENOENT (No such file or directory)
[ 363a613e77] open("/lib64/tls/libc.so.6", O_RDONLY) = -1 ENOENT (No such file or directory)
[ 363a613de5] stat("/lib64/tls", 0x7fffc007eb00) = -1 ENOENT (No such file or directory)
[ 363a613e77] open("/lib64/x86_64/libc.so.6", O_RDONLY) = -1 ENOENT (No such file or directory)
[ 363a613de5] stat("/lib64/x86_64", 0x7fffc007eb00) = -1 ENOENT (No such file or directory)
[ 363a613e77] open("/lib64/libc.so.6", O_RDONLY) = 3
[ 363a613ed7] read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\220\332\241:6\0\0\0"..., 832) = 832
[ 363a613e34] fstat(3, {st_mode=S_IFREG|0755, st_size=1722304, ...}) = 0
[ 363a613ffa] mmap(0x363aa00000, 3502424, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x363aa00000
[ 363a614057] mprotect(0x363ab4e000, 2097152, PROT_NONE) = 0
[ 363a613ffa] mmap(0x363ad4e000, 20480, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x14e000) = 0x363ad4e000
[ 363a613ffa] mmap(0x363ad53000, 16728, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0x363ad53000
[ 363a613ea7] close(3) = 0
[ 363a613ffa] mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x2b904b227000
[ 363a6022df] arch_prctl(ARCH_SET_FS, 0x2b904b227390) = 0
[ 363a614057] mprotect(0x363ad4e000, 16384, PROT_READ) = 0
[ 363a614057] mprotect(0x363a81b000, 4096, PROT_READ) = 0
[ 363aacc90a] brk(0) = 0x1eb9a000
[ 363aacc90a] brk(0x1ebbb000) = 0x1ebbb000
[ 363aac5ca0] open("/usr/lib/locale/locale-archive", O_RDONLY) = 3
[ 363aac5454] fstat(3, {st_mode=S_IFREG|0644, st_size=56413888, ...}) = 0
[ 363aad0cfa] mmap(NULL, 56413888, PROT_READ, MAP_PRIVATE, 3, 0) = 0x2b904b228000
[ 363aa2868b] close(3) = 0
[ 363aac5454] fstat(1, {st_mode=S_IFCHR|0620, st_rdev=makedev(136, 1), ...}) = 0
[ 363aad0cfa] mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x2b904e7f5000
[ 363aac6420] write(1, "./hw32\n", 7./hw32
) = 7
[ 363aac6330] close(1) = 0
[ 363aad0d27] munmap(0x2b904e7f5000, 4096) = 0
[ 363aa9a85f] exit_group(0) = ?
[rfriesen@nsnjplpc2 c-test]$

Valery Reznic · 02-27-2011, 04:16 PM

Your hw32 still tried to load 64 bit libraries.
Hmm. I copied your hw32 to 64-bit box and it's run OK, so I think we can rule out
linking problem and say, that problem in your environment.

Also when I run hw32 on my 64-bt box under strace I got following:

Code:

execve("./hw32.txt", ["./hw32.txt"], [/* 30 vars */]) = 0
[ Process PID=32556 runs in 32 bit mode. ]
brk(0)                                  = 0x88b0000

I.e on my box starce knows that it run 32-bit process, and on yours - not.

What happened when you run

Code:

/lib/ld-linux.so.2 ./hw32

frznchckn · 02-27-2011, 04:25 PM

[rfriesen@nsnjplpc2 c-test]$ /lib/ld-linux.so.2 ./hw32
/lib/ld-linux.so.2 ./hw32

Valery Reznic · 02-27-2011, 04:41 PM

Quote:

Originally Posted by frznchckn

[rfriesen@nsnjplpc2 c-test]$ /lib/ld-linux.so.2 ./hw32
/lib/ld-linux.so.2 ./hw32

Hmmm. Looks like when one tried to run 32-bit executable nothing happened, just command line displayed.

what is output of

Code:

./hw 1

Code:

./hw 1 2

Also hat is output of

Code:

ls /proc/sys/fs/binfmt_misc/

May be someone played with binfmt_mis on your box?

And I am go to sleep...

frznchckn · 02-27-2011, 04:51 PM

[rfriesen@nsnjplpc2 c-test]$ ./hw32 1
./hw32 1
[rfriesen@nsnjplpc2 c-test]$ ./hw32 1 2
./hw32 1 2
[rfriesen@nsnjplpc2 c-test]$ ls /proc/sys/fs/binfmt_misc/
32bits register status

johnsfine · 02-27-2011, 06:58 PM

I haven't read them carefully yet, but the pages found on a google search for
"/proc/sys/fs/binfmt_misc/32bits"
seem to say there is a security flaw in 32 bit support for 64 bit RHEL and a work around to disable 32 bit support, and your system seems to include that disable of 32 bit support.

https://access.redhat.com/kb/docs/DOC-40265

http://forums.cpanel.net/f185/x86_64...tml#post692582

Those say the following command as root will re enable 32 bit exectuables. Note I don't understand enough about the original security exploit to comment constructively on whether there is a good reason to leave 32 bit support disabled.

echo -1 > /proc/sys/fs/binfmt_misc/32bits

The whole discussion looks old enough to expect the original security flaw has been fixed correctly (if you reboot after installing a kernel update) and you are seeing the symptoms of a drastic patch that you no longer even need. But all of that is guesswork. I don't really know any status on RHEL kernel updates.

Quote:

Originally Posted by Valery Reznic

Code:

ls /proc/sys/fs/binfmt_misc/

And I am go to sleep...

Right before going to sleep you solved the mystery. Thanks, as often happens I got to learn more while trying to help someone than I learn working on my own issues.

I took your answer a tiny bit further, hopefully far enough for the OP to understand it.