question about different checksum of binaries

marozsas · 08-20-2010, 10:07 AM

Dears,

We have 4 servers running CentOS 5.5.

I realize some binaries between systems have different checksums even they have the same package.

In fact, copying the binaries from each system to another machine running a live distro (based on a DVD) and calculating the checksums, they are different. (this is just to be sure theses calculations are done in a machine not compromised)

the facts: (below, I will show only two files from distinct packages from 2 systems - but in fact, this happens with several files, from several packages, in at least 4 systems)

server A and server B both have the same output: (there is no output from the last command)

Code:

# rpm -qf --queryformat "name: %{NAME} version:%{VERSION} rel:%{RELEASE} arch: %{ARCH}\n" /usr/bin/sha1sum /usr/bin/elinks
name: coreutils version:5.97 rel:23.el5_4.1 arch: x86_64
name: elinks version:0.11.1 rel:6.el5_4.1 arch: x86_64
# ls -l /usr/bin/sha1sum /usr/bin/elinks
-rwxr-xr-x 1 root root 1089264 Oct 26  2009 /usr/bin/elinks
-rwxr-xr-x 1 root root   32472 Oct 27  2009 /usr/bin/sha1sum
# rpm -qV coreutils elinks
#

server A, sha1sum

Code:

a394074b596385600efab838e34586b9a7f7ae6a  /usr/bin/elinks
49d26e198a5660608f101c68630b4acc7674bc41  /usr/bin/sha1sum

server B, sha1sum

Code:

af456681b4493034ed159c84cf120ea82dca9bda  /usr/bin/elinks
6ecd6554e088b60c9015ccf81835347909af8bc2  /usr/bin/sha1sum

As additional information, both systems have chkrootkit and rkhunter (with a check followed by a --propupd after each system update) and none inform any problem.

I don't have a exhaustive list of different checksums in all 4 systems. But some files I randomly check are:

Code:

usr/bin/elinks
usr/bin/find
usr/sbin/lsof
usr/bin/passwd
usr/bin/killall
usr/bin/pstree
usr/bin/pstree.x11
bin/rpm
usr/bin/rpm2cpio
usr/bin/rpmdb
usr/bin/rpmquery
usr/bin/rpmsign
usr/bin/rpmverify
bin/mountpoint
sbin/init
sbin/sulogin
bin/mount
usr/sbin/vipw
usr/bin/wget
usr/bin/sha1sum
usr/bin/md5sum

Also, the machines I got the data above were off (unplugged from outlet) since Feb 15 2010 (they are spares) with the last system update in Feb 10 2010, and all 4 have SELinux enable and in enforcing mode, not directly attached to internet, not serving anything outside our restricted LAN, with firewall enabled and allowing only DNS, DHCP, SAMBA, SSH.

so, the verify rpm returns ok, the files have the same size, the packages are the same but even so, they have different checksums (also using md5). How this is possible ? What is the explanation ?

unSpawn · 08-20-2010, 12:17 PM

Prelinking?

marozsas · 08-20-2010, 01:12 PM

uhmm....I'm not sure how prelink will change the checksum...the results are the same after a prelink on /usr/bin/elinks /usr/bin/sha1sum

on system A:

Code:

# sha1sum /usr/bin/elinks /usr/bin/sha1sum
a394074b596385600efab838e34586b9a7f7ae6a  /usr/bin/elinks
49d26e198a5660608f101c68630b4acc7674bc41  /usr/bin/sha1sum
# prelink /usr/bin/elinks /usr/bin/sha1sum
# sha1sum /usr/bin/elinks /usr/bin/sha1sum
a394074b596385600efab838e34586b9a7f7ae6a  /usr/bin/elinks
49d26e198a5660608f101c68630b4acc7674bc41  /usr/bin/sha1sum
#

on system B:

Code:

# sha1sum /usr/bin/elinks /usr/bin/sha1sum
af456681b4493034ed159c84cf120ea82dca9bda  /usr/bin/elinks
6ecd6554e088b60c9015ccf81835347909af8bc2  /usr/bin/sha1sum
# prelink /usr/bin/elinks /usr/bin/sha1sum
# sha1sum /usr/bin/elinks /usr/bin/sha1sum
af456681b4493034ed159c84cf120ea82dca9bda  /usr/bin/elinks
6ecd6554e088b60c9015ccf81835347909af8bc2  /usr/bin/sha1sum
#

they still are different....

unSpawn · 08-20-2010, 01:40 PM

Quote:

Originally Posted by marozsas

I'm not sure how prelink will change the checksum

On both systems running 'rpm -q --dump elinks | grep `prelink -y --md5 /usr/bin/elinks | awk '{print $1}'`;' might explain it?

anomie · 08-20-2010, 01:47 PM

@marozsas:

IIRC, prelink does change the cryptographic hash. (At very least it changes mtime.) On your test above, you may be comparing already "prelinked" binaries.

For this reason, prelink is disabled (/etc/sysconfig/prelink) on my systems that are running a HIDS.

unSpawn · 08-20-2010, 02:57 PM

Quote:

Originally Posted by anomie

For this reason, prelink is disabled (/etc/sysconfig/prelink) on my systems that are running a HIDS.

Ah ;-p Samhain supports prelink and so does Aide (please read whole thread for details).

anomie · 08-20-2010, 03:02 PM

Hmm, thanks for the patch. Can't promise that I'll apply it, but it is good to be aware of.

marozsas · 08-20-2010, 04:03 PM

hi anomie, hi unspawn,

@unspawn: so, what I understood is you want to check if the md5sum returned by rpm is the same ms5 returned by prelink, right ?

here a simple script I used on both systems to return the info you asked:

Code:

#!/bin/bash
echo prelink -y --md5 /usr/bin/elinks
prelink -y --md5 /usr/bin/elinks
echo prelink -y --md5 /usr/bin/md5sum
prelink -y --md5 /usr/bin/md5sum
echo rpm -q --dump elinks pipe grep /usr/bin/elinks
rpm -q --dump elinks | grep /usr/bin/elinks
echo rpm -q --dump coreutils pipe grep /usr/bin/md5sum
rpm -q --dump coreutils | grep /usr/bin/md5sum
echo /usr/bin/md5sum /usr/bin/md5sum /usr/bin/elinks
/usr/bin/md5sum /usr/bin/md5sum /usr/bin/elinks

on server A:

Code:

prelink -y --md5 /usr/bin/elinks
65a265294f2f403dd4624835d20bacc9  /usr/bin/elinks
prelink -y --md5 /usr/bin/md5sum
c1f12e1919f578d75892e0ad9c841851  /usr/bin/md5sum
rpm -q --dump elinks pipe grep /usr/bin/elinks
/usr/bin/elinks 1064856 1256603845 65a265294f2f403dd4624835d20bacc9 0100755 root root 0 0 0 X
rpm -q --dump coreutils pipe grep /usr/bin/md5sum
/usr/bin/md5sum 26152 1256609348 c1f12e1919f578d75892e0ad9c841851 0100755 root root 0 0 0 X
/usr/bin/md5sum /usr/bin/md5sum /usr/bin/elinks
b4b220fcd68c43d13c2196499befa168  /usr/bin/md5sum
5c97ff530716f766115efee5918b2324  /usr/bin/elinks

server B:

Code:

prelink -y --md5 /usr/bin/elinks
65a265294f2f403dd4624835d20bacc9  /usr/bin/elinks
prelink -y --md5 /usr/bin/md5sum
c1f12e1919f578d75892e0ad9c841851  /usr/bin/md5sum
rpm -q --dump elinks pipe grep /usr/bin/elinks
/usr/bin/elinks 1064856 1256603845 65a265294f2f403dd4624835d20bacc9 0100755 root root 0 0 0 X
rpm -q --dump coreutils pipe grep /usr/bin/md5sum
/usr/bin/md5sum 26152 1256609348 c1f12e1919f578d75892e0ad9c841851 0100755 root root 0 0 0 X
/usr/bin/md5sum /usr/bin/md5sum /usr/bin/elinks
04a4b2e49ab156a66870573414e7f7b2  /usr/bin/md5sum
ed417d0a6cc3dd752ab0763627baf34a  /usr/bin/elinks

so, the md5 returned by rpm and by prelink ARE THE SAME in both systems ! uff...a relief indeed !
What remains strange is the md5 (sha1) returned by /usr/bin/md5sum (or /usr/bin/sha1sum) be different in each system (I've prelinked /usr/bin/md5sum and /usr/bin/sha1sum and /usr/bin/elinks just in case)

@anonie: this is new for me, thanks to make it clear. Let me understand: If I update a library in this system but not re-link a binary which depends on that library, the binary will return different hashs before and after prelink on the binary, that is right ?

anyway, why md5sum or sha1sum are returning different hashs in each system ? (of course, at this point we can assume the system is not compromised - at least the 2 systems were off for a long time - they were turned on for this investigation a few hours ago and unplugged from the network, just in case)

anomie · 08-23-2010, 12:38 PM

Quote:

Originally Posted by marozsas

this is new for me, thanks to make it clear. Let me understand: If I update a library in this system but not re-link a binary which depends on that library, the binary will return different hashs before and after prelink on the binary, that is right ?

Dunno if I can definitely speak to that exact scenario. I recommend that you test and check the results.

I can demonstrate this much (Fedora 13 system, BTW):

Code:

# openssl dgst -sha1 /bin/more
SHA1(/bin/more)= b7cc7f4108c81b3dc5b65e2b839e60cadb636d60

# prelink -v /bin
Laying out 52 libraries in virtual address space 00101000-50000000
...
Prelinking /bin/more
...

# openssl dgst -sha1 /bin/more
SHA1(/bin/more)= e709c416366f67f09238a21f1038bdb459d87f33