objdump might prove... enlightening. Particularly a diff of objdump of the "bad" versions and the known-good version.

I've set this up so we will see what happens.Not yet. I only got it setup yesterday and am still tinkering with the configuration. I'm getting data back via BASE, but nothing signifigant yet. I've downloaded the most recent set of snort rules, but I'm still trying to figure out how to install them.English please? j/k Never done it, but I'm willing to give anything a shot to keep from recreating the same scenario that allowed these guys in in the first place.OK, I read the man page for objdump to get some idea as to what I was looking at. There are signifigant differences between the known good version of /usr/sbin/tripwire and the cracked version of /usr/sbin/tripwire (surprise, surprise).

The header section of the cracked version shows some additional commands: DYNAMIC, EH_FRAME and STACK w/ what I'm assuming are memory addresses.

The cracked version shows lots of dynamic libraries being loaded that aren't loaded in the good version: libcrypto (which came up cracked yesterday), libm, libgcc_s, libc. It also has INIT, FINI, HASH, SYMENT w/ what I'm again ssuming are memory address (e.g. 0x804cc6c, 0x819d1e8, ...). It also has has hex address in the left column, e.g. 0x6ffffef9, 0x6fffef8, 0x6ffffdf7, ... The good version of tripwire does not have a Dynamic section.

The Sections sections shows some differences, but nothing significant.

The Symbol table shows the most dramatic differences. The good version does not have any symbols. The cracked version has many many symbols. Most of them start with the string "_Z" That will be followed by something that looks like code, e.g.: "_Z41__static_initialization_and_destruction_0ii" I've looked through these symbols and have not found anything of interest. So what does all of this mean?

OK, I've done some more checking with objdump and have found something interesting. The Dynamic Section of every binary that has been modified ends with the following lines:Does anyone have any idea what this means? Advice is appreciated.

Could you run ldd against the modified binaries to give a list of the linked files. Taking an otherwise normal app and linking it against a malicious library is an effective means of creating a trojan that escapes detection fairly easily. I believe those hex numbers are memory addresses where dynamically loaded objects would get loaded during execution. Definitely take note if apps are linked to libs that they shouldn't be, like ps getting linked to an evil libcrypto or something. Also how big is the average difference in file size. If it's large, you could have some kind of direct elf infection.

Ran this threw a few of them and nothing out of the ordinary came up. None of the memory addresses listed by objdump correspond to a library listed by ldd. Although, there is no telling what what ldd might have been modified to say. I've moved our apps and will be booting off of a Knoppix CD later today, so maybe that will reveal something more helpful, or at least more trustworthy. Did a comparison using hte first tripwire report that came up bad. Size diffs range from 42 bytes to 11k, average of 2.9k, median of 2k. The elf infection could be something to look b/c I've noticed a lot of libraries modified as well.

Snort hasn't found anything yet. I dump all net traffic from last night into some logs, so I'll be reviewing those today to see if there is anything odd in them. Will post back if/when I find anything.

Also, I've been looking over some previous tripwire reports. Turns out that the first major changes were on July 28 @ 4:02AM. I left for vacation that day, so that's probably why I missed it before. Most of the binaries modified on that date are the same ones in the list above, e.g. callback, sshd, stunnel, tripwire, etc. Lots of libraries changed at that point: librpm, libfontconfig, libtiff, libwvutils, libcurl... In that report, inodes, CRCs and MD5s changed, but very few modify times or sizes were changed.

This is actually quite interesting. What does a disassembly (objdump -d) show at the addresses from the Dynamic section? Which set of command line args did you use to retrieve those addresses?

For sshd, I actually get a slightly different set of addresses:Not sure, but I think what you are asking me is this: So, it doesn't look like anything is at those addresses.

Hrrm, what about the first half of the addresses? (0x6fff...) maybe grep for 6ffffe or 6ffffd?

Sorry, I should have mentioned that in my last post. I tried looking for those also and nothing was found.

That's very odd. objdump -x doesn't give me two columns of numbers in the Dynamic section, but rather one of words and one of numbers.

Right... I get the same thing, than all of the cracked binaries have lines like the above at the end of the dynamic section:

Well I think I may have found at least part of the answer. It seems my workstation is now exhibiting the same signs as this server. Many binaries are showing the same addresses at the end of the Dynamic section of objdump. I don't have any idea which was compromised first, but my workstation is being just as secretive at revealing information. The workstation is even stranger b/c it sits on a LAN behind a firewall, with all the same protections as servers. The only thing exposed to the outside world is SSH. We are a small company, so I know I'm the only with the know how to do something like this.

FWIW, I rebooted the server off of Knoppix this afternoon. I found nothing that I didn't already know, e.g. no hidden cron jobs, magical hidden files. Just the same memory addresses in objdump that were there before. I also reviewed all network traffic from last night and found nothing out of the ordinary, so it still doesn't appear that whoever did this is taking advantage of the boxes for anything malicious.

Looks like I'm in for a fun Friday.

Out of my own curiousity, would it be possible for you to create a tarball of your compromised binaries? I'd like to try some strace and disassembly work on them to see if I can figure out what they're supposed to do, and possibly even the origin of the binaries. To be honest, I'm mostly curious about their intent. Though, I imagine you're probably curious at this point too.

(I have a personal interest in computer forensics.)

I'm getting the feeling this might have something to do with prelinking. It should be run about the same time as yum by cron. All of the target dirs seem to appear in /etc/prelink.conf. Prelinking binaries can add entries to the end of the .dynamic section for virtual mem addresses locations. It also may explain why some test are coming back clean(those that don't look at the elf header) while others are flagging them (those that do check). Try undoing the prelinking with prelink -au binary and rerun tripwire.

That's actually a brilliant point. Wish I had thought about that.