LinuxQuestions.org
Share your knowledge at the LQ Wiki.
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - Security
User Name
Password
Linux - Security This forum is for all security related questions.
Questions, tips, system compromises, firewalls, etc. are all included here.

Notices


Reply
  Search this Thread
Old 08-15-2005, 11:20 PM   #31
Matir
LQ Guru
 
Registered: Nov 2004
Location: San Jose, CA
Distribution: Ubuntu
Posts: 8,507

Rep: Reputation: 125Reputation: 125

objdump might prove... enlightening. Particularly a diff of objdump of the "bad" versions and the known-good version.
 
Old 08-16-2005, 10:50 AM   #32
TruckStuff
Member
 
Registered: Apr 2002
Posts: 498

Original Poster
Rep: Reputation: 30
Quote:
Originally posted by Capt_Caveman
Again, the time of the modification suggests that it's a scripted routine running at the same time. It might be informative to replace procps package with a good version again and then set the immutable attribute (chattr +i filename) on one of the modified files and see what happens. Hopefully if it's some benign rogue process, then it may generate an error. Though it's interesting that logger was modified too...
I've set this up so we will see what happens.
Quote:
Originally posted by Capt_Caveman
Any luck with collecting data with Snort?
Not yet. I only got it setup yesterday and am still tinkering with the configuration. I'm getting data back via BASE, but nothing signifigant yet. I've downloaded the most recent set of snort rules, but I'm still trying to figure out how to install them.
Quote:
Originally posted by Capt_Caveman
How comfortable are you with disassembling binaries?
English please? j/k Never done it, but I'm willing to give anything a shot to keep from recreating the same scenario that allowed these guys in in the first place.
Quote:
Originally posted by Matir
objdump might prove... enlightening. Particularly a diff of objdump of the "bad" versions and the known-good version.
OK, I read the man page for objdump to get some idea as to what I was looking at. There are signifigant differences between the known good version of /usr/sbin/tripwire and the cracked version of /usr/sbin/tripwire (surprise, surprise).

The header section of the cracked version shows some additional commands: DYNAMIC, EH_FRAME and STACK w/ what I'm assuming are memory addresses.

The cracked version shows lots of dynamic libraries being loaded that aren't loaded in the good version: libcrypto (which came up cracked yesterday), libm, libgcc_s, libc. It also has INIT, FINI, HASH, SYMENT w/ what I'm again ssuming are memory address (e.g. 0x804cc6c, 0x819d1e8, ...). It also has has hex address in the left column, e.g. 0x6ffffef9, 0x6fffef8, 0x6ffffdf7, ... The good version of tripwire does not have a Dynamic section.

The Sections sections shows some differences, but nothing significant.

The Symbol table shows the most dramatic differences. The good version does not have any symbols. The cracked version has many many symbols. Most of them start with the string "_Z" That will be followed by something that looks like code, e.g.: "_Z41__static_initialization_and_destruction_0ii" I've looked through these symbols and have not found anything of interest. So what does all of this mean?
 
Old 08-16-2005, 02:21 PM   #33
TruckStuff
Member
 
Registered: Apr 2002
Posts: 498

Original Poster
Rep: Reputation: 30
OK, I've done some more checking with objdump and have found something interesting. The Dynamic Section of every binary that has been modified ends with the following lines:
Code:
  0x6ffffef9  0x8047a94
  0x6ffffdf7  0xf0
  0x6ffffef8  0x8047b84
  0x6ffffdf6  0x1e0
Does anyone have any idea what this means? Advice is appreciated.
 
Old 08-17-2005, 11:47 PM   #34
Capt_Caveman
Senior Member
 
Registered: Mar 2003
Distribution: Fedora
Posts: 3,658

Rep: Reputation: 69
Could you run ldd against the modified binaries to give a list of the linked files. Taking an otherwise normal app and linking it against a malicious library is an effective means of creating a trojan that escapes detection fairly easily. I believe those hex numbers are memory addresses where dynamically loaded objects would get loaded during execution. Definitely take note if apps are linked to libs that they shouldn't be, like ps getting linked to an evil libcrypto or something. Also how big is the average difference in file size. If it's large, you could have some kind of direct elf infection.
 
Old 08-18-2005, 09:32 AM   #35
TruckStuff
Member
 
Registered: Apr 2002
Posts: 498

Original Poster
Rep: Reputation: 30
Quote:
Originally posted by Capt_Caveman
Could you run ldd against the modified binaries to give a list of the linked files. Taking an otherwise normal app and linking it against a malicious library is an effective means of creating a trojan that escapes detection fairly easily. I believe those hex numbers are memory addresses where dynamically loaded objects would get loaded during execution. Definitely take note if apps are linked to libs that they shouldn't be, like ps getting linked to an evil libcrypto or something.
Ran this threw a few of them and nothing out of the ordinary came up. None of the memory addresses listed by objdump correspond to a library listed by ldd. Although, there is no telling what what ldd might have been modified to say. I've moved our apps and will be booting off of a Knoppix CD later today, so maybe that will reveal something more helpful, or at least more trustworthy.
Quote:
Originally posted by Capt_Caveman
Also how big is the average difference in file size. If it's large, you could have some kind of direct elf infection.
Did a comparison using hte first tripwire report that came up bad. Size diffs range from 42 bytes to 11k, average of 2.9k, median of 2k. The elf infection could be something to look b/c I've noticed a lot of libraries modified as well.

Snort hasn't found anything yet. I dump all net traffic from last night into some logs, so I'll be reviewing those today to see if there is anything odd in them. Will post back if/when I find anything.

Also, I've been looking over some previous tripwire reports. Turns out that the first major changes were on July 28 @ 4:02AM. I left for vacation that day, so that's probably why I missed it before. Most of the binaries modified on that date are the same ones in the list above, e.g. callback, sshd, stunnel, tripwire, etc. Lots of libraries changed at that point: librpm, libfontconfig, libtiff, libwvutils, libcurl... In that report, inodes, CRCs and MD5s changed, but very few modify times or sizes were changed.

Last edited by TruckStuff; 08-18-2005 at 09:43 AM.
 
Old 08-18-2005, 11:25 AM   #36
Matir
LQ Guru
 
Registered: Nov 2004
Location: San Jose, CA
Distribution: Ubuntu
Posts: 8,507

Rep: Reputation: 125Reputation: 125
This is actually quite interesting. What does a disassembly (objdump -d) show at the addresses from the Dynamic section? Which set of command line args did you use to retrieve those addresses?
 
Old 08-18-2005, 11:49 AM   #37
TruckStuff
Member
 
Registered: Apr 2002
Posts: 498

Original Poster
Rep: Reputation: 30
Quote:
Originally posted by Matir
Which set of command line args did you use to retrieve those addresses?
Code:
# objdump -x `which sshd`
For sshd, I actually get a slightly different set of addresses:
Code:
  0x6ffffef9  0x8047dd4
  0x6ffffdf7  0xdc
  0x6ffffef8  0x8047eb0
  0x6ffffdf6  0x168
Quote:
Originally posted by Matir
This is actually quite interesting. What does a disassembly (objdump -d) show at the addresses from the Dynamic section?
Not sure, but I think what you are asking me is this:
Code:
# objdump -d `which sshd`| grep 8047dd4
# objdump -d `which sshd`| grep 8047eb0
So, it doesn't look like anything is at those addresses.
 
Old 08-18-2005, 12:40 PM   #38
Matir
LQ Guru
 
Registered: Nov 2004
Location: San Jose, CA
Distribution: Ubuntu
Posts: 8,507

Rep: Reputation: 125Reputation: 125
Hrrm, what about the first half of the addresses? (0x6fff...) maybe grep for 6ffffe or 6ffffd?
 
Old 08-18-2005, 01:47 PM   #39
TruckStuff
Member
 
Registered: Apr 2002
Posts: 498

Original Poster
Rep: Reputation: 30
Sorry, I should have mentioned that in my last post. I tried looking for those also and nothing was found.
 
Old 08-18-2005, 02:02 PM   #40
Matir
LQ Guru
 
Registered: Nov 2004
Location: San Jose, CA
Distribution: Ubuntu
Posts: 8,507

Rep: Reputation: 125Reputation: 125
That's very odd. objdump -x doesn't give me two columns of numbers in the Dynamic section, but rather one of words and one of numbers.
 
Old 08-18-2005, 02:18 PM   #41
TruckStuff
Member
 
Registered: Apr 2002
Posts: 498

Original Poster
Rep: Reputation: 30
Right... I get the same thing, than all of the cracked binaries have lines like the above at the end of the dynamic section:
Code:
# objdump -x `which sshd`
<snip>
Dynamic Section:
  NEEDED      libwrap.so.0
  NEEDED      libpam.so.0
  NEEDED      libdl.so.2
  NEEDED      libresolv.so.2
  NEEDED      libcrypto.so.0.9.8
  NEEDED      libutil.so.1
  NEEDED      libz.so.1
  NEEDED      libnsl.so.1
  NEEDED      libcrypt.so.1
  NEEDED      libc.so.6
  INIT        0x804b4e8
  FINI        0x80780ec
  HASH        0x8048148
  STRTAB      0x8047148
  SYMTAB      0x8048a18
  STRSZ       0xc79
  SYMENT      0x10
  DEBUG       0x0
  PLTGOT      0x8086128
  PLTRELSZ    0x880
  PLTREL      0x11
  JMPREL      0x804ac68
  REL         0x804ac38
  RELSZ       0x30
  RELENT      0x8
  VERNEED     0x804ab98
  VERNEEDNUM  0x3
  VERSYM      0x804a942
  0x6ffffef9  0x8047dd4
  0x6ffffdf7  0xdc
  0x6ffffef8  0x8047eb0
  0x6ffffdf6  0x168
<snip>
 
Old 08-18-2005, 05:36 PM   #42
TruckStuff
Member
 
Registered: Apr 2002
Posts: 498

Original Poster
Rep: Reputation: 30
Well I think I may have found at least part of the answer. It seems my workstation is now exhibiting the same signs as this server. Many binaries are showing the same addresses at the end of the Dynamic section of objdump. I don't have any idea which was compromised first, but my workstation is being just as secretive at revealing information. The workstation is even stranger b/c it sits on a LAN behind a firewall, with all the same protections as servers. The only thing exposed to the outside world is SSH. We are a small company, so I know I'm the only with the know how to do something like this.

FWIW, I rebooted the server off of Knoppix this afternoon. I found nothing that I didn't already know, e.g. no hidden cron jobs, magical hidden files. Just the same memory addresses in objdump that were there before. I also reviewed all network traffic from last night and found nothing out of the ordinary, so it still doesn't appear that whoever did this is taking advantage of the boxes for anything malicious.

Looks like I'm in for a fun Friday.

Last edited by TruckStuff; 08-18-2005 at 05:38 PM.
 
Old 08-18-2005, 06:44 PM   #43
Matir
LQ Guru
 
Registered: Nov 2004
Location: San Jose, CA
Distribution: Ubuntu
Posts: 8,507

Rep: Reputation: 125Reputation: 125
Out of my own curiousity, would it be possible for you to create a tarball of your compromised binaries? I'd like to try some strace and disassembly work on them to see if I can figure out what they're supposed to do, and possibly even the origin of the binaries. To be honest, I'm mostly curious about their intent. Though, I imagine you're probably curious at this point too.

(I have a personal interest in computer forensics.)
 
Old 08-18-2005, 08:37 PM   #44
Capt_Caveman
Senior Member
 
Registered: Mar 2003
Distribution: Fedora
Posts: 3,658

Rep: Reputation: 69
I'm getting the feeling this might have something to do with prelinking. It should be run about the same time as yum by cron. All of the target dirs seem to appear in /etc/prelink.conf. Prelinking binaries can add entries to the end of the .dynamic section for virtual mem addresses locations. It also may explain why some test are coming back clean(those that don't look at the elf header) while others are flagging them (those that do check). Try undoing the prelinking with prelink -au binary and rerun tripwire.
 
Old 08-18-2005, 08:42 PM   #45
Matir
LQ Guru
 
Registered: Nov 2004
Location: San Jose, CA
Distribution: Ubuntu
Posts: 8,507

Rep: Reputation: 125Reputation: 125
That's actually a brilliant point. Wish I had thought about that.
 
  


Reply


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off



LinuxQuestions.org > Forums > Linux Forums > Linux - Security

All times are GMT -5. The time now is 09:45 PM.

Main Menu
Advertisement
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
Facebook: linuxquestions Google+: linuxquestions
Open Source Consulting | Domain Registration