LinuxQuestions.org
Latest LQ Deal: Latest LQ Deals
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - Security
User Name
Password
Linux - Security This forum is for all security related questions.
Questions, tips, system compromises, firewalls, etc. are all included here.

Notices


Reply
  Search this Thread
Old 08-11-2005, 03:55 PM   #16
TruckStuff
Member
 
Registered: Apr 2002
Posts: 498

Original Poster
Rep: Reputation: 30

Quote:
Originally posted by Capt_Caveman
What was the specific cron job run at that time?
Code:
# cat /var/log/cron | grep "Aug  9 04:0"
Aug  9 04:00:01 server crond[20462]: (me) CMD (/home/php/backup/db-backup)    // <-- MySQL backup script
Aug  9 04:00:01 server crond[20464]: (me) CMD (/home/php/tracking/track-relco.php)   // <-- Nightly maintenance script
Aug  9 04:00:01 server crond[20466]: (root) CMD (/pub/awstats-6.4/wwwroot/cgi-bin/awstats.pl -config=server.domain.com -update > /dev/null)   // <-- AwStats
Aug  9 04:01:01 server crond[20476]: (root) CMD (run-parts /etc/cron.hourly)
Aug  9 04:02:01 server crond[20478]: (root) CMD (run-parts /etc/cron.daily)
Aug  9 04:02:04 server anacron[20909]: Updated timestamp for job `cron.daily' to 2005-08-09
The frist three cron jobs have been running for >6 months each and have never caused a problem.
Quote:
Originally posted by Capt_Caveman
Assuming it's cron.daily or cron.hourly what are the various cron jobs that could be run at those times? (check /etc/cron.daily/ or /etc/cron.hourly/ for anything relevent)
Code:
# ls -al /etc/cron.daily /etc/cron.hourly
/etc/cron.daily:
total 92
drwxr-xr-x   2 root root 4096 Aug  8 11:33 .
drwxr-xr-x  57 root root 4096 Aug  9 04:02 ..
lrwxrwxrwx   1 root root   28 May  2 09:57 00-logwatch -> ../log.d/scripts/logwatch.pl
-rwxr-xr-x   1 root root  418 Nov 19  2004 00-makewhatis.cron
-rwxr-xr-x   1 root root  276 Sep 28  2004 0anacron
-rwxr-xr-x   1 root root  180 Oct 19  2004 logrotate
-rwxr-xr-x   1 root root 2133 Nov 23  2004 prelink
-rwxr-xr-x   1 root root  104 Nov  1  2004 rpm
-rwxr-xr-x   1 root root   82 Oct 20  2004 slocate.cron
-rwxr-xr-x   1 root root  286 Aug 13  2004 tmpwatch
-rwxr-xr-x   1 root root  136 Jul  5 16:22 yum.cron
-rwxr-xr-x   1 root root   38 Aug  8 11:33 yum-update-check

/etc/cron.hourly:
total 16
drwxr-xr-x   2 root root 4096 Sep 20  2004 .
drwxr-xr-x  57 root root 4096 Aug  9 04:02 ..
The only non-standard script on FC3 is yum-update-check. This is a script to email me a check for yum updates every night (but not do them).

I'll check out the strace thing and see what I find.

Last edited by TruckStuff; 08-11-2005 at 03:56 PM.
 
Old 08-11-2005, 05:18 PM   #17
Matir
LQ Guru
 
Registered: Nov 2004
Location: San Jose, CA
Distribution: Ubuntu
Posts: 8,507

Rep: Reputation: 125Reputation: 125
I have a sneaky suspicion yum-update-check may be related, because its modification/creation time is awfully close to the time this thread started. Mind posting the contents of the file?

According to your first tripwire report, the tripwire database was last updated before the first run of the current version of that file (Aug 8, 13: something hours). That script would've most likely been run at 4am on Aug 9.
 
Old 08-11-2005, 05:25 PM   #18
sundialsvcs
LQ Guru
 
Registered: Feb 2004
Location: SE Tennessee, USA
Distribution: Gentoo, LFS
Posts: 9,078
Blog Entries: 4

Rep: Reputation: 3176Reputation: 3176Reputation: 3176Reputation: 3176Reputation: 3176Reputation: 3176Reputation: 3176Reputation: 3176Reputation: 3176Reputation: 3176Reputation: 3176
You really need to get some kind of bootable-CD, like those little "emergency disks" that Red Hat used to supply that would fit into your wallet. You need to establish "something you can still trust."

From this "trusted" environment, you then need to, say, re-download (and check the signature of) enough "trustworthy" packages from your distro vendor, and reinstall them, to further expand the system portions that you can "trust." You should particularly observe what kernel-packages are loaded or available to load. This will assist you in deciding whether or not you have, in fact, been hacked.

For what it's worth and from what I see, I remain un-persuaded. This tripwire report demands to be explained and you must not rest until you have done it, but this is a tripwire that suggests "a great, huge splat" instead of a targeted modification. It does not fit the modus operandi that I would intuitively expect to see here...
 
Old 08-11-2005, 05:33 PM   #19
Matir
LQ Guru
 
Registered: Nov 2004
Location: San Jose, CA
Distribution: Ubuntu
Posts: 8,507

Rep: Reputation: 125Reputation: 125
Might be modus operandi of someone who wanted to make it look like it wasn't the modus operandi of a cracker.

But I still think the script yum-update-check might be related.
 
Old 08-12-2005, 08:22 AM   #20
TruckStuff
Member
 
Registered: Apr 2002
Posts: 498

Original Poster
Rep: Reputation: 30
Quote:
Originally posted by Matir
I have a sneaky suspicion yum-update-check may be related, because its modification/creation time is awfully close to the time this thread started. Mind posting the contents of the file?
I suppose its possible depending on how yum checks for updates...
Code:
# cat /etc/cron.daily/yum-update-check
#!/bin/bash
/usr/bin/yum check-update
 
Old 08-12-2005, 10:30 AM   #21
Matir
LQ Guru
 
Registered: Nov 2004
Location: San Jose, CA
Distribution: Ubuntu
Posts: 8,507

Rep: Reputation: 125Reputation: 125
I guess I must be wrong then. That's a very simple script, of course. I guess it's just an odd coincidence. What is in yum.cron? (Just to eliminate the obvious suspects)
 
Old 08-12-2005, 03:30 PM   #22
TruckStuff
Member
 
Registered: Apr 2002
Posts: 498

Original Poster
Rep: Reputation: 30
Code:
# cat /etc/cron.daily/yum.cron
#!/bin/sh

if [ -f /var/lock/subsys/yum ]; then
        /usr/bin/yum -R 10 -e 0 -d 0 -y update yum
        /usr/bin/yum -R 120 -e 0 -d 0 -y update
fi
Code:
# ls -1 /var/lock/subsys/
acpid
atd
crond
haldaemon
iptables
kudzu
local
messagebus
mysql
network
qmail
smartd
sshd
syslog
xinetd
/var/lock/subsys/yum is intended to be created when the yum monitoring "daemon" is on and:
Code:
# /etc/init.d/yum status
Nightly yum update is disabled.
# chkconfig --list yum
yum             0:off   1:off   2:off   3:off   4:off   5:off   6:off
 
Old 08-12-2005, 03:35 PM   #23
Matir
LQ Guru
 
Registered: Nov 2004
Location: San Jose, CA
Distribution: Ubuntu
Posts: 8,507

Rep: Reputation: 125Reputation: 125
is there any chance /var/lock/subsys/yum somehow was accidentally created at some point?
 
Old 08-12-2005, 04:26 PM   #24
TruckStuff
Member
 
Registered: Apr 2002
Posts: 498

Original Poster
Rep: Reputation: 30
Anything is possible. There is no process that I am aware of on this box that would have created that file. But even if we suppose that they did, there would be logs of the updates in /var/log/yum.log (which there aren't). Also, I've reviewed recent FC3 updates, and those wouldn't explain all the files mentioned above.
 
Old 08-12-2005, 05:33 PM   #25
Matir
LQ Guru
 
Registered: Nov 2004
Location: San Jose, CA
Distribution: Ubuntu
Posts: 8,507

Rep: Reputation: 125Reputation: 125
Ah. Well, there goes that theory.
 
Old 08-15-2005, 08:57 AM   #26
TruckStuff
Member
 
Registered: Apr 2002
Posts: 498

Original Poster
Rep: Reputation: 30
Well, I'm now convinced. On Thursday, I installed a known-good version of the ProCPS package, just to see what would happen. On Friday morning, at 4:02AM, all of the binaries provided by the procps package (ps, w, kill, etc, etc...) were modified (md5s, times, size, etc...).

I've downloaded Knoppix and will boot off of that as soon as I can get all of our apps off of this server. Perhaps that will reveal something as to wtf is going on.

But now I go back to the original question of this thread: How in the hell did these guys get in?? I've got strong passwords on everything. No unneccesary programs running. Current versions of all packages that are running. All the usual bells and whistles.
 
Old 08-15-2005, 09:16 AM   #27
Matir
LQ Guru
 
Registered: Nov 2004
Location: San Jose, CA
Distribution: Ubuntu
Posts: 8,507

Rep: Reputation: 125Reputation: 125
That's a good question. The best way to get more information about what they're doing is by checking to see HOW files were altered. For example, maybe comparing the current 'ps' vs a known good one. It might help us to learn what their intent is.
 
Old 08-15-2005, 09:24 AM   #28
TruckStuff
Member
 
Registered: Apr 2002
Posts: 498

Original Poster
Rep: Reputation: 30
I've examined the output of 'strings `which ps`' and didn't see anything unusual. The result was similar to a known-good version of PS. I did an strace as Capt_Caveman recommended on ps, but I' haven't done enough w/ traces to know what's good/not good. Right now, I really want to avoid recreating the exact same scenario that allowed this person to get in in the first place before I reinstall.
 
Old 08-15-2005, 04:21 PM   #29
TruckStuff
Member
 
Registered: Apr 2002
Posts: 498

Original Poster
Rep: Reputation: 30
So have I stumped everyone? If the source of a security breah is unable to be determined, what it the SOP for most folks? Reinstall the same configuration and hope for the best?
 
Old 08-15-2005, 05:53 PM   #30
Capt_Caveman
Senior Member
 
Registered: Mar 2003
Distribution: Fedora
Posts: 3,658

Rep: Reputation: 69
Again, the time of the modification suggests that it's a scripted routine running at the same time. It might be informative to replace procps package with a good version again and then set the immutable attribute (chattr +i filename) on one of the modified files and see what happens. Hopefully if it's some benign rogue process, then it may generate an error. Though it's interesting that logger was modified too...

When you take it offline and reboot with a cdrom distro, not only take a look at md5s, but also look at the rest of the file system for any new files and verify that /etc/passwd looks normal. Looking over the list of modified files, I noticed a number of filesystem manipulation tools. So make sure to take a look at the disk itself for any new or suspicious looking partitions. Before shutting it down, make a tarball of /proc and look at /proc/modules.

Any luck with collecting data with Snort? How comfortable are you with disassembling binaries?
 
  


Reply


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off



LinuxQuestions.org > Forums > Linux Forums > Linux - Security

All times are GMT -5. The time now is 01:16 AM.

Main Menu
Advertisement
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
Facebook: linuxquestions Google+: linuxquestions
Open Source Consulting | Domain Registration