LinuxQuestions.org
Welcome to the most active Linux Forum on the web.
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - Security
User Name
Password
Linux - Security This forum is for all security related questions.
Questions, tips, system compromises, firewalls, etc. are all included here.

Notices


Reply
  Search this Thread
Old 02-20-2007, 02:42 PM   #1
oskiborga
LQ Newbie
 
Registered: Feb 2007
Posts: 11

Rep: Reputation: 0
ftp_scanner hack - cant delete it from /dev/ /fb/ftp_scanner


My machine has been hacked with this ftp_scanner, find locates the file as:
find /dev -name *ftp_scanner*
/dev/ /fb/ftp_scanner.c
/dev/ /fb/ftp_scanner

I cant access /fb, it is a herd link to /fb0:
crw------- 1 root root 29, 0 Jan 30 2003 fb0

I cant do a ls on it, nor delete it. How can I do this ?

Thanks a lot in advance!
 
Old 02-20-2007, 04:45 PM   #2
unSpawn
Moderator
 
Registered: May 2001
Posts: 29,383
Blog Entries: 55

Rep: Reputation: 3558Reputation: 3558Reputation: 3558Reputation: 3558Reputation: 3558Reputation: 3558Reputation: 3558Reputation: 3558Reputation: 3558Reputation: 3558Reputation: 3558
Hello and welcome to LQ, sorry to see it had to be on such a sad occasion. Hope you like it here.

Before you remove the "evidence" I would suggest you see who is on the system now, if this or anything else suspicious is running. Usually I don't recycle, but you best take the steps to perform from here starting from "How did this exploit get onto my system ?" If you have questions before doing that please make sure you first mitigate the situation (steps 1 - 3).


I cant do a ls on it, nor delete it.
The directory is /dev/ /fb/. See the space in between?

Last edited by unSpawn; 02-20-2007 at 04:47 PM.
 
Old 02-20-2007, 05:20 PM   #3
oskiborga
LQ Newbie
 
Registered: Feb 2007
Posts: 11

Original Poster
Rep: Reputation: 0
I think the system is totally compromised so I will trash the pc in some time and setup a new one. The ftp_scanner proceses (dozens) where not seen by ps ax, only netstat -anp showed them, then I was able to do a killall... so I guess there are a couple system command binaries hacked also... that means the system is trash. How can I prevent them to change root password? If they do that I'm fried because the machine is in another city, its a web server. I don't have a clue on how they get in, but it could have been months ago, I saw couple of evidences but I was too bussy... anyway, do you know how can I access those files ? When you do:
[root] find /dev -name *ftp_scanner*
/dev/ /fb/ftp_scanner.c
/dev/ /fb/ftp_scanner

I can't ls or rm -f /dev/fb/ftp_scanner.c nor ls -al /fb/ftp_scanner.c

What do you think ?
 
Old 02-20-2007, 05:55 PM   #4
Quakeboy02
Senior Member
 
Registered: Nov 2006
Distribution: Debian Squeeze 2.6.32.9 SMP AMD64
Posts: 3,320

Rep: Reputation: 126Reputation: 126
Quote:
[root] find /dev -name *ftp_scanner*
/dev/ /fb/ftp_scanner.c
/dev/ /fb/ftp_scanner

I can't ls or rm -f /dev/fb/ftp_scanner.c nor ls -al /fb/ftp_scanner.c
Try the following:
Code:
ls /dev/\ /fb
(Note the space after the backslash)
or perhaps
ls /dev/" "/fb
I can't remember which works.
To my untrained eye, it looks like there is a subdirectory under dev with the value " " (space character).

Added:
I just created a directory named " ". I could access the contents with either the backslash space or the double quoted space. It seems to me that the doublequoted space (" ") worked with the fewest confusions. But, I have no idea whether the hacker has your root password. But, I'm betting yes.

Last edited by Quakeboy02; 02-20-2007 at 06:04 PM.
 
Old 02-20-2007, 06:31 PM   #5
oskiborga
LQ Newbie
 
Registered: Feb 2007
Posts: 11

Original Poster
Rep: Reputation: 0
Thanks. It was there,
ls -al /dev/" "/fb
total 92
drwxr-xr-x 2 root root 4096 Feb 20 15:51 .
drwxr-xr-x 3 root root 4096 Feb 20 15:17 ..
-rwxr-xr-x 1 root root 82 Mar 10 2006 0-100
-rwxr-xr-x 1 root root 84 Mar 10 2006 100-200
-rwxr-xr-x 1 root root 84 Mar 10 2006 200-255
-rwxr-xr-x 1 root root 19262 Mar 10 2006 ftp_scanner
-rw-r--r-- 1 root root 17267 Mar 10 2006 ftp_scanner.c
-rw-r--r-- 1 root root 70 Feb 20 15:51 log
-rw-r--r-- 1 root root 2094 Mar 10 2006 o
-rw-r--r-- 1 root root 16061 Mar 10 2006 pass
-rwxr-xr-x 1 root root 118 Mar 10 2006 run
-rw-r--r-- 1 root root 14 Mar 10 2006 users
 
Old 02-20-2007, 06:35 PM   #6
Quakeboy02
Senior Member
 
Registered: Nov 2006
Distribution: Debian Squeeze 2.6.32.9 SMP AMD64
Posts: 3,320

Rep: Reputation: 126Reputation: 126
Code:
-rwxr-xr-x 1 root root 19262 Mar 10 2006 ftp_scanner
And the bad news is that root is the owner.
 
Old 02-20-2007, 07:19 PM   #7
unSpawn
Moderator
 
Registered: May 2001
Posts: 29,383
Blog Entries: 55

Rep: Reputation: 3558Reputation: 3558Reputation: 3558Reputation: 3558Reputation: 3558Reputation: 3558Reputation: 3558Reputation: 3558Reputation: 3558Reputation: 3558Reputation: 3558
root is the owner
- Notify everyone (also colo) who have access to the box to stay away and inspect and change their access passwords.
- Save process, network and open files listing off-site.
- Kill all publicly accessable services except SSH (to access the box) and make sure they can't be enabled again.
- Compress and copy the passwd/shadow and log dir off-site.
- Raise the firewall to only allow traffic from your IP (or /24 range if you've got a dynamic IP) and optionally your distro's repo's.
- Verify the system using your package managers features and make sure your initscripts are clean.
- Reinstall the SSH daemon and libraries. If you're unsure install it next to the other (other dir) and make it listen through (X)inetd on another port. Don't forget to modify this Xinetd service access restrictions or set it in the firewall.
- Optionally reboot the box if you can be reasonably certain it comes up again w/o problems.
These steps don't make the box "better" but can help mitigate the situation. Do not postpone. Do not leave out steps unless you know what the consequences are. Use caution and common sense.



To my untrained eye (..)
Heh. To my well-trained eyes you just haven't been reading the whole thread too well ;-p I already posted that.
 
Old 02-20-2007, 07:39 PM   #8
Quakeboy02
Senior Member
 
Registered: Nov 2006
Distribution: Debian Squeeze 2.6.32.9 SMP AMD64
Posts: 3,320

Rep: Reputation: 126Reputation: 126
"To my untrained eye (..)
Heh. To my well-trained eyes you just haven't been reading the whole thread too well ;-p I already posted that."

You're right. I don't know why, but my eyes slid right over your post without registering your last paragraph. Sorry 'bout that. Was only trying to help - and learn.
 
Old 02-20-2007, 07:40 PM   #9
oskiborga
LQ Newbie
 
Registered: Feb 2007
Posts: 11

Original Poster
Rep: Reputation: 0
I cant close it down because it is a live server with several clients (web sites). I guess I'm at their mercy. I guess I will spent the next 3/4 days setting everything up again in a new box, then take it there. I think they are there since several days. In /var/log/messages there were a couple of unusual entries today around the time the ftp_scanner launched (lot of bandwidth usage and high uptime):
Lots of:
kernel: 66.239.40.21 sent an invalid ICMP error to a broadcast
then several:
[proftpd] fork(): Resource temporarily unavailable
and then:
kernel: NET: 1 messages suppressed

I guess those are due to the bottleneck in the bandwidth, maybe the fork() due to the number of processes opened by the ftp_scanner daemon.
 
Old 02-20-2007, 08:20 PM   #10
unSpawn
Moderator
 
Registered: May 2001
Posts: 29,383
Blog Entries: 55

Rep: Reputation: 3558Reputation: 3558Reputation: 3558Reputation: 3558Reputation: 3558Reputation: 3558Reputation: 3558Reputation: 3558Reputation: 3558Reputation: 3558Reputation: 3558
I cant close it down because it is a live server with several clients (web sites). I guess I'm at their mercy.
I would like to take the time to argue for looking at it the other way.


Professionality, integrity and openness.
Your customers will find out sooner or later. Eventually you'll have to migrate or temporarily offload their sites to an intermediate server. If you're a professional hoster, then your clients probably chose to pay you for that level of service you provide. It means you deal with facts and deal with them in a way that supports your clients businesses. It means they put their money where their trust is. Hiding the compromise
- does not benefit your customers especially not if they're relying on secured transactions,
- shows you do not earn their trust,
- shows you don't actually care about their business.
Alerting customers and keeping them informed and performing an efficient relocation/recovery will actually gain your customers support since they get to see how you perform in critical situations.


Responsability and obligation.
It is your responsability to your own business, your clients and the whole Internet to practice "safe hex". As long as you keep the server compromised, up and accessable you are a liability to the whole community. There are no compelling and valid reasons to keep the box as operable as it is now. If you can't correct the situation right now then at least execute steps to keep your customers safe. If you want me to help you by looking at data send me an email.



I don't know why, but my eyes slid right over your post without registering your last paragraph.
No problem, no problem. And I do appreciate the effort you're putting into this.
 
Old 02-20-2007, 08:56 PM   #11
oskiborga
LQ Newbie
 
Registered: Feb 2007
Posts: 11

Original Poster
Rep: Reputation: 0
ok, please dont get mad with me, I'm struggling here.
I've found couple of things:
/bin/ps
/bin/netstat
/bin/ls
where altered, they had user 122 114 I guess those are tipical users from a know rootkit, I dont know much about this. I restored original copies from a server I have at home with exact same OS. I had to previously restore attributes with chattr because I (root) was not able to delete /bin/ps, netstat and ls, they where chattr +ais inmutable.
And now doing a ps I can see 2 funny services:
root 1001 0.0 0.1 1856 628 ? S 22:10 0:00 /sbin/ttyload -q
root 1004 0.0 0.1 1504 532 ? R 22:10 0:00 ttymon tymon

I've never saw those 2 before... They are run at sturtup, dont know yet whre from.

Now netstat shows:
tcp 0 0 0.0.0.0:31336 0.0.0.0:* LISTEN 1001/ttyload

I did not saw that before.
 
Old 02-20-2007, 11:14 PM   #12
oskiborga
LQ Newbie
 
Registered: Feb 2007
Posts: 11

Original Poster
Rep: Reputation: 0
List of files owned by user 122 (non existent) (after restored good copy of /usr/bin/find):

/usr/bin/pstree
/usr/bin/find
/usr/bin/dir
/usr/bin/md5sum
/usr/bin/top
/usr/bin/slocate
/usr/lib/libsh/.bashrc
/usr/lib/libsh/.sniff/shsniff
/usr/lib/libsh/.sniff/shp
/usr/lib/libsh/shsb
/usr/lib/libsh/hide
/usr/sbin/lsof
/bin/netstat
/bin/ls
/bin/ps
/lib/libsh.so/shhk
/lib/libsh.so/shhk.pub
/lib/libsh.so/shrs
/sbin/ifconfig
 
Old 02-21-2007, 03:28 AM   #13
Crito
Senior Member
 
Registered: Nov 2003
Location: Knoxville, TN
Distribution: Kubuntu 9.04
Posts: 1,168

Rep: Reputation: 53
Quote:
Originally Posted by unSpawn
I cant close it down because it is a live server with several clients (web sites). I guess I'm at their mercy.
I would like to take the time to argue for looking at it the other way.
I just had to emphasize what upSpawn already said. The "but it's a colo box" argument doesn't fly. The more you delay the more data your clients lose. To be on safe side they'll have to DISCARD ALL CHANGES SINCE THE COMPROMISE! That's why you should have reverted to the last known good backup immediately, even if it meant staying up until 5AM rebuilding everything. That's how you get away with nobody finding out: by having a disaster recovery plan and implementing it in a timely fashion.

Heck, I've quit contracts where management refused to provide the equipment I needed for disaster recovery. Rest assured it won't be their fault when something goes wrong and I refuse to accept financial responsibility for someone else's bad decisions.
 
Old 02-21-2007, 05:18 AM   #14
unSpawn
Moderator
 
Registered: May 2001
Posts: 29,383
Blog Entries: 55

Rep: Reputation: 3558Reputation: 3558Reputation: 3558Reputation: 3558Reputation: 3558Reputation: 3558Reputation: 3558Reputation: 3558Reputation: 3558Reputation: 3558Reputation: 3558
ok, please dont get mad with me, I'm struggling here.
I'm not mad at you. I'm helping you understand what consequences your actions have.


I've found couple of things (..) altered, they had user 122 114 I guess those are tipical users from a know rootkit
This and your listing of files point to the SHV4 rootkit.


I restored original copies from a server I have at home with exact same OS. (..) And now doing a ps I can see 2 funny services:
root 1001 0.0 0.1 1856 628 ? S 22:10 0:00 /sbin/ttyload -q
root 1004 0.0 0.1 1504 532 ? R 22:10 0:00 ttymon tymon

"chattr -ius /sbin/ttyload && shred /sbin/ttyload"
"grep ttymon -r /etc" (/etc/inittab: /sbin/ttymon?).



@Crito:
I just had to emphasize what upSpawn already said.
Awww. Was mutilating my handle really necessary?...


To be on safe side they'll have to DISCARD ALL CHANGES SINCE THE COMPROMISE!
That is just the tip of the iceberg. Think passwords, stolen data or items that's been tampered with.


That's why you should have reverted to the last known good backup immediately,
If you don't have independant means of verification in place the phrase "last known good" has no meaning. He already showed he didn't harden/audit the server properly/timely.


That's how you get away with nobody finding out
IMNSHO that's irresponsible and unwanted behaviour.
If one made a mistake one should deal with it.
Not don't try and dodge responsabilities.
 
Old 02-21-2007, 06:07 AM   #15
nx5000
Senior Member
 
Registered: Sep 2005
Location: Out
Posts: 3,307

Rep: Reputation: 55
Out of curiousity, it would be interesting to know the version of kernel that you had at the time, as well as version of running daemons (like ssh,apache,..)
But I guess you are now re-installing
 
  


Reply


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off



Similar Threads
Thread Thread Starter Forum Replies Last Post
delete dev direcory , how can recover? jimmyjiang Red Hat 5 10-18-2006 02:38 PM
Delete /dev/console Hal Linux - Security 2 06-13-2006 04:05 PM
I cannot access EITHER CD drive! And there's no /dev/hdc or /dev/hdd or /dev/cdrom! Dmalic Linux - Hardware 13 11-18-2005 07:11 PM
[Udev] Can I delete the content of /dev? lokee Linux - Software 6 02-04-2005 08:30 PM
udev likes to delete /dev/sda* but not create them :( edman007 Linux - Software 2 01-18-2005 05:02 PM

LinuxQuestions.org > Forums > Linux Forums > Linux - Security

All times are GMT -5. The time now is 08:43 AM.

Main Menu
Advertisement
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
Facebook: linuxquestions Google+: linuxquestions
Open Source Consulting | Domain Registration