My machine has been hacked with this ftp_scanner, find locates the file as:
find /dev -name *ftp_scanner*
/dev/ /fb/ftp_scanner.c
/dev/ /fb/ftp_scanner

I cant access /fb, it is a herd link to /fb0:
crw------- 1 root root 29, 0 Jan 30 2003 fb0

I cant do a ls on it, nor delete it. How can I do this ?

Thanks a lot in advance!

Hello and welcome to LQ, sorry to see it had to be on such a sad occasion. Hope you like it here.

Before you remove the "evidence" I would suggest you see who is on the system now, if this or anything else suspicious is running. Usually I don't recycle, but you best take the steps to perform from here starting from "How did this exploit get onto my system ?" If you have questions before doing that please make sure you first mitigate the situation (steps 1 - 3).


I cant do a ls on it, nor delete it.
The directory is /dev/ /fb/. See the space in between?

I think the system is totally compromised so I will trash the pc in some time and setup a new one. The ftp_scanner proceses (dozens) where not seen by ps ax, only netstat -anp showed them, then I was able to do a killall... so I guess there are a couple system command binaries hacked also... that means the system is trash. How can I prevent them to change root password? If they do that I'm fried because the machine is in another city, its a web server. I don't have a clue on how they get in, but it could have been months ago, I saw couple of evidences but I was too bussy... anyway, do you know how can I access those files ? When you do:
[root] find /dev -name *ftp_scanner*
/dev/ /fb/ftp_scanner.c
/dev/ /fb/ftp_scanner

I can't ls or rm -f /dev/fb/ftp_scanner.c nor ls -al /fb/ftp_scanner.c

What do you think ?

Try the following:
To my untrained eye, it looks like there is a subdirectory under dev with the value " " (space character).

Added:
I just created a directory named " ". I could access the contents with either the backslash space or the double quoted space. It seems to me that the doublequoted space (" ") worked with the fewest confusions. But, I have no idea whether the hacker has your root password. But, I'm betting yes.

Thanks. It was there,
ls -al /dev/" "/fb
total 92
drwxr-xr-x 2 root root 4096 Feb 20 15:51 .
drwxr-xr-x 3 root root 4096 Feb 20 15:17 ..
-rwxr-xr-x 1 root root 82 Mar 10 2006 0-100
-rwxr-xr-x 1 root root 84 Mar 10 2006 100-200
-rwxr-xr-x 1 root root 84 Mar 10 2006 200-255
-rwxr-xr-x 1 root root 19262 Mar 10 2006 ftp_scanner
-rw-r--r-- 1 root root 17267 Mar 10 2006 ftp_scanner.c
-rw-r--r-- 1 root root 70 Feb 20 15:51 log
-rw-r--r-- 1 root root 2094 Mar 10 2006 o
-rw-r--r-- 1 root root 16061 Mar 10 2006 pass
-rwxr-xr-x 1 root root 118 Mar 10 2006 run
-rw-r--r-- 1 root root 14 Mar 10 2006 users

And the bad news is that root is the owner.

root is the owner
- Notify everyone (also colo) who have access to the box to stay away and inspect and change their access passwords.
- Save process, network and open files listing off-site.
- Kill all publicly accessable services except SSH (to access the box) and make sure they can't be enabled again.
- Compress and copy the passwd/shadow and log dir off-site.
- Raise the firewall to only allow traffic from your IP (or /24 range if you've got a dynamic IP) and optionally your distro's repo's.
- Verify the system using your package managers features and make sure your initscripts are clean.
- Reinstall the SSH daemon and libraries. If you're unsure install it next to the other (other dir) and make it listen through (X)inetd on another port. Don't forget to modify this Xinetd service access restrictions or set it in the firewall.
- Optionally reboot the box if you can be reasonably certain it comes up again w/o problems.
These steps don't make the box "better" but can help mitigate the situation. Do not postpone. Do not leave out steps unless you know what the consequences are. Use caution and common sense.



To my untrained eye (..)
Heh. To my well-trained eyes you just haven't been reading the whole thread too well ;-p I already posted that.

"To my untrained eye (..)
Heh. To my well-trained eyes you just haven't been reading the whole thread too well ;-p I already posted that."

You're right. I don't know why, but my eyes slid right over your post without registering your last paragraph. Sorry 'bout that. Was only trying to help - and learn.

I cant close it down because it is a live server with several clients (web sites). I guess I'm at their mercy. I guess I will spent the next 3/4 days setting everything up again in a new box, then take it there. I think they are there since several days. In /var/log/messages there were a couple of unusual entries today around the time the ftp_scanner launched (lot of bandwidth usage and high uptime):
Lots of:
kernel: 66.239.40.21 sent an invalid ICMP error to a broadcast
then several:
[proftpd] fork(): Resource temporarily unavailable
and then:
kernel: NET: 1 messages suppressed

I guess those are due to the bottleneck in the bandwidth, maybe the fork() due to the number of processes opened by the ftp_scanner daemon.

I cant close it down because it is a live server with several clients (web sites). I guess I'm at their mercy.
I would like to take the time to argue for looking at it the other way.


Professionality, integrity and openness.
Your customers will find out sooner or later. Eventually you'll have to migrate or temporarily offload their sites to an intermediate server. If you're a professional hoster, then your clients probably chose to pay you for that level of service you provide. It means you deal with facts and deal with them in a way that supports your clients businesses. It means they put their money where their trust is. Hiding the compromise
- does not benefit your customers especially not if they're relying on secured transactions,
- shows you do not earn their trust,
- shows you don't actually care about their business.
Alerting customers and keeping them informed and performing an efficient relocation/recovery will actually gain your customers support since they get to see how you perform in critical situations.


Responsability and obligation.
It is your responsability to your own business, your clients and the whole Internet to practice "safe hex". As long as you keep the server compromised, up and accessable you are a liability to the whole community. There are no compelling and valid reasons to keep the box as operable as it is now. If you can't correct the situation right now then at least execute steps to keep your customers safe. If you want me to help you by looking at data send me an email.



I don't know why, but my eyes slid right over your post without registering your last paragraph.
No problem, no problem. And I do appreciate the effort you're putting into this.

ok, please dont get mad with me, I'm struggling here.
I've found couple of things:
/bin/ps
/bin/netstat
/bin/ls
where altered, they had user 122 114 I guess those are tipical users from a know rootkit, I dont know much about this. I restored original copies from a server I have at home with exact same OS. I had to previously restore attributes with chattr because I (root) was not able to delete /bin/ps, netstat and ls, they where chattr +ais inmutable.
And now doing a ps I can see 2 funny services:
root 1001 0.0 0.1 1856 628 ? S 22:10 0:00 /sbin/ttyload -q
root 1004 0.0 0.1 1504 532 ? R 22:10 0:00 ttymon tymon

I've never saw those 2 before... They are run at sturtup, dont know yet whre from.

Now netstat shows:
tcp 0 0 0.0.0.0:31336 0.0.0.0:* LISTEN 1001/ttyload

I did not saw that before.

List of files owned by user 122 (non existent) (after restored good copy of /usr/bin/find):

/usr/bin/pstree
/usr/bin/find
/usr/bin/dir
/usr/bin/md5sum
/usr/bin/top
/usr/bin/slocate
/usr/lib/libsh/.bashrc
/usr/lib/libsh/.sniff/shsniff
/usr/lib/libsh/.sniff/shp
/usr/lib/libsh/shsb
/usr/lib/libsh/hide
/usr/sbin/lsof
/bin/netstat
/bin/ls
/bin/ps
/lib/libsh.so/shhk
/lib/libsh.so/shhk.pub
/lib/libsh.so/shrs
/sbin/ifconfig

I just had to emphasize what upSpawn already said. The "but it's a colo box" argument doesn't fly. The more you delay the more data your clients lose. To be on safe side they'll have to DISCARD ALL CHANGES SINCE THE COMPROMISE! That's why you should have reverted to the last known good backup immediately, even if it meant staying up until 5AM rebuilding everything. That's how you get away with nobody finding out: by having a disaster recovery plan and implementing it in a timely fashion.

Heck, I've quit contracts where management refused to provide the equipment I needed for disaster recovery. Rest assured it won't be their fault when something goes wrong and I refuse to accept financial responsibility for someone else's bad decisions.

ok, please dont get mad with me, I'm struggling here.
I'm not mad at you. I'm helping you understand what consequences your actions have.


I've found couple of things (..) altered, they had user 122 114 I guess those are tipical users from a know rootkit
This and your listing of files point to the SHV4 rootkit.


I restored original copies from a server I have at home with exact same OS. (..) And now doing a ps I can see 2 funny services:
root 1001 0.0 0.1 1856 628 ? S 22:10 0:00 /sbin/ttyload -q
root 1004 0.0 0.1 1504 532 ? R 22:10 0:00 ttymon tymon

"chattr -ius /sbin/ttyload && shred /sbin/ttyload"
"grep ttymon -r /etc" (/etc/inittab: /sbin/ttymon?).



@Crito:
I just had to emphasize what upSpawn already said.
Awww. Was mutilating my handle really necessary?...


To be on safe side they'll have to DISCARD ALL CHANGES SINCE THE COMPROMISE!
That is just the tip of the iceberg. Think passwords, stolen data or items that's been tampered with.


That's why you should have reverted to the last known good backup immediately,
If you don't have independant means of verification in place the phrase "last known good" has no meaning. He already showed he didn't harden/audit the server properly/timely.


That's how you get away with nobody finding out
IMNSHO that's irresponsible and unwanted behaviour.
If one made a mistake one should deal with it.
Not don't try and dodge responsabilities.

Out of curiousity, it would be interesting to know the version of kernel that you had at the time, as well as version of running daemons (like ssh,apache,..)
But I guess you are now re-installing