Hey.

I found some processes running (as user httpd) on some of my ports, particularly 6667.

bash 28527 httpd 15u IPv4 240191 TCP xxx.xxx.xxx.xxx:40335->undernet.xs4all.nl:ircd (SYN_SENT)

Somethign like that. If I kill the processes they return after a few minutes. I tried running chkrootkit and rkhunter and both tell be that the box is clean. However, I believe someone managed to install an irc rootkit or something.

Any suggestion how I can trace it and kill it. Server is a centos 4.6 server.

Probably not a rootkit. Both chkrootkit and rkhunter have a good reputation. That being said, you definitely have a rogue process on your computer. Being run as httpd means that they PROBABLY didn't get root on your box and are using some web-based app to exploit your system. (Check your web-based apps.)

Try using "ps axjf" or looking at the "PPID" (parent process ID) field from ps to figure out where the processes are coming from.

Thanks!

When I do "lsof -i tcp:6667" I get PID 28793. Then when I do "ps axjf" and glook for the PPID of 28793, I get PPID 1, and the process 28793 has a command showing only "bash".

Furthermore, another admin has sent us this:

"...and had an altered/rogue process installed on it, and was part of a botnet that was found on our network. "

The bot files can usually be found by running these one line
commands as the root user.

find / -exec grep -l "undernet" {} +
find / -exec grep -l "sybnc" {} +
find / -name "*.set" | perl -pe 's/.\/\w+-(\w+)-.*/$1/' | sort uniq

however, none of these commands produce a result, and instead generate an error: "find: missing argument to `-exec'"

Can anyone suggest how we can find these rogue processes and kill them permanently? Everytime I kill the pids they just respawn.

Yeah, those commands look a bit odd to me. Not sure what shell they're using with the "+" operator.

Anyway:
Try looking at /proc/28793/fd to see what files are open by that process. Odds are one of them is a malicious script. You can also look for psybnc instances via:

ok I see a bunch of files/shortcuts like 0, 1, 2, and so on. only file 0 seems to be valid, the rest are links to nonexisting files. but file 0 links to /tmp/tested/LinkEvents and the contents are:

1254197701 EnergyMech started...
1254211450 New Nick Host7747 -> Host
1254211450 New Nick nastrand_ -> nastrand
1254211460 New Nick Flori7962 -> Florin
1254211473 New Nick nastrand_ -> nastrand
1254211481 New Nick root4844 -> root
1254211489 New Nick user_____ -> user
1254211491 New Nick Flori3746 -> Florin
1254211491 New Nick user_____ -> user
1254211495 New Nick test__ -> test
1254211501 New Nick error____ -> error
1254211511 New Nick test4293 -> test
1254211511 New Nick eminem_ -> eminem

by the way, the find statement you posted, what does the "sybnc" stand for?

I'm sure you have already googled this now and found that it is an IRC bot written in C.

http://www.energymech.net/

Which would make me believe that /tmp/tested/LinkEvents is the bot's log file.

sybnc is common in the naming of IRC bouncers. Such as:

psyBNC is an easy-to-use, multi-user, permanent IRC-Bouncer with many features. Some of its features include symmetric ciphering of talk and connections (Blowfish and IDEA), the possibility of linking multiple bouncers to an internal network including a shared partyline, vhost- and relay support to connected bouncers and an extensive online help system. Many other helpful functions are included. It compiles on Linux, FreeBSD, SunOs and Solaris.

So the find statement he told you to run basically says this (ignoring the sort):

Start at / and for every file you come across search in the file for the string "sybnc".

Under /proc/28793/ is there a "cwd"? That stands for "current working directory" and could show you where the executable was ran from so possibly the installation folder.

However a common trick is to delete the executable after you run it. So another trick for ya, if you copy /proc/28793/exe somewhere else it will give you a copy of the executable ran.

So, yes it looks like you have a bot on your box. Taking care of it isn't really the concerning part, the question I would be asking is how did it get installed?

nomb

thanks. kinda got it sorted out. looks like the executable was only in /tmp/ and i managed to delete the file and so far I have not seen it reoccurring.

as for the means it got in, i have sorted that as well.

thanks again for the assistance.

Deleting binaries is like treating symptoms, not causes.


Please share.

I too would like to know what you found out.

we turned off firewall when we were troubleshooting some network issues on that particular box. Then we saw someone log into ssh, as we get emails every time someone logs into the server. We were able to enable the firewall after that and kick out the unauthorized personnel, but not until they were able to install something. after scanning the directories/folders it seems the server is clean and they were only able to upload to the /tmp directory.

I'd wonder how someone got SSH access to that machine. That's a pretty significant compromise.

In my environment, we'd pull that machine from the network, format and reinstall.

the machine is actually an old server of ours. all, well most, of our accounts and production clients are on different servers behind our network. this machine is outside the firewall and is just running on apf. we used to use this before on our hsphere cluster but since moved to new servers when we switched control panels. that being said, we've kinda stopped updating this box and it's just been sitting in our network, hence it's not as secure as the other servers. but since someone managed to get inside, i think it's about time to re-image this box and make use of it.

hope this makes more sense on how the intrusion happened in the first place and how we have sorted it out

Thanks for the update.

So basically your shop neglected the machine, allowing it to be subverted. There's no need to defend that but imagine the effect for other 'net users this having been gone undetected. I suggest you take the breach as your cue to shape up.


Sure. It's just a shame it requires an incident to make people see.

yes i won't defend that. my fault, and will definitely take this as a lesson. bright side for me is that we saw this before it can do some real damage on others' network and our network. but then again, it could have been prevented had we not forgotten about the box.

thanks again guys. much appreciated.