My fedora box is keep sending things out at port 6664 (ircd?).. well right now I use firestarter to kind of block the outgoing traffic at port 6664... but how do I find out what went wrong? I'm sure I got hacked somehow.

I'm very new to computer security...can anyone tell me what to look for? Step by step would be nice

I think is my http server that is cause me problem.. cause when I restart my httpd, I've notice that it stop sending thing out at port 6664

Can anyone give me some good ideas to secure a http server? and how to detect the cause of the problem

If your system has been compromised, then it's very likely that you'll need to completely reformat and re-install unless you can be absolutely certain of how extensive the compromise was and that *all* traces of the compromise have been removed.

That being said, you need to determine exactly how the system was accessed and what changes have been made to the system. It would have helped to track down the exact httpd processes and see what user it was running under, what options and if it really was the http daemon and not some other binary with a similar name. It's probably a good idea to take a look at the list of processes executing on the system anyway to see if anything else looks suspicious (use the command ps aux). Check to see what network services are running using netstat -pantu and lsof -i. Another good place to start is to use Fedoras package manager to check the integrity of system binaries using rpm -Va. Check the contents of the file system for suspicious files or dirs, especially look in /tmp and the httpd server root. Check /etc/passwd for any new usernames or any users other than root with a UID/GID of 0. Download and run a scan with chkrootkit or rootkit hunter. Finally, make sure to check all the system logs for any suspicious looking messages or errors. Obviously make sure to look at the http logs.

If you have a file alteration scanner like tripwire or samhain already installed then now would be a good time to run a check.

What is considered to be suspicious?

In the /tmp folder, there are some files that are owned by user apache...
I think those file owned by user apache are suspicious.
What should I do with them? Just delete them?
What should or shouldn't be in the /tmp folder?

[root@www tmp]# ls -l
total 3880
-rwxr--r-- 1 apache apache 33927 Dec 16 02:23 b
-rwxr-xr-x 1 apache apache 103 Dec 15 23:22 conect
-rw------- 1 apache apache 299008 Dec 21 08:07 core.10633
-rw------- 1 apache apache 299008 Dec 20 00:50 core.10984
-rw------- 1 apache apache 299008 Dec 19 07:24 core.13233
-rwxr--r-- 1 apache apache 469240 Dec 19 16:29 d
-rw-r--r-- 1 apache apache 469240 Dec 19 16:29 d.1
-rw-r--r-- 1 apache apache 469240 Dec 19 16:29 d.2
-rw-r--r-- 1 apache apache 469240 Dec 19 16:29 d.3
-rw-r--r-- 1 apache apache 469240 Dec 19 16:29 d.4
-rwxr--r-- 1 apache apache 33927 Dec 19 02:11 g
drwx------ 3 root root 4096 Dec 21 17:04 gconfd-root
-rw-r--r-- 1 apache apache 2309 Dec 21 18:43 listen.log
-rwxr--r-- 1 apache apache 462364 Dec 19 02:06 lordnikon
-rwxr--r-- 1 apache apache 462908 Dec 15 23:06 mass
-rwxr-xr-x 1 apache apache 94 Dec 19 16:47 mirela
-rw-r--r-- 1 apache apache 97 Dec 20 12:56 mirela.1
-rw-r--r-- 1 apache apache 97 Dec 20 12:56 mirela.2
-rw-r--r-- 1 apache apache 97 Dec 20 12:56 mirela.3
-rw-r--r-- 1 apache apache 97 Dec 20 12:56 mirela.4
-rwxr-xr-x 1 apache apache 116 Dec 19 02:07 nikons
drwx------ 2 root root 4096 Dec 21 18:44 orbit-root
-rwxr--r-- 1 apache apache 34913 Dec 19 19:11 w
-rw-r--r-- 1 apache apache 34913 Dec 20 12:33 w.1


I ran rkhunter, and it didn't detect any rootkit.

I got something like this when I do netstat -ptav

tcp 0 1 192.168.0.102:43914 66-146-172-103.skyrive:http FIN_WAIT1 -
tcp 0 1 192.168.0.102:33480 66-146-165-37.skyriver:http FIN_WAIT1 -
tcp 0 1 192.168.0.102:60663 oslo2.no.eu.undernet.o:ircd SYN_SENT 23356/bas

Is there a way to find out what the PID/Program name the "-" ??
is there a way to find out where is that /bas program located??

I believe only the files owned by root (gconfd-root and orbit-root) are normal. The core.xxx files are likely apache core dumps (check the CoreDumpDirectory directive in httpd.conf to verify that's where core dumps are sent). Everything else looks like it belongs to the cracker. There are a couple of executables and some associated files. I wouldn't delete any of them right now. In fact take a look at the contents. Also as root run: find / -newer /tmp/mass . Anything show up in the apache or system logs? Anything else from all the other checks?

I would absolutely recommend taking the system offline immediately.

what does find / -newer /tmp/mass do ?

Is there a way to find out what the PID/Program name the "-" ??
It's likely a defunct process that has terminated but the network socket is still shutting down (hence the FIN_WAIT).

is there a way to find out where is that /bas program located??
cat /proc/23356/cmdline

Finds all files that have been created/modified since that file was created.

I think those are what you call massrootkit scripts used by script kiddies.

I used to get them, they use a vulnerability in your php scripts or something like that...

If they haven't gotten yet to your root account, there's still time. You simply delete those files in the tmp folder, enable php safe mode, kill the running script process by using fuser and kill.

You also have to change permissions of your /tmp folder so that they won't ever come in again the /tmp folder.

harden also your ssh by not allowing root logins and always use RSA keys. Another way is to install apf and bfd from rxnetworks. bfd is called a brute force daemon and works with apf which is a firewall - it blocks those brute force attacks automatically.

Another significant determinent to those guys is setting your ssh server to work in another port, which only you know. Or disable ssh completely

I tried the / -newer /tmp/mass and look at the log file.. nothing new was added to the log..

However, I've notice when my computer was trying to send stuff via port 6664, there are something like this going on in the system log file

Dec 21 19:29:32 www kernel: Outbound IN= OUT=eth0 SRC=192.168.0.102 DST=195.54.102.4 LEN=44 TOS=0x00 PREC=0x00 TTL=64 ID=40148 DF PROTO=TCP SPT=60868 DPT=6667 WINDOW=5840 RES=0x00 SYN URGP=0

Dec 21 19:29:35 www kernel: Outbound IN= OUT=eth0 SRC=192.168.0.102 DST=195.54.102.4 LEN=44 TOS=0x00 PREC=0x00 TTL=64 ID=40150 DF PROTO=TCP SPT=60868 DPT=6667 WINDOW=5840 RES=0x00 SYN URGP=0

Dec 21 19:29:35 www kernel: Outbound IN= OUT=eth0 SRC=192.168.0.102 DST=161.53.178.240 LEN=44 TOS=0x00 PREC=0x00 TTL=64 ID=44634 DF PROTO=TCP SPT=51598 DPT=6667 WINDOW=5840 RES=0x00 SYN URGP=0

etc etc etc (a lot of them)...

I think my firewall block all of them because I've a restrictive outgoing traffic policy.. and from its event, it said blocked...

Anyway, for ssh.. I didn't disable it ...but my port 22 was blocked completely.. I wonder how is that a problem.

Right now, I deleted all those files. Killed all those processes, and looks like the sending finally stop.

What kind of permission for the /tmp file should I configure?
Right now it's 777 (oh my! it was wide open..sht)

If I had a rootkit on my fedora box... should rkhunter be able to detect it? just wondering...

You're missing the point here. Those files were not the cause of the compromise, just a symptom. You need to identify the means by which they were put there in the first place. If you're apache server is vulnerable or if your content is being abused (like CGI abuse) then they can simply upload all of those scripts again.

If a rootkit has been installed on the system, then root was compromised (by definition a rootkit is designed to hide the signs of a root compromise) and a full format and reinstall is necessary, simply deleting files is not even *REMOTELY* enough. That being said, I haven't seen anything yet to suggest that root was compromised and rkhunter should find common rootkits. However deleting files, killing processes, and sweeping everything under the rug is foolish. In fact by deleting files you're removing possible clues as to how the compromise occured. If you're not seeing anything in your Apache logs (like suspicious URLs /var/www/html/access) then you should be concerned.

Capt_Caveman, yes I'm totally agree with you. In fact I'm concerned..but rebuild the entire system is not an option right now.

I would like to know how they compromised my /tmp folder too.. but I've limited expertise.

Right now I kind of harden the /tmp folder a bit following the suggestion give by:
http://www.experts-exchange.com/Secu..._21093545.html

If there's anything else you can suggest me to do.. it would be great

Thank you to all... I really appreciated your help

well some things you can do is to use tools like tripwire and bastille. Tripwire is a tool that records changes to files on your system. Bastille is a system hardener. Another good tool is nessus which is a port and vulnerability scanner. One thing you'll want to do is to run chkconfig --list to see what's starting up at boot. You can go through that list and ask yourself, does this service needs to be run? For all the services that you deem need to be run, you need to make sure they're updated. For packages like apache, mysql, and php, I'd recommend uninstalling your distro's packages of these programs and compile the latest packages from each's websites. Read your logs often and take note of anything suspicious. Check your distro's package manager to update software on your machine. I always check at least once a day. I think this is just the basics and I'm probably not doing everything possible, but I've been pretty clean all this time.