Can't remove suspicious process running under apache

NightRook · 10-23-2011, 01:09 PM

(Fedora 12)

Hello everyone, I have suspicious processes that I found running on my system this week. I can't get rid off them. I need some help. The processes seem to be invoked by crond but I can't find the processes or scripts. I killed the processes and they stop and disappear temporarily. A few seconds later, they come back again. I have used netstat to see where they are coming from and what they are using. What I found out was that the processes seems to be running under apache and they seems to use the crond service but I also see ircd and I don't have ircd service running(I think? I searched for that service and I don't see it). I used whois on both offending IP addresses and it seems that I have someone from Denmark and Germany messing with my server. I used iptables to drop all traffic from both ip addresses but the processes still run and I am getting nervous not knowing what these processes are doing. HELP!!! Thank you in advance to all Linux gurus out there. I know a bit about Linux but apparently not enough to deal with these issues. I need guidance from the masters.

PHP Code:



[root@server1 ~]# netstat -ntp
Active Internet connections (w/o servers)
Proto Recv-Q Send-Q Local Address               Foreign Address             State       PID/Program name   
tcp        0      0 192.168.0.100:54515         195.54.159.109:6667         ESTABLISHED 409/crond           
tcp        0      0 192.168.0.100:40412         194.124.229.59:6667         ESTABLISHED 409/crond           
tcp        0      0 192.168.0.100:22            192.168.0.101:42952         ESTABLISHED 32481/0             
[root@server1 ~]# 
[root@server1 ~]# netstat -utaep
Active Internet connections (servers and established)
Proto Recv-Q Send-Q Local Address               Foreign Address             State       User       Inode      PID/Program name   
tcp        0      0 *:ssh                       *:*                         LISTEN      root       13621      1356/sshd           
tcp        0      0 localhost.localdomain:ipp   *:*                         LISTEN      root       13023      1173/cupsd          
tcp        0      0 localhost.localdomain:smtp  *:*                         LISTEN      root       14655      1657/sendmail: acce 
tcp        0      0 *:mysql                     *:*                         LISTEN      mysql      13829      1498/mysqld         
tcp        0      0 *:sunrpc                    *:*                         LISTEN      root       12410      1056/rpcbind        
tcp        0      0 *:42195                     *:*                         LISTEN      root       12818      1118/rpc.statd      
tcp        0      0 server1:54515               195.54.159.109:ircd         ESTABLISHED apache     382731     409/crond           
tcp        0      0 server1:40412               clanserver4u.de.quaken:ircd ESTABLISHED apache     382727     409/crond           
tcp        0      0 server1:ssh                 192.168.0.101:42952         ESTABLISHED root       380067     32481/0             
tcp        0      0 *:ssh                       *:*                         LISTEN      root       13623      1356/sshd           
tcp        0      0 server1:ipp                 *:*                         LISTEN      root       13022      1173/cupsd          
tcp        0      0 *:sunrpc                    *:*                         LISTEN      root       12415      1056/rpcbind        
udp        0      0 *:870                       *:*                                     root       12804      1118/rpc.statd      
udp        0      0 *:mdns                      *:*                                     avahi      12587      1091/avahi-daemon:  
udp        0      0 *:sunrpc                    *:*                                     root       12405      1056/rpcbind        
udp        0      0 *:ipp                       *:*                                     root       13026      1173/cupsd          
udp        0      0 *:41849                     *:*                                     apache     382689     409/crond           
udp        0      0 server1:ntp                 *:*                                     ntp        14381      1373/ntpd           
udp        0      0 localhost.localdomain:ntp   *:*                                     root       13681      1373/ntpd           
udp        0      0 *:ntp                       *:*                                     root       13675      1373/ntpd           
udp        0      0 *:fido                      *:*                                     avahi      12588      1091/avahi-daemon:  
udp        0      0 *:807                       *:*                                     root       12409      1056/rpcbind        
udp        0      0 *:bootpc                    *:*                                     root       13519      1316/dhclient       
udp        0      0 *:33743                     *:*                                     root       12815      1118/rpc.statd      
udp        0      0 *:sunrpc                    *:*                                     root       12412      1056/rpcbind        
udp        0      0 fe80::21b:11ff:fe1a:ntp     *:*                                     ntp        13880      1373/ntpd           
udp        0      0 server1:ntp                 *:*                                     root       13680      1373/ntpd           
udp        0      0 *:ntp                       *:*                                     root       13676      1373/ntpd           
udp        0      0 *:807                       *:*                                     root       12414      1056/rpcbind        
[root@server1 ~]# 
[root@server1 ~]# whois 195.54.159.109
[Querying whois.ripe.net]
[whois.ripe.net]
% This is the RIPE Database query service.
% The objects are in RPSL format.
%
% The RIPE Database is subject to Terms and Conditions.
% See http://www.ripe.net/db/support/db-terms-conditions.pdf

% Note: this output has been filtered.
%       To receive output for a database update, use the "-B" flag.

% Information related to '195.54.159.0 - 195.54.159.127'

inetnum:      195.54.159.0 - 195.54.159.127
netname:      SE-AH-JKP-BACKBONE
descr:        Song Networks Svenska AB
country:      SE
admin-c:      TSA12-RIPE
tech-c:       TSA12-RIPE
status:       ASSIGNED PA
remarks:      INFRA-AW
mnt-by:       TELE1-SE-MNT
source:       RIPE # Filtered

role:            TDC Song AB
address:         TDC Song AB
address:         PO Box 764
address:         Norra Malmvagen 143
address:         S-191 27 Sollentuna
address:         SE
admin-c:         MC20742-RIPE
admin-c:         KS3459-RIPE
tech-c:          MC20742-RIPE
tech-c:          KS3459-RIPE
mnt-by:          TELE1-SE-MNT
nic-hdl:         TSA12-RIPE
source:          RIPE # Filtered
abuse-mailbox:   abuse@tdcsong.se

% Information related to '195.54.128.0/19AS3292'

route:          195.54.128.0/19
descr:          TDC Song AB
origin:         AS3292
remarks:        This network is assigned to se.tele1 customers
remarks:        in Sweden. In case of routing problem, please
remarks:        contact peering@sn.net, in case of inappropriate
remarks:        usage or attacks please mail abuse@tdcsong.se
mnt-by:         TELE1-SE-MNT
source:         RIPE # Filtered


[root@server1 ~]# traceroute 195.54.159.109
traceroute to 195.54.159.109 (195.54.159.109), 30 hops max, 60 byte packets
 1  192.168.0.1 (192.168.0.1)  0.491 ms  0.696 ms  2.534 ms
 2  * * *
 3  10.5.25.106 (10.5.25.106)  20.053 ms  20.356 ms  20.332 ms
 4  10.5.24.1 (10.5.24.1)  20.565 ms  20.634 ms  20.714 ms
 5  10.20.12.10 (10.20.12.10)  20.454 ms  20.510 ms  20.550 ms
 6  216.55.10.61 (216.55.10.61)  22.091 ms  12.831 ms  19.595 ms
 7  ae1d0.mcr1.chicago-il.us.xo.net (216.156.1.81)  18.805 ms  18.747 ms  18.728 ms
 8  vb1700.rar3.chicago-il.us.xo.net (216.156.0.161)  29.703 ms  30.112 ms  29.750 ms
 9  207.88.14.194.ptr.us.xo.net (207.88.14.194)  25.220 ms  25.191 ms  25.164 ms
10  206.111.2.138.ptr.us.xo.net (206.111.2.138)  36.601 ms  36.586 ms  25.690 ms
11  TDC-TeleDenmark-New-York.TenGigabitEthernet2-3.ar4.NYC1.gblx.net (146.82.35.202)  43.990 ms  43.875 ms  43.844 ms
12  88-131-138-29.se.ip.tdc.net (88.131.138.29)  150.940 ms  145.642 ms  145.607 ms
13  88-131-138-30.se.ip.tdc.net (88.131.138.30)  145.978 ms  146.557 ms  146.434 ms
14  * * *
15  * * *
16  * * *
17  * * *
18  * * *
19  * * *
20  * * *
21  * * *
22  * * *
23  * * *
24  * * *
25  * * *
26  * * *
27  * * *
28  * * *
29  * * *
30  * * *
[root@server1 ~]#
[root@server1 ~]# whois 195.124.229.59
[Querying whois.ripe.net]
[whois.ripe.net]
% This is the RIPE Database query service.
% The objects are in RPSL format.
%
% The RIPE Database is subject to Terms and Conditions.
% See http://www.ripe.net/db/support/db-terms-conditions.pdf

% Note: this output has been filtered.
%       To receive output for a database update, use the "-B" flag.

% Information related to '195.124.229.0 - 195.124.229.255'

inetnum:      195.124.229.0 - 195.124.229.255
netname:      ZUGANG-NET
descr:        Ingenieurbuero fuer Computer & Telekommunikation
descr:        Naundorfer Str. 67
descr:        D-01640 Coswig
descr:        Germany
country:      DE
admin-c:      MB1493-RIPE
tech-c:       MB1493-RIPE
status:       ASSIGNED PA
mnt-by:       UUNETDE-I
source:       RIPE # Filtered

irt:            IRT-MCI-DE
address:        ISG/IP Network Security
address:        MCI
address:        Sebrathweg 20
address:        D-44149 Dortmund
address:        DE
phone:          +49 231 972 1550
fax-no:         +49 231 972 2703
abuse-mailbox:  abuse@de.uu.net
org:            ORG-UD1-RIPE
admin-c:        WERT1-RIPE
tech-c:         WERT1-RIPE
auth:           MD5-PW $1$kDn7y2p6$w4Tryx44frd5s/x9s5cCW/
auth:           PGPKEY-5960220D
auth:           PGPKEY-5E478DDC
auth:           PGPKEY-92479A5D
auth:           PGPKEY-B4826395
auth:           PGPKEY-C3F46E69
irt-nfy:        ip-net-sec@de.mci.com
irt-nfy:        registrar@eu.uu.net
mnt-by:         MCI-EMEA-M-MNT
source:         RIPE # Filtered

person:       Mathias Beck
address:      Ingenieurbuero fuer Computer & Telekommunikation
address:      Naundorfer Str. 67
address:      D-01640 Coswig
address:      Germany
phone:        +49 3523 62900
fax-no:       +49 3523 62908
nic-hdl:      MB1493-RIPE
mnt-by:       ROM-MIKNET
source:       RIPE # Filtered

% Information related to '195.124.0.0/14AS702'

route:          195.124.0.0/14
descr:          UUNETDE-AGG-195-124
origin:         AS702
holes:          195.125.34.0/24
member-of:      AS702:RS-DE,
                AS702:RS-DE-PA,
                AS702:RS-DE-PULLUP
inject:         upon static
aggr-mtd:       outbound
mnt-routes:     AS12638-MNT {195.125.34.0/24^+}
mnt-by:         WCOM-EMEA-RICE-MNT
source:         RIPE # Filtered


[root@server1 ~]# traceroute 195.124.229.59
traceroute to 195.124.229.59 (195.124.229.59), 30 hops max, 60 byte packets
 1  192.168.0.1 (192.168.0.1)  0.525 ms  0.696 ms  1.925 ms
 2  * * *
 3  10.5.25.106 (10.5.25.106)  19.007 ms  18.991 ms  18.963 ms
 4  10.5.24.2 (10.5.24.2)  20.099 ms  20.082 ms  20.245 ms
 5  10.20.12.5 (10.20.12.5)  18.873 ms  19.811 ms  25.204 ms
 6  * * *
 7  * * *
 8  MCI-level3-ae.chicago3.level3.net (4.68.110.186)  25.191 ms  14.378 ms  24.286 ms
 9  0.ae3.XL4.CHI13.ALTER.NET (152.63.66.77)  19.458 ms  19.376 ms  19.272 ms
10  0.xe-3-2-0.IL2.DCA6.ALTER.NET (152.63.43.81)  42.524 ms  41.480 ms  41.849 ms
11  * * *
12  * * *
13  * * *
14  * * *
15  * * *
16  * * *
17  * * *
18  * * *
19  * * *
20  * * *
21  * * *
22  * * *
23  * * *
24  * * *
25  * * *
26  * * *
27  * * *
28  * * *
29  * * *
30  * * *
[root@server1 ~]# 
[root@server1 ~]# iptables -L
Chain INPUT (policy ACCEPT)
target     prot opt source               destination         
ACCEPT     all  --  anywhere             anywhere            state RELATED,ESTABLISHED 
ACCEPT     icmp --  anywhere             anywhere            
ACCEPT     all  --  anywhere             anywhere            
ACCEPT     tcp  --  anywhere             anywhere            state NEW tcp dpt:ftp 
ACCEPT     tcp  --  anywhere             anywhere            state NEW tcp dpt:imaps 
ACCEPT     tcp  --  anywhere             anywhere            state NEW tcp dpt:smtp 
ACCEPT     udp  --  anywhere             anywhere            state NEW udp dpt:openvpn 
ACCEPT     tcp  --  anywhere             anywhere            state NEW tcp dpt:pop3s 
ACCEPT     udp  --  anywhere             anywhere            state NEW udp dpt:netbios-ns 
ACCEPT     udp  --  anywhere             anywhere            state NEW udp dpt:netbios-dgm 
ACCEPT     tcp  --  anywhere             anywhere            state NEW tcp dpt:netbios-ssn 
ACCEPT     tcp  --  anywhere             anywhere            state NEW tcp dpt:microsoft-ds 
ACCEPT     udp  --  anywhere             anywhere            state NEW udp dpt:netbios-ns 
ACCEPT     udp  --  anywhere             anywhere            state NEW udp dpt:netbios-dgm 
ACCEPT     tcp  --  anywhere             anywhere            state NEW tcp dpt:https 
ACCEPT     tcp  --  anywhere             anywhere            state NEW tcp dpt:ssh 
ACCEPT     tcp  --  anywhere             anywhere            state NEW tcp dpt:http 
ACCEPT     tcp  --  anywhere             anywhere            state NEW tcp dpt:ups 
REJECT     all  --  anywhere             anywhere            reject-with icmp-host-prohibited 
DROP       all  --  underworld2.no.quakenet.org  anywhere            
DROP       all  --  clanserver4u.de.quakenet.org  anywhere            

Chain FORWARD (policy ACCEPT)
target     prot opt source               destination         
REJECT     all  --  anywhere             anywhere            reject-with icmp-host-prohibited 

Chain OUTPUT (policy ACCEPT)
target     prot opt source               destination         
[root@server1 ~]#

unSpawn · 10-23-2011, 01:59 PM

Run

Code:

( /bin/ps acxfwwwe 2>&1; /usr/sbin/lsof -Pwln 2>&1; /bin/ls -al /var/spool/cron 2>&1; /usr/bin/lastlog 2>&1; /usr/bin/last 2>&1; /usr/bin/who -a 2>&1 ) > /tmp/stats.log

as root to save process listings, then run 'service crond stop' to stop the cron daemon. Then kill the process 'kill -9 409'. Deny outgoing new connections in the TCP port range IRC uses until you've got a fix on things. Obfuscate IP addresses if necessary and attach /tmp/stats.log as plain text file if unsure.

BTW also kill kill your web server for the time being. Usually one of the common points of entry. And please stop RPC and CUPS unless you really, really need it.
BTW[1] your firewall allows too much through even for your needs. Review would be good.

NightRook · 10-23-2011, 02:37 PM

I have attached the file created by the code and stopped cups and I don't see the rpc service. I see rpcbind. I stopped that one. I also stopped crond. I can't stop apache. I need it. My students in high school need to access moodle to do their assignments. I see that the file upload failed. It exceeds the size permitted. Is there anywhere that I can email it or should I post it here in two parts, or attach two different files(part 1 and 2).

Hey, better yet. Let me post it here:

http://www.memories2treasure.net/support

unSpawn · 10-23-2011, 06:04 PM

Thanks for the log. I'll get to that.

Quote:

Originally Posted by NightRook

I (..) stopped cups and I don't see the rpc service. I see rpcbind. I stopped that one.

Good. Don't run what you don't need. If you need it then set account and network restrictions.

Quote:

Originally Posted by NightRook

I can't stop apache. I need it. My students in high school need to access moodle to do their assignments.

With all due respect I disagree. What you and your students need is a rodent-free server to work on. BTW IMHO a server shouldn't need to run X or (a deprecated version of) Fedora anyway.

Quote:

Originally Posted by NightRook

I see that the file upload failed. It exceeds the size permitted.

What? Where? Anyway, check all your Apache logs (run through Logwatch for quick wins?). You're bound to find some fun there.

Now for the log you sent me: crond runs with 2 PIDs: 1599, 13872 and 409. The 3rd one is the job which has a working directory of "/var/tmp/.q" (notice the dot), the crontab it runs is "/var/tmp/.q/crond" and it writes output to "/var/tmp/.q/LinkEvents" which, while you shouldn't value file names much, points to an IRC bot like EnergyMech. A plain web server shouldn't need to initiate much connections, after all HTTP requests mostly result in return traffic, to block that kind of ports on servers:

Code:

/sbin/iptables -A OUTPUT -o eth0 -m state --state NEW -p tcp --dport 6660:7000 -j DROP

As for /var/spool/cron:

Code:

total 16
drwx------.  2 root   root 4096 2011-10-08 11:00 .
drwxr-xr-x. 16 root   root 4096 2011-03-06 19:09 ..
-rw-------.  1 apache root   45 2011-10-08 11:00 apache
-rw-------.  1    504 root   48 2010-09-29 17:08 test

...apparently somebody saw fit to allow Apache a crontab (why?) and there's a user ominously called "test". Apache never logged in but user "test" did:

Code:

Username         Port     From             Latest
test     pts/0        143.111.n.n    Sat Oct 16 08:20 - 14:13  (05:53)    
test     pts/0        68.111.n.n     Wed Sep 29 17:06 - 05:43 (4+12:37)   
test     pts/2        119.188.n.n    Sun Sep 26 16:55 - 17:06  (00:11)    
test     pts/0        119.188.n.n    Sun Sep 26 16:50 - 17:06  (00:16)    
test     pts/1        119.188.n.n    Sun Sep 26 16:42 - 17:06  (00:24)    
test     pts/0        64.150.n.n     Tue Sep 14 12:40 - 14:26  (01:46)

...and seemingly from a variety of places including AS4837 (the infamous Shandong Province Network).

That's all fun if you need a test account (you don't) and if your students are truly international (I assert you're located somewhere in AS6325) but I doubt that:

Code:

alex      pts/0   AS9318   (.kr)  Thu  Nov  4   02:31  -  02:32  (00:01)
alex      pts/0   AS8708   (.ro)  Tue  Nov  16  00:41  -  00:42  (00:00)
alex      pts/0   AS8708   (.ro)  Tue  Nov  16  10:10  -  10:11  (00:00)
alex      pts/0   AS8708   (.ro)  Fri  Nov  12  17:11  -  17:12  (00:00)
alex      pts/0   AS8708   (.ro)  Tue  Nov  9   12:28  -  12:31  (00:02)
alex      pts/0   AS8708   (.ro)  Thu  Nov  11  09:22  -  09:22  (00:00)
alex      pts/0   AS8708   (.ro)  Mon  Nov  15  01:07  -  01:07  (00:00)
alex      pts/0   AS8708   (.ro)  Wed  Nov  10  07:14  -  07:14  (00:00)
alex      pts/1   AS8708   (.ro)  Thu  Nov  4   02:31  -  02:33  (00:01)
alex      pts/1   AS8708   (.ro)  Thu  Nov  18  03:40  -  03:40  (00:00)
alex      pts/2   AS8708   (.ro)  Thu  Nov  18  09:58  -  09:59  (00:00)
chris     pts/0   AS9318   (.kr)  Thu  Nov  4   02:53  -  03:18  (00:25)
chris     pts/1   AS9318   (.kr)  Thu  Nov  4   02:55  -  03:21  (00:25)
chris     pts/2   AS8708   (.ro)  Thu  Nov  4   02:56  -  03:00  (00:04)
david     pts/2   AS9318   (.kr)  Thu  Nov  4   03:01  -  03:21  (00:20)
david     pts/3   AS9318   (.kr)  Thu  Nov  4   02:58  -  03:18  (00:20)
david     pts/4   AS9318   (.kr)  Thu  Nov  4   03:03  -  03:23  (00:20)
david     pts/4   AS8708   (.ro)  Thu  Nov  4   02:59  -  02:59  (00:00)
frank     pts/5   AS9318   (.kr)  Thu  Nov  4   03:10  -  03:18  (00:08)
frank     pts/6   AS9318   (.kr)  Thu  Nov  4   03:12  -  03:21  (00:08)
frank     pts/8   AS9318   (.kr)  Thu  Nov  4   03:14  -  03:23  (00:08)
john      pts/0   AS4837   (.cn)  Thu  Nov  4   16:35  -  16:37  (00:01)
john      pts/0   AS8615   (.ru)  Wed  Nov  17  07:38  -  17:58  (10:20)
john      pts/0   AS41749  (.ro)  Thu  Nov  4   15:40  -  15:40  (00:00)
john      pts/0   AS41749  (.ro)  Tue  Nov  9   06:45  -  06:46  (00:00)
john      pts/0   AS41749  (.ro)  Wed  Nov  17  07:34  -  07:38  (00:03)
john      pts/10  AS9318   (.kr)  Thu  Nov  4   03:17  -  03:21  (00:04)
john      pts/12  AS9318   (.kr)  Thu  Nov  4   03:18  -  03:23  (00:04)
john      pts/1   AS8615   (.ru)  Thu  Nov  18  03:44  -  09:50  (06:06)
john      pts/7   AS9318   (.kr)  Thu  Nov  4   03:14  -  03:18  (00:04)
juan      pts/0   AS9318   (.kr)  Thu  Nov  4   03:19  -  03:23  (00:03)
juan      pts/11  AS9318   (.kr)  Thu  Nov  4   03:17  -  03:21  (00:03)
juan      pts/9   AS9318   (.kr)  Thu  Nov  4   03:15  -  03:18  (00:03)
leonardo  pts/0   AS16586  (.us)  Sat  Sep  18  01:44  -  01:51  (00:07)
test      pts/0   AS4837   (.cn)  Sun  Sep  26  16:50  -  17:06  (00:16)
test      pts/1   AS4837   (.cn)  Sun  Sep  26  16:42  -  17:06  (00:24)
test      pts/2   AS4837   (.cn)  Sun  Sep  26  16:55  -  17:06  (00:11)

So while you say you can't stop Apache I say you should. Keeping your server vulnerable and compromised accounts open to public access makes it a risk not only for you but all of us on the 'net.

NightRook · 10-23-2011, 08:50 PM

Thank you very much sir. I deleted the files and killed the processes that kept returning. After doing that, the processes have not returned. The server is running on level 3. I started the web server and samba again and everything seems to be working ok. Thank you very much. You really know your stuff.

Noway2 · 10-24-2011, 08:04 AM

It looks like you may have, at least temporarily, stopped the "rodent infestation". I think at this point there are two courses of action that would be prudent. One, would be to make a very thorough investigation of your system to make sure that there are no more rodent droppings anywhere.

Two, would be to take steps to better harden your system against this type of situation. The number one thing that you can do is keep your applications and your distribution up to date as this will close many vulnerabilities that allow this type of exploit. Looking over your log files, I notice that it appears you have students logging in remotely, presumably via SSH. Have you taken steps to protect SSH? Third, it looks like you are allowing direct login to root, which is not a good idea. With the number of users you have, you have lots of opportunity for poor passwords increasing your risks of brute force compromise. Fourth, as unSpawn pointed out run only the services you need. Make sure all other ports and traffic is closed off. Fifth, you really should be making a point of routinely monitoring your logs. It looks like this event occurred almost a month prior. Once you know your system is clean, a tool to help provide a summary report of the system status may alert you to these types of events sooner.

NightRook · 10-27-2011, 10:06 PM

Thanks for all the advice. I haven't seen any more suspicious connections on my system any more. What tools would you recommend for monitoring of these types of attacks Noway2?

Noway2 · 10-28-2011, 07:46 AM

Quote:

What tools would you recommend for monitoring of these types of attacks?

Given the description of your system in this thread: lots of student users who can connect remotely and who make use of web based services, here are some suggestions:

1 - use Logwatch to get a daily scan of the highlights of your system and get in the habit of reading it. I know that it can sometimes feel like a chore, but it typically only takes a few seconds to a minute each day. After a short while you will get to know what is standard for your system and changes will jump out at you as items for deeper inquiry.
2 - use a HIDS to keep track of changes to your system binaries and other protected locations. An example of where this may have been helpful in your cases is you would have been alerted to the changes to the cron tables. This will also scan for hidden files.
3 - With a large number of users and web services, you may get a lot off DNS traffic. Consider carefully what you are using for a DNS. Also consider using a proxy for your outbound connections. If you are allow users to have remote shell accounts, this system can easily be used as a tunnel for web traffic. A proxy would help control this.
4 - Put up a firewall and only allow INPUT connections on the services you want to make public.
5 - IRC seems to be the backbone of a lot of mal-activity. Consider blocking these ports on the output. Also consider blocking other channels like TOR and other .onion type channels.
6 - use key based authentication and require it on your critical access accounts. Use a super strong password on root and don't allow direct root login.
7 - Investigate mod security for Apache and be sure to keep any web service applications such as PHP or a content manager up to date.
8 - establish a practice of running a system update check on a routine basis, e.g. every Monday morning.
9 - require each user who will be connecting via DHCP to your system to preregister and provide a MAC addresses and then filter the DHCP on it, only giving an address to verified users. Consider placing them in a subnet and restrict their access via ACLs on the switches and routers.
10 - If you use SSH shell accounts for your users, consider using chroot to contain them to their own file system space. This will help keep them from being promiscuous with others data.