Is there a known DoS for apache 2.0.52 (RHEL 4)

zhjim · 11-24-2009, 04:02 AM

Hi folks,

I have an odd occurence on on of our testing servers. The log files of the apache indicate some IIS vulnerability from 2005.

Code:

77.223.197.161 - - [18/Nov/2009:16:38:06 +0100] "POST /_vti_bin/_vti_aut/fp30reg.dll HTTP/1.1" 301 377 "-" "-" 1656 622 111039 +
77.223.197.161 - - [18/Nov/2009:16:38:06 +0100] "SEARCH /\x90\x04H\x04H\x04H\--truncated

As far as I know this just does not work on apache. But the strange thing is that on Sunday the 20 this little thing happened

Code:

[Sun Nov 22 04:02:06 2009] [notice] SIGHUP received.  Attempting to restart
[Sun Nov 22 04:02:06 2009] [notice] seg fault or similar nasty error detected in the parent process

The SIGHUP should be coming from logrotate but why is it crashing the apache?

Nother thing I found in the logs is this

Code:

89.42.16.72 - - [18/Nov/2009:11:50:39 +0100] "GET //phpmyadmin//scripts/setup.php HTTP/1.1" 301 351 "-" "Plesk" 157 653 367 +
[Wed Nov 18 11:50:42 2009] [notice] child pid 9377 exit signal Segmentation fault (11)

89.42.16.72 - - [18/Nov/2009:12:09:07 +0100] "GET //phpMyAdmin//scripts/setup.php HTTP/1.1" 301 351 "-" "Plesk" 157 653 393 +
[Wed Nov 18 12:09:08 2009] [notice] child pid 19536 exit signal Segmentation fault (11)

This is a composite of my error and my access log. As there are more Segfaults than access they might not really match.

My question at last is how can I find out what realy happen? Is it wise to save the machine for analysis?
Reinstallation is planned for today (also I found no rootkit or file alteration with chkrootkit and rkhunter)

Did I miss out on anything? You need more info?
Best Regards
Zhjim

abefroman · 11-24-2009, 05:57 PM

Quote:

Originally Posted by zhjim

Hi folks,

I have an odd occurence on on of our testing servers. The log files of the apache indicate some IIS vulnerability from 2005.

Code:

77.223.197.161 - - [18/Nov/2009:16:38:06 +0100] "POST /_vti_bin/_vti_aut/fp30reg.dll HTTP/1.1" 301 377 "-" "-" 1656 622 111039 +
77.223.197.161 - - [18/Nov/2009:16:38:06 +0100] "SEARCH /\x90\x04H\x04H\x04H\--truncated

As far as I know this just does not work on apache. But the strange thing is that on Sunday the 20 this little thing happened

Code:

[Sun Nov 22 04:02:06 2009] [notice] SIGHUP received.  Attempting to restart
[Sun Nov 22 04:02:06 2009] [notice] seg fault or similar nasty error detected in the parent process

The SIGHUP should be coming from logrotate but why is it crashing the apache?

Nother thing I found in the logs is this

Code:

89.42.16.72 - - [18/Nov/2009:11:50:39 +0100] "GET //phpmyadmin//scripts/setup.php HTTP/1.1" 301 351 "-" "Plesk" 157 653 367 +
[Wed Nov 18 11:50:42 2009] [notice] child pid 9377 exit signal Segmentation fault (11)

89.42.16.72 - - [18/Nov/2009:12:09:07 +0100] "GET //phpMyAdmin//scripts/setup.php HTTP/1.1" 301 351 "-" "Plesk" 157 653 393 +
[Wed Nov 18 12:09:08 2009] [notice] child pid 19536 exit signal Segmentation fault (11)

This is a composite of my error and my access log. As there are more Segfaults than access they might not really match.

My question at last is how can I find out what realy happen? Is it wise to save the machine for analysis?
Reinstallation is planned for today (also I found no rootkit or file alteration with chkrootkit and rkhunter)

Did I miss out on anything? You need more info?
Best Regards
Zhjim

I would try just recompiling apache before doing a full resinstall.

Does it only Seg fault when setup.php for phpmyadmin is accessed?

anomie · 11-24-2009, 06:25 PM

There is a known DoS attack for Apache, period. But I don't see any evidence (in your post) that you're experiencing it.

Are you actually running phpMyAdmin on that server? Is someone able to run setup.php from the 'net?

zhjim · 11-25-2009, 02:49 AM

@anomie Your totally right that I don't suffer from the named DoS. But that does not mean there wasn't one.

Anyways all of the access are to not existing URI's. Here is a better example of seg faults

Code:

216.177.204.39 - - [20/Nov/2009:21:44:29 +0100] "GET /boutique/install.txt HTTP/1.1" 301 342 "-" "Toata dragostea mea pentru diavola" 195 597 164 -
216.177.204.39 - - [20/Nov/2009:21:44:29 +0100] "GET /cart/install.txt HTTP/1.1" 301 338 "-" "Toata dragostea mea pentru diavola" 191 589 139 -
216.177.204.39 - - [20/Nov/2009:21:44:29 +0100] "GET /store/install.txt HTTP/1.1" 301 339 "-" "Toata dragostea mea pentru diavola" 192 591 198 -

[Fri Nov 20 21:44:29 2009] [notice] child pid 14636 exit signal Segmentation fault (11)
[Fri Nov 20 21:44:29 2009] [notice] child pid 14637 exit signal Segmentation fault (11)
[Fri Nov 20 21:44:29 2009] [notice] child pid 14638 exit signal Segmentation fault (11)

They for sure match up.

The only reason why I'm upset is that a normal SIGHUP crashes the server right after a row of attacks. Might just be that some of the logs are missing. Like it could be after a break in.

I'm still unsure what to do. And reinstall is painless it brings on no changes

unSpawn · 11-25-2009, 12:27 PM

Quote:

Originally Posted by zhjim

@anomie

Please take care to address all replies made and not just the last one, TIA.

Quote:

Originally Posted by zhjim

Code:

"Toata dragostea mea pentru diavola"

Indeed not a DoS but a vulnerability scanner (searching for the line should show). Shown non-existent URI's return 301 ("Moved permanently") and there is no evidence (debugging information) that any of the logged lines segfaulted any particular HTTPd child. (For all we know it could be caused by a recent added or changed .so object.) The IP you last posted in the 216.177.192.0/19 (AS AS22364) range is a known scanner (see Dshield). There's also no evidence of protective measures you put deployed to regulate ("iptables -m recent"?) or restrict access (mod_security?). So you're both right we're missing some and on top of that basic system administration information:
- details about any legitimate software installation / upgrade / update / (re)configuration,
- complete logs (contact me by email if you want to share data off-site but do note that sending large emails or attachments w/o my prior and explicit request will get you blacklisted without further response from me.)
- complete Chkrootkit / Tiger / Rootkit Hunter / package manager (verify mode) (/ Aide, Samhain or even tripwire) reporting.

zhjim · 11-26-2009, 04:39 AM

Quote:

Originally Posted by unSpawn

Please take care to address all replies made and not just the last one, TIA.

My apologise for that.

I did not try out if setup.php segfaults the daemon cause I did not bring it up again. Cause to this is that I'm unsure what to do and don't want them to have a target again. Somehow silly cause it would only take a few minutes to check on this...
Some further research showed that it almost certain is APC (php cache) is causing this segfaults as well as the segfault of the parent process.

Quote:

Originally Posted by unSpawn

Indeed not a DoS but a vulnerability scanner (searching for the line should show). Shown non-existent URI's return 301 ("Moved permanently") and there is no evidence (debugging information) that any of the logged lines segfaulted any particular HTTPd child. (For all we know it could be caused by a recent added or changed .so object.)

Hm to be frankly I dunno how to give debugging information. Don't know any way to debug php applications that are called from apache...
But to me it seems that the lines from the error log

Code:

[Wed Nov 18 11:50:42 2009] [notice] child pid 9377 exit signal Segmentation fault (11)

when combined with those from the access log

Code:

89.42.16.72 - - [18/Nov/2009:11:50:39 +0100] "GET //phpmyadmin//scripts/setup.php HTTP/1.1" 301 351 "-" "Plesk" 157 653 367 +

are an evidence for it. This is a bad example but the lines in post #4 clearly show it.
Or am I misinterpreting things?

Quote:

Originally Posted by unSpawn

The IP you last posted in the 216.177.192.0/19 (AS AS22364) range is a known scanner (see Dshield). There's also no evidence of protective measures you put deployed to regulate ("iptables -m recent"?) or restrict access (mod_security?). So you're both right we're missing some and on top of that basic system administration information:
- details about any legitimate software installation / upgrade / update / (re)configuration,
- complete logs (contact me by email if you want to share data off-site but do note that sending large emails or attachments w/o my prior and explicit request will get you blacklisted without further response from me.)
- complete Chkrootkit / Tiger / Rootkit Hunter / package manager (verify mode) (/ Aide, Samhain or even tripwire) reporting.

I will gather the needed information and post them later. But will take some time.

Thanks to all of you for your input.

zhjim · 11-26-2009, 10:22 AM

Here are the logfiles for
[URL REMOVED]tiger[]
[URL REMOVED]rkhunter[]
[URL REMOVED]chkrootkit[]

and rpm -V for all the packages.

Code:

....L...    /dev/MAKEDEV
S.5....T  c /etc/updatedb.conf
S.5....T  c /etc/sysconfig/rhn/sources
S.5....T    /opt/HE/share/ghostscript/8.60/lib/Fontmap.GS
S.5....T  c /etc/webalizer.conf
.....UG.    /var/www/usage
.....UG.    /var/www/usage/msfree.png
.....UG.    /var/www/usage/webalizer.png
Unsatisfied dependencies for pam_mysql-0.6.2-2.rhel4.i386: libmysqlclient.so.14, libmysqlclient.so.14(libmysqlclient_14)
.....UG.    /opt/HE/nagios/.ssh
.....UG.    /opt/HE/nagios/.ssh/authorized_keys
.....UG.  c /opt/HE/etc/php.d/curl.ini
.....UG.    /opt/HE/lib/php/modules/curl.so
.....UG.  c /opt/HE/etc/php.d/gd.ini
.....UG.    /opt/HE/lib/php/modules/gd.so
S.5....T  c /etc/ntp.conf
.....UG.  c /opt/HE/etc/php.d/bz2.ini
.....UG.    /opt/HE/lib/php/modules/bz2.so
.....UG.  c /opt/HE/etc/php.d/dba.ini
.....UG.    /opt/HE/lib/php/modules/dba.so
.....UG.  c /opt/HE/etc/php.d/hash.ini
.....UG.    /opt/HE/lib/php/modules/hash.so
.....UG.  c /opt/HE/etc/php.d/json.ini
.....UG.    /opt/HE/lib/php/modules/json.so
.....UG.  c /opt/HE/etc/php.d/mcrypt.ini
.....UG.    /opt/HE/lib/php/modules/mcrypt.so
.....UG.  c /opt/HE/etc/php.d/mysql.ini
.....UG.    /opt/HE/lib/php/modules/mysql.so
.....UG.  c /opt/HE/etc/php.d/pdo.ini
.....UG.    /opt/HE/lib/php/modules/pdo.so
.....UG.  c /opt/HE/etc/php.d/soap.ini
.....UG.    /opt/HE/lib/php/modules/soap.so
.....UG.  c /opt/HE/etc/php.d/tokenizer.ini
.....UG.    /opt/HE/lib/php/modules/tokenizer.so
S.5....T  c /etc/httpd/conf.d/HEphp5.conf
S.5....T  c /opt/HE/etc/php.ini
S.5....T    /opt/HE/lib/httpd/modules/libphp5.so
.....UG.  c /opt/HE/etc/php.d/xmlreader.ini
.....UG.    /opt/HE/lib/php/modules/xmlreader.so
S.5....T  c /etc/cron.d/he-check-updates
.M......    /usr/bin/event_rpcgen.py
S.5....T  c /etc/sudoers
S.5....T  c /etc/openldap/ldap.conf
S.5....T  c /etc/pam.d/system-auth
S.5....T  c /etc/ssh/sshd_config
S.5....T  c /etc/cron.d/sysstat
S.5....T  c /etc/sysconfig/sysstat
.....UG.  c /opt/HE/etc/php.d/openssl.ini
.....UG.    /opt/HE/lib/php/modules/openssl.so
.....UG.  c /opt/HE/etc/php.d/bcmath.ini
.....UG.    /opt/HE/lib/php/modules/bcmath.so
.....UG.  c /opt/HE/etc/php.d/ctype.ini
.....UG.    /opt/HE/lib/php/modules/ctype.so
.....UG.  c /opt/HE/etc/php.d/dom.ini
.....UG.    /opt/HE/lib/php/modules/dom.so
.....UG.  c /opt/HE/etc/php.d/gettext.ini
.....UG.    /opt/HE/lib/php/modules/gettext.so
.....UG.  c /opt/HE/etc/php.d/iconv.ini
.....UG.    /opt/HE/lib/php/modules/iconv.so
.....UG.  c /opt/HE/etc/php.d/mbstring.ini
.....UG.    /opt/HE/lib/php/modules/mbstring.so
.....UG.  c /opt/HE/etc/php.d/mhash.ini
.....UG.    /opt/HE/lib/php/modules/mhash.so
.....UG.  c /opt/HE/etc/php.d/mysqli.ini
.....UG.    /opt/HE/lib/php/modules/mysqli.so
.....UG.  c /opt/HE/etc/php.d/posix.ini
.....UG.    /opt/HE/lib/php/modules/posix.so
.....UG.  c /opt/HE/etc/php.d/sqlite.ini
.....UG.    /opt/HE/lib/php/modules/sqlite.so
.....UG.  c /opt/HE/etc/php.d/xmlwriter.ini
.....UG.    /opt/HE/lib/php/modules/xmlwriter.so
S.5....T  c /etc/httpd/conf/httpd.conf
.M....G.    /var/log/httpd
.M...UG.    /var/www/html
.....UG.  c /opt/HE/etc/php.d/pdo_sqlite.ini
.....UG.    /opt/HE/lib/php/modules/pdo_sqlite.so
.....UG.  c /opt/HE/etc/php.d/xsl.ini
.....UG.    /opt/HE/lib/php/modules/xsl.so
S.5....T  c /etc/httpd/conf.d/ssl.conf
.M......    /usr/lib/vmware-tools/sbin32/vmware-hgfsmounter

gathered with

Code:

for i in $(rpm -qa | sed  "s/-[[:digit:]].*//g" ); do rpm -V $i; done

To be honest this totally overwhelms me. First real incident quest I am on. And the lack of Aide/samhain/tripwire clearly show me missing skills and knowledge about good system adminsitration

Hope you bear with me. Your help is very welcome and needed.

zhjim · 11-26-2009, 10:25 AM

All in all the reports seem not bad to me. There are some regards to ssh config with root allowed and protocol 1 still allowed, but the rest seems alright (to some degree).
Any input on this? Did I missed out on a critical part?

unSpawn · 11-26-2009, 05:54 PM

Quote:

Originally Posted by zhjim

All in all the reports seem not bad to me.

Well, yes and no...
- You already indicated having issues with APC. Searching for APC + segfaults gets you loads of posts and bug reports investigate. That would be a regular admin task: find out what version you run (check for updates?), check how it's compiled (maybe temporarily remove optimization during recompile?), check what caching method(s) you use.
- As this is a "test server" I wonder how it gets hit by scanners. To me only production servers, or rather their (web application) firewall or (reverse) proxy should be exposed to the 'net (and depending on network topology they probably should be in a separate routing segment, isolated from your LAN segments). To me a staging or test server is not a host one would expose to the 'net and certainly not allow unrestricted access to.
- Your logs show severe problems in terms of access: possibly bad SSH settings (protocol 1?, root access?), exposed RPC services and whatnot, no firewall rules. That is asking for trouble.

Since this is a testing server I assert it's expendable and I therefore wonder how much time we should spend investigating (would make good practicing though: your call). Regardless I suggest you raise your firewall to only allow traffic from and to your management IP (range), look closely at running processes and files in (daemon or publicly) accessable temp dirs, review your logs more thoroughly (broader view: not focus solely on this specific incident) and review recent software installation / upgrade / update / (re)configuration.

// BTW, after reviewing your logs I decided to remove the links because they show your hosts FQDN and you do not want that.

zhjim · 11-27-2009, 03:24 AM

Quote:

Originally Posted by unSpawn

Well, yes and no...
- You already indicated having issues with APC. Searching for APC + segfaults gets you loads of posts and bug reports investigate. That would be a regular admin task: find out what version you run (check for updates?), check how it's compiled (maybe temporarily remove optimization during recompile?), check what caching method(s) you use.

Quote:

Originally Posted by unSpawn

- As this is a "test server" I wonder how it gets hit by scanners. To me only production servers, or rather their (web application) firewall or (reverse) proxy should be exposed to the 'net (and depending on network topology they probably should be in a separate routing segment, isolated from your LAN segments). To me a staging or test server is not a host one would expose to the 'net and certainly not allow unrestricted access to.

This one is used by a variety of developers some inhouse some outhouse. As our company does not use VPN it would be a big hassle to have proper access policy. I know that with VPN on the server we could handle that but what am I to say I'm in charge for doing things but not for deciding things. It's all a bit weired.
I hope through this incident to raise some more attention to such security related topics.

Quote:

Originally Posted by unSpawn

- Your logs show severe problems in terms of access: possibly bad SSH settings (protocol 1?, root access?), exposed RPC services and whatnot, no firewall rules. That is asking for trouble.

Totaly agree on this. Will work on this.

Quote:

Originally Posted by unSpawn

Since this is a testing server I assert it's expendable and I therefore wonder how much time we should spend investigating (would make good practicing though: your call). Regardless I suggest you raise your firewall to only allow traffic from and to your management IP (range), look closely at running processes and files in (daemon or publicly) accessable temp dirs, review your logs more thoroughly (broader view: not focus solely on this specific incident) and review recent software installation / upgrade / update / (re)configuration.

I'll take it as a test for me to higher my skills.

Quote:

Originally Posted by unSpawn

// BTW, after reviewing your logs I decided to remove the links because they show your hosts FQDN and you do not want that.

I knew I forgot something. Thanks for the heads-up.

I guess those things have to happen till I wake up. Thank you unspawn for your efforts and none the less to the other two poster. You helped me get my head around. (And showed me my weakness I have in proper administration).
I'll see if i can get the hosting company to make a copy of the files before I reinstall from backups last week.

Cheers Zhjim