Linux - SecurityThis forum is for all security related questions.
Questions, tips, system compromises, firewalls, etc. are all included here.
Notices
Welcome to LinuxQuestions.org, a friendly and active Linux Community.
You are currently viewing LQ as a guest. By joining our community you will have the ability to post topics, receive our newsletter, use the advanced search, subscribe to threads and access many other special features. Registration is quick, simple and absolutely free. Join our community today!
Note that registered members see fewer ads, and ContentLink is completely disabled once you log in.
If you have any problems with the registration process or your account login, please contact us. If you need to reset your password, click here.
Having a problem logging in? Please visit this page to clear all LQ-related cookies.
Get a virtual cloud desktop with the Linux distro that you want in less than five minutes with Shells! With over 10 pre-installed distros to choose from, the worry-free installation life is here! Whether you are a digital nomad or just looking for flexibility, Shells can put your Linux machine on the device that you want to use.
Exclusive for LQ members, get up to 45% off per month. Click here for more info.
As far as I know this just does not work on apache. But the strange thing is that on Sunday the 20 this little thing happened
Code:
[Sun Nov 22 04:02:06 2009] [notice] SIGHUP received. Attempting to restart
[Sun Nov 22 04:02:06 2009] [notice] seg fault or similar nasty error detected in the parent process
The SIGHUP should be coming from logrotate but why is it crashing the apache?
This is a composite of my error and my access log. As there are more Segfaults than access they might not really match.
My question at last is how can I find out what realy happen? Is it wise to save the machine for analysis?
Reinstallation is planned for today (also I found no rootkit or file alteration with chkrootkit and rkhunter)
Did I miss out on anything? You need more info?
Best Regards
Zhjim
As far as I know this just does not work on apache. But the strange thing is that on Sunday the 20 this little thing happened
Code:
[Sun Nov 22 04:02:06 2009] [notice] SIGHUP received. Attempting to restart
[Sun Nov 22 04:02:06 2009] [notice] seg fault or similar nasty error detected in the parent process
The SIGHUP should be coming from logrotate but why is it crashing the apache?
This is a composite of my error and my access log. As there are more Segfaults than access they might not really match.
My question at last is how can I find out what realy happen? Is it wise to save the machine for analysis?
Reinstallation is planned for today (also I found no rootkit or file alteration with chkrootkit and rkhunter)
Did I miss out on anything? You need more info?
Best Regards
Zhjim
I would try just recompiling apache before doing a full resinstall.
Does it only Seg fault when setup.php for phpmyadmin is accessed?
@anomie Your totally right that I don't suffer from the named DoS. But that does not mean there wasn't one.
Anyways all of the access are to not existing URI's. Here is a better example of seg faults
Code:
216.177.204.39 - - [20/Nov/2009:21:44:29 +0100] "GET /boutique/install.txt HTTP/1.1" 301 342 "-" "Toata dragostea mea pentru diavola" 195 597 164 -
216.177.204.39 - - [20/Nov/2009:21:44:29 +0100] "GET /cart/install.txt HTTP/1.1" 301 338 "-" "Toata dragostea mea pentru diavola" 191 589 139 -
216.177.204.39 - - [20/Nov/2009:21:44:29 +0100] "GET /store/install.txt HTTP/1.1" 301 339 "-" "Toata dragostea mea pentru diavola" 192 591 198 -
[Fri Nov 20 21:44:29 2009] [notice] child pid 14636 exit signal Segmentation fault (11)
[Fri Nov 20 21:44:29 2009] [notice] child pid 14637 exit signal Segmentation fault (11)
[Fri Nov 20 21:44:29 2009] [notice] child pid 14638 exit signal Segmentation fault (11)
They for sure match up.
The only reason why I'm upset is that a normal SIGHUP crashes the server right after a row of attacks. Might just be that some of the logs are missing. Like it could be after a break in.
I'm still unsure what to do. And reinstall is painless it brings on no changes
Please take care to address all replies made and not just the last one, TIA.
Quote:
Originally Posted by zhjim
Code:
"Toata dragostea mea pentru diavola"
Indeed not a DoS but a vulnerability scanner (searching for the line should show). Shown non-existent URI's return 301 ("Moved permanently") and there is no evidence (debugging information) that any of the logged lines segfaulted any particular HTTPd child. (For all we know it could be caused by a recent added or changed .so object.) The IP you last posted in the 216.177.192.0/19 (AS AS22364) range is a known scanner (see Dshield). There's also no evidence of protective measures you put deployed to regulate ("iptables -m recent"?) or restrict access (mod_security?). So you're both right we're missing some and on top of that basic system administration information:
- details about any legitimate software installation / upgrade / update / (re)configuration,
- complete logs (contact me by email if you want to share data off-site but do note that sending large emails or attachments w/o my prior and explicit request will get you blacklisted without further response from me.)
- complete Chkrootkit / Tiger / Rootkit Hunter / package manager (verify mode) (/ Aide, Samhain or even tripwire) reporting.
Please take care to address all replies made and not just the last one, TIA.
My apologise for that.
I did not try out if setup.php segfaults the daemon cause I did not bring it up again. Cause to this is that I'm unsure what to do and don't want them to have a target again. Somehow silly cause it would only take a few minutes to check on this...
Some further research showed that it almost certain is APC (php cache) is causing this segfaults as well as the segfault of the parent process.
Quote:
Originally Posted by unSpawn
Indeed not a DoS but a vulnerability scanner (searching for the line should show). Shown non-existent URI's return 301 ("Moved permanently") and there is no evidence (debugging information) that any of the logged lines segfaulted any particular HTTPd child. (For all we know it could be caused by a recent added or changed .so object.)
Hm to be frankly I dunno how to give debugging information. Don't know any way to debug php applications that are called from apache...
But to me it seems that the lines from the error log
Code:
[Wed Nov 18 11:50:42 2009] [notice] child pid 9377 exit signal Segmentation fault (11)
are an evidence for it. This is a bad example but the lines in post #4 clearly show it.
Or am I misinterpreting things?
Quote:
Originally Posted by unSpawn
The IP you last posted in the 216.177.192.0/19 (AS AS22364) range is a known scanner (see Dshield). There's also no evidence of protective measures you put deployed to regulate ("iptables -m recent"?) or restrict access (mod_security?). So you're both right we're missing some and on top of that basic system administration information:
- details about any legitimate software installation / upgrade / update / (re)configuration,
- complete logs (contact me by email if you want to share data off-site but do note that sending large emails or attachments w/o my prior and explicit request will get you blacklisted without further response from me.)
- complete Chkrootkit / Tiger / Rootkit Hunter / package manager (verify mode) (/ Aide, Samhain or even tripwire) reporting.
I will gather the needed information and post them later. But will take some time.
Here are the logfiles for
[URL REMOVED]tiger[]
[URL REMOVED]rkhunter[]
[URL REMOVED]chkrootkit[]
and rpm -V for all the packages.
Code:
....L... /dev/MAKEDEV
S.5....T c /etc/updatedb.conf
S.5....T c /etc/sysconfig/rhn/sources
S.5....T /opt/HE/share/ghostscript/8.60/lib/Fontmap.GS
S.5....T c /etc/webalizer.conf
.....UG. /var/www/usage
.....UG. /var/www/usage/msfree.png
.....UG. /var/www/usage/webalizer.png
Unsatisfied dependencies for pam_mysql-0.6.2-2.rhel4.i386: libmysqlclient.so.14, libmysqlclient.so.14(libmysqlclient_14)
.....UG. /opt/HE/nagios/.ssh
.....UG. /opt/HE/nagios/.ssh/authorized_keys
.....UG. c /opt/HE/etc/php.d/curl.ini
.....UG. /opt/HE/lib/php/modules/curl.so
.....UG. c /opt/HE/etc/php.d/gd.ini
.....UG. /opt/HE/lib/php/modules/gd.so
S.5....T c /etc/ntp.conf
.....UG. c /opt/HE/etc/php.d/bz2.ini
.....UG. /opt/HE/lib/php/modules/bz2.so
.....UG. c /opt/HE/etc/php.d/dba.ini
.....UG. /opt/HE/lib/php/modules/dba.so
.....UG. c /opt/HE/etc/php.d/hash.ini
.....UG. /opt/HE/lib/php/modules/hash.so
.....UG. c /opt/HE/etc/php.d/json.ini
.....UG. /opt/HE/lib/php/modules/json.so
.....UG. c /opt/HE/etc/php.d/mcrypt.ini
.....UG. /opt/HE/lib/php/modules/mcrypt.so
.....UG. c /opt/HE/etc/php.d/mysql.ini
.....UG. /opt/HE/lib/php/modules/mysql.so
.....UG. c /opt/HE/etc/php.d/pdo.ini
.....UG. /opt/HE/lib/php/modules/pdo.so
.....UG. c /opt/HE/etc/php.d/soap.ini
.....UG. /opt/HE/lib/php/modules/soap.so
.....UG. c /opt/HE/etc/php.d/tokenizer.ini
.....UG. /opt/HE/lib/php/modules/tokenizer.so
S.5....T c /etc/httpd/conf.d/HEphp5.conf
S.5....T c /opt/HE/etc/php.ini
S.5....T /opt/HE/lib/httpd/modules/libphp5.so
.....UG. c /opt/HE/etc/php.d/xmlreader.ini
.....UG. /opt/HE/lib/php/modules/xmlreader.so
S.5....T c /etc/cron.d/he-check-updates
.M...... /usr/bin/event_rpcgen.py
S.5....T c /etc/sudoers
S.5....T c /etc/openldap/ldap.conf
S.5....T c /etc/pam.d/system-auth
S.5....T c /etc/ssh/sshd_config
S.5....T c /etc/cron.d/sysstat
S.5....T c /etc/sysconfig/sysstat
.....UG. c /opt/HE/etc/php.d/openssl.ini
.....UG. /opt/HE/lib/php/modules/openssl.so
.....UG. c /opt/HE/etc/php.d/bcmath.ini
.....UG. /opt/HE/lib/php/modules/bcmath.so
.....UG. c /opt/HE/etc/php.d/ctype.ini
.....UG. /opt/HE/lib/php/modules/ctype.so
.....UG. c /opt/HE/etc/php.d/dom.ini
.....UG. /opt/HE/lib/php/modules/dom.so
.....UG. c /opt/HE/etc/php.d/gettext.ini
.....UG. /opt/HE/lib/php/modules/gettext.so
.....UG. c /opt/HE/etc/php.d/iconv.ini
.....UG. /opt/HE/lib/php/modules/iconv.so
.....UG. c /opt/HE/etc/php.d/mbstring.ini
.....UG. /opt/HE/lib/php/modules/mbstring.so
.....UG. c /opt/HE/etc/php.d/mhash.ini
.....UG. /opt/HE/lib/php/modules/mhash.so
.....UG. c /opt/HE/etc/php.d/mysqli.ini
.....UG. /opt/HE/lib/php/modules/mysqli.so
.....UG. c /opt/HE/etc/php.d/posix.ini
.....UG. /opt/HE/lib/php/modules/posix.so
.....UG. c /opt/HE/etc/php.d/sqlite.ini
.....UG. /opt/HE/lib/php/modules/sqlite.so
.....UG. c /opt/HE/etc/php.d/xmlwriter.ini
.....UG. /opt/HE/lib/php/modules/xmlwriter.so
S.5....T c /etc/httpd/conf/httpd.conf
.M....G. /var/log/httpd
.M...UG. /var/www/html
.....UG. c /opt/HE/etc/php.d/pdo_sqlite.ini
.....UG. /opt/HE/lib/php/modules/pdo_sqlite.so
.....UG. c /opt/HE/etc/php.d/xsl.ini
.....UG. /opt/HE/lib/php/modules/xsl.so
S.5....T c /etc/httpd/conf.d/ssl.conf
.M...... /usr/lib/vmware-tools/sbin32/vmware-hgfsmounter
gathered with
Code:
for i in $(rpm -qa | sed "s/-[[:digit:]].*//g" ); do rpm -V $i; done
To be honest this totally overwhelms me. First real incident quest I am on. And the lack of Aide/samhain/tripwire clearly show me missing skills and knowledge about good system adminsitration
Hope you bear with me. Your help is very welcome and needed.
Last edited by unSpawn; 11-26-2009 at 04:35 PM.
Reason: //Removed URIs
All in all the reports seem not bad to me. There are some regards to ssh config with root allowed and protocol 1 still allowed, but the rest seems alright (to some degree).
Any input on this? Did I missed out on a critical part?
Well, yes and no...
- You already indicated having issues with APC. Searching for APC + segfaults gets you loads of posts and bug reports investigate. That would be a regular admin task: find out what version you run (check for updates?), check how it's compiled (maybe temporarily remove optimization during recompile?), check what caching method(s) you use.
- As this is a "test server" I wonder how it gets hit by scanners. To me only production servers, or rather their (web application) firewall or (reverse) proxy should be exposed to the 'net (and depending on network topology they probably should be in a separate routing segment, isolated from your LAN segments). To me a staging or test server is not a host one would expose to the 'net and certainly not allow unrestricted access to.
- Your logs show severe problems in terms of access: possibly bad SSH settings (protocol 1?, root access?), exposed RPC services and whatnot, no firewall rules. That is asking for trouble.
Since this is a testing server I assert it's expendable and I therefore wonder how much time we should spend investigating (would make good practicing though: your call). Regardless I suggest you raise your firewall to only allow traffic from and to your management IP (range), look closely at running processes and files in (daemon or publicly) accessable temp dirs, review your logs more thoroughly (broader view: not focus solely on this specific incident) and review recent software installation / upgrade / update / (re)configuration.
// BTW, after reviewing your logs I decided to remove the links because they show your hosts FQDN and you do not want that.
Well, yes and no...
- You already indicated having issues with APC. Searching for APC + segfaults gets you loads of posts and bug reports investigate. That would be a regular admin task: find out what version you run (check for updates?), check how it's compiled (maybe temporarily remove optimization during recompile?), check what caching method(s) you use.
Quote:
Originally Posted by unSpawn
- As this is a "test server" I wonder how it gets hit by scanners. To me only production servers, or rather their (web application) firewall or (reverse) proxy should be exposed to the 'net (and depending on network topology they probably should be in a separate routing segment, isolated from your LAN segments). To me a staging or test server is not a host one would expose to the 'net and certainly not allow unrestricted access to.
This one is used by a variety of developers some inhouse some outhouse. As our company does not use VPN it would be a big hassle to have proper access policy. I know that with VPN on the server we could handle that but what am I to say I'm in charge for doing things but not for deciding things. It's all a bit weired.
I hope through this incident to raise some more attention to such security related topics.
Quote:
Originally Posted by unSpawn
- Your logs show severe problems in terms of access: possibly bad SSH settings (protocol 1?, root access?), exposed RPC services and whatnot, no firewall rules. That is asking for trouble.
Totaly agree on this. Will work on this.
Quote:
Originally Posted by unSpawn
Since this is a testing server I assert it's expendable and I therefore wonder how much time we should spend investigating (would make good practicing though: your call). Regardless I suggest you raise your firewall to only allow traffic from and to your management IP (range), look closely at running processes and files in (daemon or publicly) accessable temp dirs, review your logs more thoroughly (broader view: not focus solely on this specific incident) and review recent software installation / upgrade / update / (re)configuration.
I'll take it as a test for me to higher my skills.
Quote:
Originally Posted by unSpawn
// BTW, after reviewing your logs I decided to remove the links because they show your hosts FQDN and you do not want that.
I knew I forgot something. Thanks for the heads-up.
I guess those things have to happen till I wake up. Thank you unspawn for your efforts and none the less to the other two poster. You helped me get my head around. (And showed me my weakness I have in proper administration).
I'll see if i can get the hosting company to make a copy of the files before I reinstall from backups last week.
LinuxQuestions.org is looking for people interested in writing
Editorials, Articles, Reviews, and more. If you'd like to contribute
content, let us know.