100's of "apache-init-server"

deathsfriend99 · 09-25-2010, 01:33 PM

I have a VPS that has 512MB of ram. I'm using it as a mail/web server. It keeps running out of memory. I know amavis/clamav are memory hogs, but I checked my ps aux and found 100's of instances of "apache-init-server" running. I killed them all, and they keep spawning back. What could be causing this. I've never seen this on a webserver before.

OS: CentOS 5.5

bathory · 09-25-2010, 03:57 PM

Hi,

You should take a look at apache logs and the output of

Code:

netstat -tan|grep 80|grep EST

unSpawn · 09-25-2010, 04:23 PM

In addition to that, I just don't know one by the name of "apache-init-server", could you please save data like this: '(/bin/ps axfwwwe; /usr/sbin/lsof -Pwln; /bin/ls -al /var/spool/cron; locate apache-init-server; netstat -antpe; lastlog; last; who -a) > /dev/shm/data.txt', scrub sensitive data if necessary and attach the plain text "data.txt" file? After attaching I'd bring down the web server and mail daemon just in case. Checking system and daemon logs is a good thing to do. Also please tell us what forum, web log, admin tool, web-based panel, etc, etc, you run, possibly with versions.

deathsfriend99 · 09-26-2010, 09:50 AM

[QUOTE=unSpawn;4108919]In addition to that, I just don't know one by the name of "apache-init-server", could you please save data like this: '(/bin/ps axfwwwe; /usr/sbin/lsof -Pwln; /bin/ls -al /var/spool/cron; locate apache-init-server; netstat -antpe; lastlog; last; who -a) > /dev/shm/data.txt', scrub sensitive data if necessary and attach the plain text "data.txt" file? After attaching I'd bring down the web server and mail daemon just in case. Checking system and daemon logs is a good thing to do. [\Quote]

I can't get the data.txt attached. It's too big for the forums max, but the netstat shows where these are coming from:

Active Internet connections (servers and established)
Proto Recv-Q Send-Q Local Address Foreign Address State User Inode PID/Program name
tcp 0 0 127.0.0.1:10024 0.0.0.0:* LISTEN 100 132951431 21813/amavisd (mast
tcp 0 0 127.0.0.1:10025 0.0.0.0:* LISTEN 0 132951797 21874/master
tcp 0 0 0.0.0.0:3306 0.0.0.0:* LISTEN 27 132951128 21701/mysqld
tcp 0 0 127.0.0.1:3310 0.0.0.0:* LISTEN 101 132950955 21613/clamd
tcp 0 0 0.0.0.0:40404 0.0.0.0:* LISTEN 48 132955866 22398/apache-init-s
tcp 0 0 127.0.0.1:631 0.0.0.0:* LISTEN 0 132950654 21535/cupsd
tcp 0 0 127.0.0.1:25 0.0.0.0:* LISTEN 0 132951679 21874/master
tcp 0 0 my.host.ip:25 0.0.0.0:* LISTEN 0 132951678 21874/master
tcp 0 1 my.host.ip:40945 200.229.199.198:25 LAST_ACK 0 0 -
tcp 0 0 127.0.0.1:10025 127.0.0.1:59516 ESTABLISHED 89 132963212 27905/smtpd
tcp 0 0 my.host.ip:59358 217.124.183.2:25 ESTABLISHED 89 132970149 21896/smtp
tcp 49 0 127.0.0.1:41139 127.0.0.1:10025 CLOSE_WAIT 100 132953245 21947/amavisd (ch1-
tcp 0 0 127.0.0.1:10025 127.0.0.1:41139 FIN_WAIT2 0 0 -
tcp 0 0 127.0.0.1:59516 127.0.0.1:10025 ESTABLISHED 100 132963199 21948/amavisd (ch1-
tcp 0 0 my.host.ip:51049 212.85.64.68:6667 ESTABLISHED 48 132971146 28018/apache-init-s
tcp 0 1 my.host.ip:57129 212.85.64.68:6667 SYN_SENT 48 132978931 28670/apache-init-s
tcp 0 1 my.host.ip:53922 212.85.64.68:6667 SYN_SENT 48 132978886 30339/apache-init-s
tcp 0 0 my.host.ip:47071 212.85.64.68:6667 ESTABLISHED 48 132978975 28406/apache-init-s
tcp 0 1 my.host.ip:40452 212.85.64.68:6667 SYN_SENT 48 132978820 32015/apache-init-s
tcp 0 0 my.host.ip:38295 212.85.64.68:6667 ESTABLISHED 48 132976381 22398/apache-init-s
tcp 0 0 :::143 :::* LISTEN 0 132951218 21743/dovecot
tcp 0 0 :::80 :::* LISTEN 0 132954239 22000/httpd
tcp 0 0 :::21 :::* LISTEN 99 132954044 21988/proftpd: (acc
tcp 0 0 :::22 :::* LISTEN 0 132950607 21526/sshd
tcp 0 0 :::993 :::* LISTEN 0 132951219 21743/dovecot
tcp 0 0 ::ffff:my.host.ip:22 ::ffff:my.home.ip:1284 ESTABLISHED 0 132950830 21591/0

No idea what these's IP's are. They trace to Swiss IP's.
I could not locate apache-init-server anywhere on the machine.
I can't bring down the server, it's my only webserver at the moment.

Quote:

Originally Posted by unSpawn

Also please tell us what forum, web log, admin tool, web-based panel, etc, etc, you run, possibly with versions.

I am running Swiftpanel 1.6 (Gameserver control panel)

Here is the full data.txt:
https://docs.google.com/document/edi...thkey=CLir4MYI

deathsfriend99 · 09-26-2010, 11:27 AM

I did find this file, along with a bunch of other files that didn't seem kosher in /var/tmp/ They were owned by apache. I don't know if they came with the VPS install or what, but they weren't mine. Looked like IRC bots and old stuff from a PBX server. Really odd. I removed them and that seems to have solved the problem for now. I ran a rootkit check just to be safe and it came up with nothing. Not sure how VPS's work, but hopefully a security compromise on someone else's VPS doesn't compromise mine. Not sure if the VPS company may have left files in the /var/tmp directory, but it seems odd to do so.

unSpawn · 09-26-2010, 03:31 PM

From data.txt:

Code:

]$ grep init data.txt
(..)
28018 ? Ss 0:00 apache-init-server SHELL=/bin/sh USER=apache PATH=. PWD=/var/tmp/vi.recover SHLVL=3 HOME=/var/www LOGNAME=apache _=./apache-init-server

]$ grep 28018 data.txt
apache-in 28018   48  cwd DIR   0,148 4096   2687362 /var/tmp/vi.recover
apache-in 28018   48  rtd DIR   0,148 4096  26495303 /
apache-in 28018   48  txt REG   0,148   504464  12132376 /var/tmp/vi.recover/apache-init-server
apache-in 28018   48  mem REG  8,6 12132376 /var/tmp/vi.recover/apache-init-server
(..)
apache-in 28018   48 0w   REG   0,148   34  12263490 /var/tmp/vi.recover/LinkEvents
apache-in 28018   48 1u  IPv4   132971146  TCP my.host.ip:51049->212.85.64.68:6667 (ESTABLISHED)
apache-in 28018   48 3u  IPv4   132964998  UDP *:34975 
tcp 0  0 my.host.ip:51049 212.85.64.68:6667 ESTABLISHED 48  132971146  28018/apache-init-s

So the created directory is /var/tmp/vi.recover, the processes run as apache and the result is an IRC bot (EnergyMech to be exactly) whose argv[0] is mimicking something apache-like. Nice try.

Quote:

Originally Posted by deathsfriend99

I can't bring down the server, it's my only webserver at the moment.

What's more important? Running an 0wned web stack or providing services in a stable and secure way?

Quote:

Originally Posted by deathsfriend99

I did find this file, along with a bunch of other files (..) I removed them

That's rather unfortunate for all sorts of reasons.

Quote:

Originally Posted by deathsfriend99

and that seems to have solved the problem for now

Treating symptoms is not the same as treating the cause. Shut down your web server. Raise the firewall to deny any traffic except SSH from your management IP/range (BTW you logging in as root over SSH is real bad) while you're investigating. Now run all system and daemon logs through Logwatch with the "--detail High --service All --range All --archives --numeric --save /tmp/logwatch.log" args. Please attach, pastebin, docs.google or email the file.

paulsm4 · 09-26-2010, 03:37 PM

Or have your provider re-initialize your VPS.

The sad truth, deathsfriend99, is that your system has been *compromised*, and you *cannot* try to use it as-is.

The Good News is that it's a VPS (and not a physical system, which you might have to re-install from scratch).

Strong suggestion: nuke your "infected" VPS, and start again with a "clean" one.

IMHO...

PS here are unSpawn's commands, in a slightly different format:

Code:

set LOG=/dev/shm/data.txt
date >> $LOG
/bin/ps axfwwwe >> $LOG
/usr/sbin/lsof -Pwln>> $LOG
/bin/ls -al /var/spool/cro n>> $LOG
locate apache-init-server >> $LOG
netstat -antpe >> $LOG
lastlog >> $LOG
last >> $LOG
who -a  >> $LOG
echo >> $LOG

GrapefruiTgirl · 09-26-2010, 03:49 PM

Moved: This thread is more suitable in Security and has been moved accordingly to help your thread/question get the exposure it deserves.

unSpawn · 09-26-2010, 03:52 PM

Quote:

Originally Posted by paulsm4

Or have your provider re-initialize your VPS. (..) your system has been *compromised*, and you *cannot* try to use it as-is. (..) nuke your "infected" VPS, and start again with a "clean" one. IMHO...

Just starting afresh, without investigation, without precautions, might re-introduce the hole and get the OPs web stack compromised again faster than you can say "root me please". So please don't.

unSpawn · 09-29-2010, 10:12 AM

Jean-michels posts have been moved to his own thread to avoid confusion.

unSpawn · 09-30-2010, 03:26 PM

* Apparently there are some scripts that (automagically re-)initialize the users crontab so to stop it from running one could stop the Cron daemon or in Bash deny any user from running crontabs by issuing '$>/etc/cron.allow' while one investigates.