As described in
this other thread, I noticed that my webalizer scripts were not running each night. After some troubleshooting, I think I have narrowed the problem down to logrotate.
What appears to be happening is that my cron.daily runs at 06:25 UTC and never completes because the logrotate script fails to complete and hangs forever. The result is that after several days, I had numerous cron.daily runs waiting on numerous logrotate runs which had somehow timed out indefinitely:
Code:
$ ps -aux | grep cron
Warning: bad ps syntax, perhaps a bogus '-'? See http://procps.sf.net/faq.html
root 567 0.0 0.0 21076 1016 ? Ss Aug18 0:02 cron
root 1575 0.0 0.0 7628 1004 pts/0 S+ 19:47 0:00 grep --color=auto cron
root 1814 0.0 0.0 4096 600 ? Ss Aug25 0:00 /bin/sh -c test -x /usr/sbin/anacron || ( cd / && run-parts --report /etc/cron.daily )
root 1815 0.0 0.0 4000 656 ? S Aug25 0:20 run-parts --report /etc/cron.daily
root 2713 0.0 0.0 4096 596 ? Ss Aug31 0:00 /bin/sh -c test -x /usr/sbin/anacron || ( cd / && run-parts --report /etc/cron.daily )
root 2714 0.0 0.0 4000 656 ? S Aug31 0:06 run-parts --report /etc/cron.daily
root 4702 0.0 0.0 4096 596 ? Ss Aug27 0:00 /bin/sh -c test -x /usr/sbin/anacron || ( cd / && run-parts --report /etc/cron.daily )
root 4703 0.0 0.0 4000 656 ? S Aug27 0:15 run-parts --report /etc/cron.daily
root 5141 0.0 0.0 4096 596 ? S Aug25 0:00 /bin/sh /etc/cron.daily/logrotate
root 5360 0.0 0.0 4096 592 ? S Aug27 0:00 /bin/sh /etc/cron.daily/logrotate
root 11235 0.0 0.0 4096 600 ? Ss Aug26 0:00 /bin/sh -c test -x /usr/sbin/anacron || ( cd / && run-parts --report /etc/cron.daily )
root 11236 0.0 0.0 4000 652 ? S Aug26 0:17 run-parts --report /etc/cron.daily
root 11726 0.0 0.0 4096 600 ? Ss Aug23 0:00 /bin/sh -c test -x /usr/sbin/anacron || ( cd / && run-parts --report /etc/cron.daily )
root 11727 0.0 0.0 4000 660 ? S Aug23 0:24 run-parts --report /etc/cron.daily
root 12105 0.0 0.0 4096 600 ? Ss Aug24 0:00 /bin/sh -c test -x /usr/sbin/anacron || ( cd / && run-parts --report /etc/cron.daily )
root 12106 0.0 0.0 4000 656 ? S Aug24 0:23 run-parts --report /etc/cron.daily
root 12169 0.0 0.0 4096 592 ? S Aug23 0:00 /bin/sh /etc/cron.daily/logrotate
root 12554 0.0 0.0 4096 592 ? S Aug31 0:00 /bin/sh /etc/cron.daily/logrotate
root 12561 0.0 0.0 4096 592 ? S Aug24 0:00 /bin/sh /etc/cron.daily/logrotate
root 14333 0.0 0.0 4096 600 ? Ss Sep01 0:00 /bin/sh -c test -x /usr/sbin/anacron || ( cd / && run-parts --report /etc/cron.daily )
root 14334 0.0 0.0 4000 652 ? S Sep01 0:03 run-parts --report /etc/cron.daily
root 14525 0.0 0.0 4096 596 ? Ss Aug30 0:00 /bin/sh -c test -x /usr/sbin/anacron || ( cd / && run-parts --report /etc/cron.daily )
root 14526 0.0 0.0 4000 652 ? S Aug30 0:08 run-parts --report /etc/cron.daily
root 14859 0.0 0.0 4096 592 ? S Aug26 0:00 /bin/sh /etc/cron.daily/logrotate
root 15213 0.0 0.0 4096 600 ? Ss Aug28 0:00 /bin/sh -c test -x /usr/sbin/anacron || ( cd / && run-parts --report /etc/cron.daily )
root 15214 0.0 0.0 4000 652 ? S Aug28 0:12 run-parts --report /etc/cron.daily
root 16820 0.0 0.0 4096 596 ? Ss Aug29 0:00 /bin/sh -c test -x /usr/sbin/anacron || ( cd / && run-parts --report /etc/cron.daily )
root 16821 0.0 0.0 4000 652 ? S Aug29 0:11 run-parts --report /etc/cron.daily
root 17310 0.0 0.0 4096 596 ? S 06:50 0:00 /bin/sh /etc/cron.daily/logrotate
root 21015 0.0 0.0 4096 592 ? S Aug29 0:00 /bin/sh /etc/cron.daily/logrotate
root 22821 0.0 0.0 4096 596 ? S Aug30 0:00 /bin/sh /etc/cron.daily/logrotate
root 23400 0.0 0.0 4096 592 ? S Aug28 0:00 /bin/sh /etc/cron.daily/logrotate
root 31289 0.0 0.0 4096 596 ? Ss Aug22 0:00 /bin/sh -c test -x /usr/sbin/anacron || ( cd / && run-parts --report /etc/cron.daily )
root 31290 0.0 0.0 4000 652 ? S Aug22 0:26 run-parts --report /etc/cron.daily
root 31306 0.0 0.0 4096 592 ? S Sep01 0:00 /bin/sh /etc/cron.daily/logrotate
root 31460 0.0 0.0 4096 596 ? S Aug22 0:00 /bin/sh /etc/cron.daily/logrotate
root 32749 0.0 0.0 4096 600 ? Ss 06:25 0:00 /bin/sh -c test -x /usr/sbin/anacron || ( cd / && run-parts --report /etc/cron.daily )
root 32750 0.0 0.0 4000 656 ? S 06:25 0:01 run-parts --report /etc/cron.daily
I am not certain, but I believe I can blame my tweaked version of
this 'untested' logrotate configuration recommended on the samhain website:
Code:
$ sudo cat /etc/logrotate.d/samhain
/var/log/samhain/*.log {
daily
rotate 365
size 5000k
nocreate
compress
dateext
mail dev@myplan.com
mailfirst
prerotate
if test -f /var/run/samhain/samhain.pid; then \
PIN=`cat /var/run/samhain/samhain.pid`; \
/bin/kill -TTI $PIN; \
sleep 1; \
AA=0; \
while test "x$AA" != "x120"; do \
let "AA = $AA + 1"; \
if test -f /var/log/samhain/samhain.log.lock; then \
sleep 1; \
else \
break; \
fi \
done; \
fi
endscript
}
When I used pkill to halt the various stuck processes above, I got a notification from the cron daemon:
Code:
/etc/cron.daily/logrotate:
ERROR: unknown signal name "TTI".
Usage:
kill pid ... Send SIGTERM to every process listed.
kill signal pid ... Send a signal to every process listed.
kill -s signal pid ... Send a signal to every process listed.
kill -l List all signal names.
kill -L List all signal names in a nice table.
kill -l signal Convert between signal numbers and names.
logrotate_script: 15: let: not found
logrotate_script: 15: let: not found
logrotate_script: 15: let: not found
logrotate_script: 15: let: not found
...
Can anyone tell me how to fix this issue?
