CRON silent failure

jlinkels · 12-19-2011, 07:40 PM

Yesterday I edited my /etc/crontab, after that, no commands from the entire /etc/crontab were executed.

The error was mine of course, somewhere I entered

Code:

11 59 * * *

instead of

Code:

59 11 * * *

However, the problem I have is that NO commands at all from that crontab were executed after this error was entered. The crontab contains about 20 entries, the wrong entry was 4 lines from the end. After correcting the error scheduling resumed as usual.

All other cron jobs in other crontabs were executed normally.

What I think is extremely scary, is that all /etc/crontab jobs were suspended without any error message. Not in /var/log/messages, not in /var/log/syslog, no e-mails. Total silence. I didn't even miss the e-mails which reported normal crontab, some user told me something did not happen as expected.

I know Linux is extremely user friendly, at least at the point of logging and reporting errors. Is this a feature or a bug?

jlinkels

blue_print · 12-21-2011, 03:50 AM

What Distro you're using?

colucix · 12-21-2011, 04:10 AM

It seems the cron daemon at first checks the syntax of the cron jobs and if it encounters some errors, it refuses to run anything. I cannot reproduce this behaviour (using CRONIE 1.4.4, a fork of the original vixie-cron with security and configuration enhancements like the ability to use pam and SELinux, on CentOS 6.0), but at least I see the following entries in /var/log/cron:

Code:

Dec 21 10:57:01 ocean-4 crond[1570]: (*system*) RELOAD (/etc/crontab)
Dec 21 10:57:01 ocean-4 crond[1570]: (CRON) bad hour (/etc/crontab)
Dec 21 10:57:01 ocean-4 CROND[7226]: (root) CMD (date >> $HOME/cron.log.ouput 2>&1)

the content of /etc/crontab being

Code:

SHELL=/bin/bash
PATH=/sbin:/bin:/usr/sbin:/usr/bin
MAILTO=root
HOME=/

57 10 * * * root date >> $HOME/cron.log.ouput 2>&1
11 59 * * * root date >> $HOME/cron.log.error 2>&1

In this case the cron daemon notices the bad hour but it runs the other command successfully, as I can see from the newly created /cron.log.ouput file. Note that HOME in my crontab is set to /. At this point I am inclined to agree with you: maybe a bug in your cron version or an unwanted behaviour for sure.

jlinkels · 12-22-2011, 11:40 AM

@blueprint: I am using Debian. On this server it is still Lenny.

@Colucix: I didn't have a /var/log/cron.log. As it appeared, it is disabled by default in /etc/syslog.conf. I think that is extremely unsmart.

However, this is the result after I uncommented this line in syslog.conf:

Code:

cron.*                          /var/log/cron.log

Code:

Dec 22 17:00:01 homeserv /USR/SBIN/CRON[12726]: (jlinkels) CMD (echo 1 > /home/jlinkels/User_Files/iradio/mp_record)
Dec 22 17:09:01 homeserv /USR/SBIN/CRON[13812]: (root) CMD (  [ -d /var/lib/php4 ] && find /var/lib/php4/ -type f -cmin +$(/usr/lib/php4/maxlifetime) -print0 | xargs -r -0 rm)
Dec 22 17:09:01 homeserv /USR/SBIN/CRON[13813]: (root) CMD (  [ -x /usr/lib/php5/maxlifetime ] && [ -d /var/lib/php5 ] && find /var/lib/php5/ -type f -cmin +$(/usr/lib/php5/maxlifetime) -delete)

Dec 22 17:14:01 homeserv /USR/SBIN/CRON[14430]: (root) CMD (^I/usr/bin/vpn_keepalive.sh > /dev/null)
Dec 22 17:16:01 homeserv /USR/SBIN/CRON[14677]: (root) CMD (^I/usr/bin/vpn_keepalive.sh > /dev/null)
Dec 22 17:17:01 homeserv /USR/SBIN/CRON[14807]: (root) CMD (   run-parts --report /etc/cron.hourly)Dec 22 17:18:01 homeserv /USR/SBIN/CRON[14928]: (root) CMD (^I/usr/bin/vpn_keepalive.sh > /dev/null)
Dec 22 17:20:01 homeserv /USR/SBIN/CRON[15175]: (root) CMD (^I/usr/bin/vpn_keepalive.sh > /dev/null)
Dec 22 17:22:01 homeserv /USR/SBIN/CRON[15400]: (root) CMD (^I/usr/bin/vpn_keepalive.sh > /dev/null)
Dec 22 17:24:01 homeserv /USR/SBIN/CRON[15654]: (root) CMD (^I/usr/bin/vpn_keepalive.sh > /dev/null)

The red code shows what is logged while the error is in the crontab file. The vpn_keepalive script is not executed, and neither is an error message entered in cron.log.

Then, at 17:12:00 I remove the error from the /etc/crontab, and the vpn_keepalive is executed again, which is shown in the blue code. Everything which runs in the red code is not in the /etc/crontab but in the other cron files. The error in /etc/crontab is in a line after a call to vpn_keepalive.

Your assumption that cron gives up immediately after reading an error seems to be right. Another incorrect issue is that of course cron logging should be enabled by default.

Since this installation is rather old, I'll check this behaviour on a new installation and file a bug report if it persists.

Thanks for the useful feedback.

jlinkels

chrism01 · 12-22-2011, 07:01 PM

It could well be a an invisible ctrl char in crontab; try using vim with ':set list' to display ctrl chars.
Incidentally, '^I' is a tab & I've never seen that in a cron log file ... could be nothing to worry about

jlinkels · 12-23-2011, 12:33 AM

The problem is that cron is not logging a failure, not why the file might be invalid. The error in the file is know (invalid time) and was my fault.

I interpreted ^I as tab character as well. It seems that cron doesn't complain about that, I have seen it for many years in the syslog, and evertime I try to remember not to use tabs in cron files in vain.

jlinkels