Cron job starts 6 hours late, rest of cron jobs work fine

pieterhouwen · 09-29-2015, 03:47 AM

Quote:

Originally Posted by JeremyBoden

No.

You have a cron job (called apt) that appears to be trying to run roughly every 30 minutes.
Many cron jobs can run simultaneously.

Then I don't know what the problem is, in crontab it says to run all in cron.daily at 00:01 , but all of the cron jobs still start 6 hours late.

JeremyBoden · 09-29-2015, 05:30 AM

Quote:

Originally Posted by cgtueno

(Smiles) Point taken. But I am a purist and prefer to edit crontab directly.
C.

You do realise that standard type crontab's are stored in /var/spool/cron/crontabs/username?
That includes root.
If you read

Code:

man cron

you will see that stuff in /etc/crontab behaves differently.

What are the contents of your /etc/crontab (feel free to obliterate sensitive information)?

pieterhouwen · 09-29-2015, 05:58 AM

Quote:

Originally Posted by pieterhouwen

While "nano /etc/crontab" gives me:

SHELL=/bin/sh
PATH=/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin

# m h dom mon dow user command
17 * * * * root cd / && run-parts --report /etc/cron.hourly
01 0 * * * root test -x /usr/sbin/anacron || ( cd / && run-parts --report /etc/cron.daily )
47 4 * * 7 root test -x /usr/sbin/anacron || ( cd / && run-parts --report /etc/cron.weekly )
52 5 1 * * root test -x /usr/sbin/anacron || ( cd / && run-parts --report /etc/cron.monthly )
#

This is my crontab

michaelk · 09-29-2015, 06:31 AM

Quote:

# m h dom mon dow user command
17 * * * * root cd / && run-parts --report /etc/cron.hourly
01 0 * * * root test -x /usr/sbin/anacron || ( cd / && run-parts --report /etc/cron.daily )
47 4 * * 7 root test -x /usr/sbin/anacron || ( cd / && run-parts --report /etc/cron.weekly )
52 5 1 * * root test -x /usr/sbin/anacron || ( cd / && run-parts --report /etc/cron.monthly )

What jobs are starting late?

In a nutshell the system cron jobs (daily, weekly, monthly) are configured such that if the computer is not running 24/7 they will run no matter what via anacron vs crond. I can only guess that with this particular server something between anacron and crond is not configured like the others.

Have you verified that crond is running? Is cron.hourly running?

pieterhouwen · 09-29-2015, 06:35 AM

Quote:

Originally Posted by michaelk

What jobs are starting late?

In a nutshell the system cron jobs (daily, weekly, monthly) are configured such that if the computer is not running 24/7 they will run no matter what via anacron vs crond. I can only guess that with this particular server something between anacron and crond is not configured like the others.

Have you verified that crond is running? Is cron.hourly running?

ps -ef | grep cron gives me the following:

root@caccini:~# ps -ef | grep cron
root 1334 1 0 Mar16 ? 00:00:37 cron
root 10210 1 0 07:30 ? 00:00:00 anacron -s
root 10429 10210 0 07:35 ? 00:00:00 /bin/sh -c run-parts --report /etc/cron.daily
root 10431 10429 0 07:35 ? 00:00:00 run-parts --report /etc/cron.daily
root 10540 10431 0 07:35 ? 00:00:00 /bin/bash /etc/cron.daily/[redacted]-data-array
root 22480 22416 0 13:33 pts/1 00:00:00 grep --color=auto cron

zhjim · 09-29-2015, 06:42 AM

Quote:

Originally Posted by pieterhouwen

I re-ran the command a little different now, output is below:

root@caccini:/etc# grep "sleep" cron.*/*
cron.daily/apt:# sleep for a random interval of time (default 30min)
cron.daily/apt:random_sleep()
cron.daily/apt: debug_echo "sleeping for $TIME seconds"
cron.daily/apt: sleep $TIME
cron.daily/apt:# sleep random amount of time to avoid hitting the
cron.daily/apt:random_sleep

Could this be the source of the problems?

The sleeping interval within apt cronjob is total okay. It makes sure that the apt system is not overrun with simultaneous access. And it only affects this single cronjob.

@Jeremy The apt cronjob only runs once through /etc/cron.daily. It just has this random sleep thing which is max 30 minutes.

Recheck the times. Once at the failing system and right after or before on a working system.
One can also set a timezone for cron specificly. Check out /etc/default/cron for a TZ stanca. As well as /etc/crontab which we already have and its not in there.
Also check out /etc/timzeone for the correct zone. Or the enviroment variable CRON_TZ and TZ itself.

Code:

printenv | grep -i tz

Also restart the cron daemon after fiddeling with timezone and date settings.

michaelk · 09-29-2015, 06:52 AM

Have you checked the log files to see if cron.hourly is running?

Looks like anacron is running the cron.daily job as you posted.

Have you compared the other servers? Are they configured the same?

pieterhouwen · 09-29-2015, 07:25 AM

Quote:

Originally Posted by michaelk

Have you checked the log files to see if cron.hourly is running?

Looks like anacron is running the cron.daily job as you posted.

Have you compared the other servers? Are they configured the same?

Yeah, I checked the other servers and they are now showing the same symptoms.
I did went back through my emails (I get a notification once a cron job completes), and the bug started after May 13th. The subject of the email also changed then.

It was (when the cron jobs were running as expected) "Cron <root@caccini> test -x /usr/sbin/anacron || ( cd / && run-parts --report /etc/cron.daily )" , but now it's "Anacron job 'cron.daily' on caccini" and the sender also changed from root@caccini to Anacron.

I have no idea what happened, I wasn't working with linux back then.
Any idea how to change this back?

michaelk · 09-29-2015, 08:39 AM

So basically anacron is set to start at 7:30 and cron.daily via anacrontab will start after a 5 minute delay (7:35) which is what you see in your post #20. As far as I know this is the default configuration. By the way anacron will also start at boot up. Not enough information to know what changed.

If anacron is not executable then the cron job will run at the time indicated

Quote:

01 0 * * * root test -x /usr/sbin/anacron || ( cd / && run-parts --report /etc/cron.daily )

JeremyBoden · 09-29-2015, 11:13 AM

The ananacron is a red herring - it's a just a useful variant of cron.

My /etc/crontab contains

Code:

# /etc/crontab: system-wide crontab
# Unlike any other crontab you don't have to run the `crontab'
# command to install the new version when you edit this file
# and files in /etc/cron.d. These files also have username fields,
# that none of the other crontabs do.

SHELL=/bin/sh
PATH=/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin

# m h dom mon dow user	command
17 *	* * *	root    cd / && run-parts --report /etc/cron.hourly
25 6	* * *	root	test -x /usr/sbin/anacron || ( cd / && run-parts --report /etc/cron.daily )
47 6	* * 7	root	test -x /usr/sbin/anacron || ( cd / && run-parts --report /etc/cron.weekly )
52 6	1 * *	root	test -x /usr/sbin/anacron || ( cd / && run-parts --report /etc/cron.monthly )
#

Note:- at boot-up cron runs anacron and cron.hourly, /etc/cron.daily, /etc/cron.weekly and /etc/cron.monthly at the appropriate times
and any other crontab commands you have placed in /etc/crontab.

So I suggest you add your database backup script to /etc/crontab

Read man cron as there are some particular restrictions about script authorities (probably already met).

pieterhouwen · 09-30-2015, 02:46 AM

Quote:

Originally Posted by JeremyBoden

The ananacron is a red herring - it's a just a useful variant of cron.

My /etc/crontab contains

Code:

# /etc/crontab: system-wide crontab
# Unlike any other crontab you don't have to run the `crontab'
# command to install the new version when you edit this file
# and files in /etc/cron.d. These files also have username fields,
# that none of the other crontabs do.

SHELL=/bin/sh
PATH=/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin

# m h dom mon dow user	command
17 *	* * *	root    cd / && run-parts --report /etc/cron.hourly
25 6	* * *	root	test -x /usr/sbin/anacron || ( cd / && run-parts --report /etc/cron.daily )
47 6	* * 7	root	test -x /usr/sbin/anacron || ( cd / && run-parts --report /etc/cron.weekly )
52 6	1 * *	root	test -x /usr/sbin/anacron || ( cd / && run-parts --report /etc/cron.monthly )
#

Note:- at boot-up cron runs anacron and cron.hourly, /etc/cron.daily, /etc/cron.weekly and /etc/cron.monthly at the appropriate times
and any other crontab commands you have placed in /etc/crontab.

So I suggest you add your database backup script to /etc/crontab

Read man cron as there are some particular restrictions about script authorities (probably already met).

So if I understand correctly you're suggesting something like this?
Instead of

01 0 * * * root test -x /usr/sbin/anacron || ( cd / && run-parts --report /etc/cron.daily

Make it

01 0 * * * root ./etc/cron.daily/apport
01 0 * * * root ./etc/cron.daily/apt
01 0 * * * root ./etc/cron.daily/aptitude
01 0 * * * root ./etc/cron.daily/bsdmainutils
01 0 * * * root ./etc/cron.daily/[redacted]-backup-configuration

And so on, there are like 20 scripts in cron.daily alone, it can't be that I have to put every script /etc/crontab manually.

Isn't there an option to have anacron start the jobs earlier?

jpollard · 09-30-2015, 07:46 AM

Quote:

Originally Posted by pieterhouwen

Actually I didn't. It was already setup when I started working at the company. How can root have no crontab while in /etc/crontab every cron job is run by root?

Cron jobs are run by the service "crond". Anyone permitted may have a crontab file run.

pieterhouwen · 09-30-2015, 07:59 AM

Quote:

Originally Posted by jpollard

Cron jobs are run by the service "crond". Anyone permitted may have a crontab file run.

Then what's the difference between Anacron and crond?

jpollard · 09-30-2015, 08:37 AM

A small assumption "Unlike cron, it does not assume that the system is running continuously."

This is only significant for jobs that would get skipped because something like a laptop might be down/suspended for a day or two.

When the system resumes the jobs that would have been skipped get run. Useful for things like log file rotation and general maintenance functions.

The standard cron service doesn't run those - under the assumption that the time they run is important (like always running accounting jobs at midnight so that the daily log/reports gets handled properly).

The choice between them is mostly up to the administrator to decide which is more important.

michaelk · 09-30-2015, 08:39 AM

cron will run a particular task at the desired time/date but can not recover if the computer is off. Most maintenance tasks are typically run at odd hours of the night so if the computer is not on 24/7 they will never run.

anacron is designed to run daily, weekly or monthly jobs not by hours but by period and delay time. So maintenance tasks will run regardless if the computer is run 24/7 or not.

anacron will run at startup or at the scheduled hour defined by
/etc/cron.d/anacron

Might be easier to change from 7:30 to whenever you desire.