What to do about missed at jobs?

I use the at command to schedule many jobs that are run only once. Recently I ran into an interesting problem when I needed to reboot the machine.

During the reboot fsck ran a forced check on one partition. The time consumed by that fs check ran past the scheduled time for an at job. The at job never ran and there was no email about missing the job.

As an additional test, I scheduled an at job, rebooted the computer, paused the boot loader until after the scheduled at job, and then continued booting the box. Same results. The missed job was deleted from the queue without an email.

This behavior is an exception to how I normally run my box, but the anamoly bothers me. How do I provide remedy?

Is there a way to have the at daemon send an email for missed jobs?

Is there a way to force the at daemon not to dismiss at jobs and run anyway?

I appreciate any discussion about handling missed at jobs.

Thanks.

I didn't think any at jobs survived a reboot.

at uses the /var/spool/at folder to store information about pending jobs.

What you could do is create a script to run before the at daemon gets shutdown to store the output from atq

This will give you the at job number, the date and time that it was due to run the queue that it was created under and the owner of the job.

The file a0000601fdfad relates to at queue a job 6

You could use this information to check if any jobs should have run while the system was down, and re-run them.

That's an interesting question. The at daemon sends an e-mail if the standard output or the standard error of the job is not null. You can force the mail to be sent using the -m option, even if the condition above is not matched. Anyway it is not clear if this works for missed jobs, when the at daemon removes the "old" and "missed" input files from the spool directory.

I cannot test the -m option and its behaviour, right now (since I cannot reboot my box). By the way, the only and definitive answer resides in the at source code. At this point, let me pronounce the classic motto: Use the Source, Luke!

Clarification: Slackware uses /var/spool/atjobs.

That might be an idea --- to mirror the atjobs directory and compare the directories. The queue assignment and job number probably won't help because when the system reboots after the assigned time, the at daemon simply deletes the job with no fanfare. By the way, the file names in /var/spool/atjobs contain the scheduled date and time.

Okay, I just tried that. No mail for the missed job.

Edit:
Doh! Let's back up. The test box I used is a spare machine --- I did not want to continually reboot my primary box to test. I wasn't receiving any mail because on the test box sendmail was not configured to start. Hmph. Next, after enabling sendmail, I discovered I had a stale alias for forwarding email. I deleted that alias, but then took a while before I realized I had to also run the newaliases command to update the change. Double hmph!

After demonstrating my fallibility to the flies on the wall, finally I got some mail messages.

Repeating the test several times, I scheduled an at job using the -m option and then rebooted. Each time I paused the boot loader until a minute or two after the job was scheduled to run. I received mail. Apparently missed at jobs are run. I then was curious how long after a reboot the at daemon would run a missed job. I scheduled a job, rebooted, and then waited 10 minutes before continuing the boot process. I received an email.

Next I repeated the scheduling, reboot, and wait process several times, except I did not use the -m option. Same results.

Conclusion: the at daemon will run jobs after the scheduled time. How long after missing the time I don't know. I'll have to run more tests with longer wait periods.

Edit: I scheduled a job last night and booted the computer about 12 hours later. The missed job still ran. Thus, with respect to my original post, I don't know why the missed job failed to run or why I failed to receive an email.

Even more interesting. I'm trying to interpret the source , based on the behaviour you describe, but since I'm not a C-programmer I encounter some difficulties. Now I'm going to schedule two jobs, both without standard output (remember that standard output not redirected to a file triggers the mail notification... the same as crontab) and I will use -m option only for one of them to see the difference.

Tomorrow when I reboot the machine, I will report the results of my test. This is an Opensuse box. Cheers!

Hi Woodsman,

based on my test, I can confirm what you already observed:

1) the at daemon executes "expired" jobs upon starting. Indeed, this was my doubt when I was looking at the source code. I started from the wrong assumption that "expired" jobs were simply removed from the spool directory, but I couldn't find the code responsible of this behaviour. Actually, atd executes all the jobs found in the spool directory for which scheduled time is less than or equal to the current time.

2) the -m option works as expected. I received mail notification only for the job submitted with -m, not for the other (both of them did not send a bit to standard output or standard error. In that case a mail notification would have been sent for both)

An aside note: the reboot was not strictly necessary to perform these tests. Starting and stopping the at daemon would suffice.

Best,
Alex

Seems to be my week for running into quirky coincidences.

As mentioned, the at daemon will run "missed" jobs. Overall, seems like a decent idea. Yet interestingly, I booted one of my systems when an at job was due to run. I did not have the system running when two previously scheduled at jobs should have run. At power up, the end result was the at daemon tried running all three jobs in scheduled order. The jobs were all similar --- TV recordings. Each at job runs a script I wrote. Basically none of them ran correctly because the script tries to kill any locks on the TV card device node. I'll have to rethink in my script how I kill mencoder device node locks.

You could rely on crontab for tv recording at a specified time and date. If you specify both the calendar day and the month, you have 1 year of time to remove the crontab entry before it tries to run again! Otherwise, you could try a check of the current time at the beginning of your script or at job (for example to see if it matches the scheduled time of the tv show) and act accordingly.

I'll think of something.

Update: I decided to write a shell script looking for stale at jobs. Specifically, because I am using the at scheduler to record TV programs and movies, my shell script is focused on those types of tasks. My script can be expanded to delete any stale job if a person desires.

To use my script I had to modify the rc.d startup scripts and not start the at daemon until after I ran my check for stale jobs. I either could have inserted a call to my script in rc.M or delay starting the at scheduler until I ran the same test in rc.local. I opted for the latter and commented out starting the at scheduler in rc.M.

If interested you can grab a copy of the script at my web site:

Building an HTPC - Part XIX

Thanks for everybody's help.