Postfix / cron / monit problem on CentOS
 

Hi all,

I've run into a strange problem on one of the servers I maintain. It's running CentOS 5, fully up to date. The server has Monit installed via RPMforge, and runs Postfix as its MTA. Every day at about 4:20am, when cron is running, Monit loses its connection to Postfix. As far as I can tell, nothing is actually going wrong with Postfix and it's always restarted just fine. I run a couple of Debian servers with very similar setups, and the problem does not occur on them.

Perhaps related, other daemons on the CentOS box often lose their pid files at the time when cron runs, which results in Monit thinking they're not running. One service in particular (zarafa-dagent) is so bad for this that I've had to completely disable monitoring of it in Monit. Again, this problem doesn't occur on the Debian boxes I maintain.

The problem isn't hugely urgent as everything still runs fine, but it does mean that pretty much every day I get false positive mail alerts from Monit, which is (a) annoying and (b) makes me more likely to miss a real alert.

Any light anyone can shed on this would be greatly appreciated! Thanks in advance.

Have a look in /var/log/cron at what cronjobs (logrotate?) are run by which users at what time?

Last night the connections failed at 4:10:30 and 4:21:58. Here's the cron log from 4:00 to 4:30...
There is nothing in cron.hourly and according to crontab -l, both root and nobody have empty user crontabs. The contents of cron.daily are:
Of these, chkrootkit and rkhunter are custom scripts based on their Debian equivalents, clamdscan-samba-shares is a custom script, mysql-backups is a shim that calls automysqlbackup and train-spamassassin is based on this sa-learn script, modified to work with amavisd.

I guess I'll have to figure this out the hard way, disabling each custom script one at a time and seeing if they affect the situation. Since it's an intermittent problem this is going to take a long time though. I was hoping someone might have seen a similar problem in the past and have some specific comment... (still hoping, any further suggestions would still be appreciated!)

Is there any other system or daemon log that shows activity (or better: errors) between 4:00 and 4:30? How about running monit in verbose ("-v") mode?

There are a couple of interesting log snippets:
I've edited out the actual file names for privacy reasons, but they're all user data rather than system files. The "Can't open file or directory" ones are files over ClamAV's size limit (>2GB). Unfortunately it doesn't seem be possible to get ClamAV to quietly ignore oversize files. The ClamAV database is getting rebuilt around this time too.

Here are the relevant bits of the maillog (apologies for length):
It's looking as though ClamAV might be overloading the system, at which point postfix chokes. In this instance Amavis appears to be processing a spam email at the same time as the clamdscan, perhaps the two have to happen at the same time for the system to reach critical load. That could explain the intermittent nature of the problem - this server gets enough mail that it's likely to be scanning a message at the time of the monit connectivity check, but it's not guaranteed.

I've enabled monit's -v option in the initscript, increased the postfix connect timeout in monit, and increased clamdscan's niceness. Will see what happens over the next few days and report back.

Thanks for the feedback so far.

Just a quick update - I've not had any problems from monit since increasing the nice level of the cron jobs and increasing monit's timeout for the services in question. Evidently the trouble was caused by excessive system load. Thanks for the help!