LinuxQuestions.org
Help answer threads with 0 replies.
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - Newbie
User Name
Password
Linux - Newbie This Linux forum is for members that are new to Linux.
Just starting out and have a question? If it is not in the man pages or the how-to's this is the place!

Notices


Reply
  Search this Thread
Old 07-03-2009, 05:56 AM   #1
linuxentry
LQ Newbie
 
Registered: Jul 2009
Posts: 2

Rep: Reputation: 0
cron error


Hi,

This is a bit long post sorry about that. Being totally new to Linux getting some difficulty in handling this particular problem. Any help on this would be great!

We have the MRTG application for router monitoring running in on a Linux Server, monitoring and polling around 300 routers. The server has 2 CPU with 6 GB RAM.

For this we have almost 200+ cronjobs scheduled to start every 5th minute. Basically to poll every router on every 5th minute.

My question is: Is there a limitation on how many cronjobs we can run on a server (both as a good practice and as a limitation of cron queue). I dont see any configuration currently on the server.

The problem on the server is we can see that the server goes down on every 2-3 days and we do not have any panic/or other messages in the message file which can relate to some kind of problem.

The only message we see is:

crond[30090]: System error


which keeps appearing on every minute or so.

This has prompted to me to suspect if the server is going down because of cron queue growing ?

Also when I see top command I can see that there are around 800-1400 zombie processes and in the ps output lots of crond [defunct] processes.

Keep
linuxentry
 
Old 07-03-2009, 07:26 AM   #2
unSpawn
Moderator
 
Registered: May 2001
Posts: 29,331
Blog Entries: 55

Rep: Reputation: 3529Reputation: 3529Reputation: 3529Reputation: 3529Reputation: 3529Reputation: 3529Reputation: 3529Reputation: 3529Reputation: 3529Reputation: 3529Reputation: 3529
Quote:
Originally Posted by linuxentry View Post
This is a bit long post sorry about that.
Don't be. The more factual information the better: it will help us help you.


Quote:
Originally Posted by linuxentry View Post
I can see that there are around 800-1400 zombie processes and in the ps output lots of crond [defunct] processes.
While crond may cause the system to run out of resources, this may indicate there could be something wrong with the jobs themselves. I would suggest to look at that first.
 
Old 07-03-2009, 08:07 AM   #3
centosboy
Senior Member
 
Registered: May 2009
Location: london
Distribution: centos5
Posts: 1,137

Rep: Reputation: 116Reputation: 116
Quote:
Originally Posted by linuxentry View Post
Hi,

This is a bit long post sorry about that. Being totally new to Linux getting some difficulty in handling this particular problem. Any help on this would be great!

We have the MRTG application for router monitoring running in on a Linux Server, monitoring and polling around 300 routers. The server has 2 CPU with 6 GB RAM.

For this we have almost 200+ cronjobs scheduled to start every 5th minute. Basically to poll every router on every 5th minute.

My question is: Is there a limitation on how many cronjobs we can run on a server (both as a good practice and as a limitation of cron queue). I dont see any configuration currently on the server.

The problem on the server is we can see that the server goes down on every 2-3 days and we do not have any panic/or other messages in the message file which can relate to some kind of problem.

The only message we see is:

crond[30090]: System error


which keeps appearing on every minute or so.

This has prompted to me to suspect if the server is going down because of cron queue growing ?

Also when I see top command I can see that there are around 800-1400 zombie processes and in the ps output lots of crond [defunct] processes.

Keep
linuxentry
Hi.

i dont wish to sound unhelpful, but how do you manage to have that many mrtg commands in the crontab?
1 or 2 should surely be enough?
You certainly wouldnt need one for each device..
All the devices should be listed in the mrtg.conf file

then a command like this

Code:
/usr/bin/mrtg /etc/mrtg/mrtg.cfg --lock-file /var/lock/mrtg/mrtg_l --confcache-file /var/lib/mrtg/mrtg.ok
would poll the mrtg config file. I would seriously look at how mrtg is set up on your server

Anyway, i would try cacti over mrtg.
It is better (better graphs, output and information)

also you need only one line of command in your crontab to monitor all your devices.
 
Old 07-10-2009, 01:14 AM   #4
linuxentry
LQ Newbie
 
Registered: Jul 2009
Posts: 2

Original Poster
Rep: Reputation: 0
hi Thanks for the reply.

The teams here have gone for multiple cron entries as sequencing the routers in a single config file may take more thant 5 minutes for one complete execution. And every router has to be polled for every 5 minutes. And there are almost 250+ routers, geographically spread across.

New Update:
The server stopped resonding after a week of operation and on the console I can see the "Out of memory killed process PID " messages also.

any help on this is greatly appreciated.

Thanks
linuxentry
 
Old 07-10-2009, 01:32 AM   #5
chrism01
LQ Guru
 
Registered: Aug 2004
Location: Sydney
Distribution: Centos 6.8, Centos 5.10
Posts: 17,240

Rep: Reputation: 2324Reputation: 2324Reputation: 2324Reputation: 2324Reputation: 2324Reputation: 2324Reputation: 2324Reputation: 2324Reputation: 2324Reputation: 2324Reputation: 2324
As unSpawn said, post #2, looks like you've got problems in your perl programs. That many defunct/zombied processes is bad... you need to find out why they do that. It will kill your server, as you've seen.
If you want to poll that many objects that oftewn, don't spawn hundreds of processes every 5 mins from cron, break them in eg 50 router blocks, and look at running each set inside a perl daemon that doesn't have to create a new process for each individual router poll.
Use a timer at the bottom of each daemon's loop and wait 5mins - (time it took to do set-of-routers).
 
Old 07-10-2009, 06:30 AM   #6
jeromeNP7
Member
 
Registered: Jun 2009
Posts: 101

Rep: Reputation: 19
Instead of cron jobs you should create a daemon application that does all the polling by itself in regular intervalls. The usual problem with often repeated cron jobs is that they are only as good a the script or application that they launch. Launching a buggy tool will simply multiply the number of problems with the number of cron jobs.

Linux

Last edited by jeromeNP7; 09-04-2009 at 09:54 PM.
 
  


Reply


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off



Similar Threads
Thread Thread Starter Forum Replies Last Post
Cron job error gocool_fedora Linux - Software 2 12-22-2008 12:50 PM
adding a perl script to cron.daily / cron.d to setup a cron job CrontabNewBIE Linux - Software 6 01-14-2008 09:16 AM
cron not working from crontab nor form /etc/cron/cron.d. What did SuSE change? JZL240I-U SUSE / openSUSE 11 01-04-2007 02:57 AM
Can any one plz explain why/what for cron.d, cron.daily, cron.weekly etc are there. mavinashbabu Linux - Newbie 4 09-21-2006 02:50 PM
No More Cron Mail, Cron Error? Xhost Linux - General 3 07-26-2004 05:28 PM


All times are GMT -5. The time now is 08:02 PM.

Main Menu
Advertisement
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
Facebook: linuxquestions Google+: linuxquestions
Open Source Consulting | Domain Registration