Hi All,

For a small organization (30 users) I run a FC6 / Sendmail / Dovecot mail server on DELL Optiplex box with 512 MB RAM. It was installed a few weeks ago and activated last week and ran fine after that. Today morning Sendmail started refusing connections because of high system load. From SysGuard and top it turns out that Sendmail was using almost all CPU time. At the same time disk activity was very high and, significantly, swap memory use increased from almost nothing to 100% in about a minute. The system practically freezes, IMAP and SMTP connections are refused. I found also the named service was stopped, but I'm not sure whether that is a cause or an effect.

After a while (could be an hour or more) the system is 'given back', including the hogged swap memory. We have been through several of these cycles today. Needless to say the users are not happy.....

Sendmail definitely is 'doing something', but what?? The progressive swap increase seems to point at a memory leak but I could not find anything on the Internet on this issue. Has anybody seen this problem? Or have an idea what we could be the cause or what we can do about it? Any hint will be appreciated!

Thanks,
Erik

I'm not sure if I'm experience enough to really give any good suggestions, but I guess any couldn't hurt. I would first make sure the server is the machine causing the services to use so much resources. (block the ports or just d/c the cable or wireless connection)

I'm not sure if SMTP or IMAP have their own log files but u might want to check, or maybe run them in verbose mode outputting to a file or terminal. (Not sure if that's possible but one would think...) Maybe it goes into messages.

Also you may want to check the niceness levels and maybe lower them for those two if they are set to high. If you lower the priority of them I wouldn't imagine the would pull so much.

Again, not sure if anything I suggested will help but these are just my thoughts.

nomb

***PS - If anything maybe having a reply will get others to look at it.

Hi Nomb,

Thanks for the reply. Let me answer your questions:
--- yes, whatever is happening is happening on the server itself. During or before an 'episode' (i.e. the near-freeze I described) there is none or hardly any Ethernet traffic (it's a small office with little email traffic).
--- yes, I checked the logs for Sendmail/Dovecot (/var/log/maillog) and for the OS (/var/log/messages). There are many entries there that are related to the problem. The Maillog shows that Sendmail stops accepting connections at a certain system load level, which it is by default configured to do. In the messages log there are many warnings about the issue (e.g. invocation of the oom-killer by all kinds of programs) and also plain out-of-memory errors that stop e.g. procmail it its tracks. But as far as I can see, they are all due to the depletion of free swap memory.
--- I haven't checked the nice level of the programs at hand. That's a good tip, thanks, and I'll defeinitely look at that tomorrow. On the other hand, this is a mail server and I wouldn't want Sendmail to be too low a priority, I think.

Thanks,
Erik

I would agree in that you might not want to put the programs at hand too low considering this is the main point of the box, however we need to get down and dirty and figure out what is causing your spikes. Is this mail server internal only? Or can it get out? I know there shouldn't be too much traffic considering its a small office but I might run wireshark(well new ethereal don't remember the name) to capture one of the spikes just to make sure. Also have you watched top during the spike to see if there are any other proccesses at all the rise with it?

nomb

I would agree in that you might not want to put the programs at hand too low considering this is the main point of the box, however we need to get down and dirty and figure out what is causing your spikes. Is this mail server internal only? Or can it get out? I know there shouldn't be too much traffic considering its a small office but I might run wireshark(well new ethereal don't remember the name) to capture one of the spikes just to make sure. Also have you watched top during the spike to see if there are any other proccesses at all the rise with it? Also you said they also stop? which one?

nomb

Hi again,
The box is both an internal and external mail server. When it goes 'down' it takes about a minute to do so, from fully working to almost frozen. That's when the swap mem use increases to 100%. I haven't run Wireshark because of the time factor and the lack of apparent traffic in SysGuard, but I'll try to do so tomorrow, including trying all this with the cable unplugged.
As far as I could see in SG, it is indeed just Sendmail that is taking the CPU capacity, up to 15%, which in SG is really a lot and amounts to taking over the box. When I managed to stop the Sendmail service (need patience for that) it shows that also services for procmail, dovecot and named have stopped / are crashed. I wondered what named has to do with this all. That's another one I'll check on tomorrow.
Thanks,
Erik

Some updates:
--- As mentioned, I ran Wireshark during the start of an event. No odd packets came by. I also tried it with the cable disconnected. Events still happened.
--- Ran Fedora under run level 3. After a minute or two, top showed something happening: procmail (not sendmail as reported by SystemGuard!) took almost all swap memmory, run as a specific non-root user. CPU load stayed low. The box still processed mail properly. Messages does not show any ugly OOM situations anymore.

So? It looks like procmail is intermittently 'doing something'. With 512MB there is not enough memory for all that, the increasing swap use points at that. It could be that running KDE (more processes and mem use) caused the system to stop procmail and freeze the box. Perhaps now without KDE there is just enough memory.

Erik

Hmmm, what about increasing the size of your swap partition?

So? It looks like procmail is intermittently 'doing something'.
Check which central / per-user procmailrc's get used, then edit and set logfile and verbose logging, then inspect logging for clues? At the same time (if you think that won't be enough) you could isolate that users process three (ps -fU $UID) to see what procmail actually *runs* as it's more like an initiator for external tools like SA, formail, grep and whatnot. If *that* doesn't provide enough leads to work on you can use lsof to get more fine-grained details of what that $UID's procmail process tree accesses or even strace them PIDs...

There ya go I knew someone more experienced would give us a hand eventually.

nomb

Hi unSpawn and Nomb,

Thanks for these tips. I'll be following up on them over the week and will report back later!

Nomb, the swap mem is set at the same of the physical memory, as is the recommendation. The disk activity could of course just be swapping pages in and out (although that doesn't explain the increase in use of swap, for that I'll need unSpawns tips). Increasing the swap memory would then not help, but increasing the physical memory would. I'm trying to get the latter increased to 1GB this week.

Best,

Erik

When you upgrade your ram remember to resize your swap partition to match. And actually I read 1.5x the amount but I'm sure 1x would work fine too.

nomb

Yeah, will do so. Thanks!
Erik