I've got a problem that is really driving me nutz. My RedHat 7.2 machine has around 14,000 users. This is the entire population of the local University. We migrated from a 6.1 machine that was under a few too many errata notices to RH 7.2. The first machine (a dell 2550, 2 x 1.3 gig PIII with 2 gig of ECC RAM). The server ran smoothly for the first day. After an undetermined frame of time, the server's load average shot through the roof. (100+ load avg) Top showed ipop3d as being the #1 process on the machine. However the interactive shell did not feel affected by this load average. Within minutes all activity had come to a halt. This machine had to be cold booted as the console would not respond.

I'll continue phase 2 of this problem in the next message.

This server was reconstructed under a new hardware configuration. (dell 6350, 4 x 450 mhz PII zeon, 768 Meg RAM). This time it stayed online for almost a week before users reported the inability to pop mail from the server. When I logged in through ssh, random shell commands would seg fault. This happened for just a few moments before a reboot was attempted. It never finished the reboot.

Just before the server froze on this hardware it did manage to log MANY kernel message: (sample below)

Apr 8 08:29:45 mailserv kernel: remove_free_dquot: dquot not on the free
list??
Apr 8 08:29:45 mailserv kernel: Unable to handle kernel NULL pointer
dereference at virtual address 00000004
Apr 8 08:29:45 mailserv kernel: printing eip:
Apr 8 08:29:45 mailserv kernel: c01583a3
Apr 8 08:29:45 mailserv kernel: *pde = 00000000
Apr 8 08:29:45 mailserv kernel: Oops: 0002
Apr 8 08:29:45 mailserv kernel: Kernel 2.4.9-31smp

Any thoughts.

Okay.. since noone is going to answer this. After about 2-3 days of research it appears the kernel rpm (2.4.9-31) must have a VFS bug dealing with quotas. My quota files keep getting corrupted, and kernel VFS errorrs keep popping up in my logs with reference to quotas. If anyone has seen this behavior please let me know.

Many thanks

Didn't know about the VFS error. I did run into a reiserfs + quotas in 2.4.10. Never got around to patching yet so I have left quotas off the systems so far. Maybe I should lookinto it this weekend.

This is a problem I would call redhat about. This system is clearly important (not something you can break and go oh well) and the solution may be as simple as upgrading an rpm or two.