FC5 slows down after period of time
 

We had a disk crash last week and since we re-installed with Fedora Core 5, the system seems to work fine while it is busy, but when we come in, in the morning, it is very, very slow.

I had to turn the power off yesterday morning. After the system rebooted, the system seemed to be fine. I monitored it all day. When we came in this morning, same problem. It takes minutes to execute commands.

A friend suggested it might be a syslog problem. I restarted syslog, that didn't help. It took over 5 minutes for "service syslog restart" to complete. This box is only used for a Samba share. There are no direct users logged in on the system.

When I run TOP, it takes a long time to redisplay. It will alternate between 0's for each CPU fields or 100% for wait. The top app in the list of apps is init. After I reboot, the system seems to routinely stay at 95% idle and the top app is Xorg.

The system is an IBM ThinkCentre Model 8194-A4U. It has a 120GB drive. It is a Pentium 4, 2.4Ghz with 768MB RAM.

Amazingly, I set up a different ThinkCentre last night that is already doing the same thing. It is a Model 8198-A2U with a 160GB drive. I believe it has a Pentium 4, 3.0Ghz with 256MB RAM. I put FC6 on and then did a yum update. The update was still running when I left. This morning yum had prompted for Y/N question and after I answered it, it is proceeding with the update, but it is very, very slow. I can't even get TOP to load.

What I'm asking is if anyone has seen this in the past and what can I check to see what the problem is. I am guessing it might have to do with something in the powersave area where maybe the CPU or the disk is shutdown due to inactivity but now it isn't re-awakening.

I've just noticed something else, on the primary system, before I reboot, I noticed that it only had about 9MB of memory left. After I reboot, it has about 331MB free. I'm not sure what would be taking up the memory.

Thanks

Do you see any errors in /var/log/messages and/or dmesg? The high I/O wait makes me suspect one or more of your drives is experiencing a problem.

I get these in /var/log/messages frequently:

Jun 12 12:51:49 acsfs smbd[2264]: [2007/06/12 12:51:49, 0] smbd/service.c:make_connection_snum(592)
Jun 12 12:51:49 acsfs smbd[2264]: Can't become connected user!
Jun 12 12:54:18 acsfs smbd[2269]: [2007/06/12 12:54:18, 0] smbd/service.c:make_connection_snum(592)

I don't see anything that jumps out on dmesg.

I have been watching TOP. Shortly after I booted, I had approx 331MB of memory free. I did a ps aux to a file to record the processes. It quickly went down to around 188MB has the first 4 shares were loaded by users. It has slowly went down now to 45MB of free memory as reported by TOP. When I do a ps aux to a separate file and compare the two, they are almost identical except the latest one has 7 shares and to ssh connections and the first one only had 4 shares and 1 ssh connection. The sizes for the shares are slightly larger. In example the largest of the 4 smbd's from the first ps aux was 12220 and now the largest of the smbd's from the second ps aux is 13672. These are the VSZ column numbers, not the RSS size.

Top shows uptime at 5:11 and 4 users.

This seems like it is a huge memory leak. I guess I don't understand why ps aux doesn't show some program's size increasing dramatically. Is there a way to better track memory size of apps to see where it is all going? At this pace, I'm not sure I'm going to make it to 5:00 this afternoon.

I also looked through /var/log/cron and I can see where cron.hourly ran without any problems through 4:00 am. But when cron.daily started a minute later, it doesn't seem to have finished. That was one of the things I noticed this morning was that the time in TOP showed around 4:00 am. I thought it was just a problem with the actual date of the system. But I wonder if there is something in the cron.daily that is killing the system.

I had left TOP up and running from the night before. It is refreshing every 3 seconds. When I checked it this morning, it would show an updated time of every 3 or 4 seconds, but the refreshes were taking more than 30 seconds. It was like they were queued up.

Thanks

There's no indication of a memory leak. Instead of using top, use free:

The value you're interested in is highlighted. While top would show "66032"free in this example, it doesn't take into consideration the amount of memory being used for cache buffers which prevents disk I/O, and can be reclaimed if memory is actually needed.

[root@acsfs ~]# free
total used free shared buffers cached
Mem: 767208 758140 9068 0 73820 480320
-/+ buffers/cache: 204000 563208
Swap: 1540088 0 1540088

I had just recently logged out of the console, then relogged in and then started a terminal session and started TOP and it fell from around 38MB free to about 9MB free. So, I'm seeing about the same thing as free is reporting.

Is there a way to flush the cache?

Thanks

Sorry, didn't format the way it should have

Two thirds of your RAM is free, and you are not using any swap. There is no memory bottleneck on your system. If you were to "flush the cache", your system would grind to a halt (you think it's bad now), as every file I/O would require a real disk I/O.

I suggest you run "smartctl -A" on each of your drives. If they are not reporting errors, they may be experiencing high recoverable counts.

That looks good. How about:

cat /proc/interrupts

Also, well after working hours, I see TOP is active as it has been all day. The CPU us fld is bouncing around 25-40% and the idle of course rarely stays up around 90%+ like it had been all day. At peak, there were 9 smbd processes, now there is two. Most users turn their PC's off at night. Again, their are no direct logins.

The apps that are staying at the very top is Xorg which now has CPU time of over 135:00:00 and floaters, which I think is a screen saver.

It looks busier now that it has most of the day.

We didn't have a console on the system until yesterday but it may be turned off. Any chance that could be a problem?

Thanks

Generally servers don't run X, and they certainly don't run screensavers - they just burn CPU for no good reason. However, while that may provide crappy response to your users, it won't cause your problem.

I'm not seeing any reason for the bad response time. You wouldn't happen to have an email server running on this system with an open relay?

Not unless it comes that way from the install. I basically installed FC5, copied over by smb.conf file, turned on Samba and let it rip. It is a very vanilla install.

Thanks,

Assuming you have all the maintenance applied, I see no reason for the performance problem you're having. My last suggestion would be to check for a compromised machine (very unlikely):

yum -y install chkrootkit

Then run:

chkrootkit -q -n

I don't have any ttys installed on this system other than the regular serial port(s) on the back of the PC.

Thanks,

That response looks normal (the tty1-6 are the console sessions on ctrl-alt-f1 though f6). There doesn't appear to be anything wrong. The I/O activity of your samba users may simply be keeping the hard drive occupied.

For the most part, the problem isn't during the day, it seems to work fine during the day and as I watch it tonight it is fine. There is no one during the night. But when we come in, in the morning, around 6:00, that is when we find it where it seems to be all jammed up to where it almost seems unresponsive.

Thanks,

I'm back, another problem. I made it a little over two days, but it is a problem again.

If I do a ps -ef, it reponds instantly. If I do a TOP, it takes a very long time to diplay the window.

This started at 9:21. I asked the main user at 9:45 how it was going and she said it just seemed to slow down again. I checked my TOP window and it still had 9:21 as the time. I checked the date command and it also said that the system date was 9:22 but the current wall clock time was after 9:45. I had checked the system around 8:30-8:45 and both times were in sync.

While watching TOP, it was doing the same thing, about every 20-30 seconds, it will redisplay even though the refresh rate is set to 3 second. But the system time displayed in TOP only goes up about 3-4 seconds on each refresh.

I am tarring the data up to save it off, it scrolled the first 50 or so files then it paused for a long time and is repeating. The total size of the data is only around 471MB, so it should only take a minute or two to tar it up, but it has already taken over 5 minutes and still isn't finished.

Here is a copy of my smartctl. At first I thought it was a problem with the drive, but not I'm not so sure. Below this is a copy of the ps -ef and top output.

Please help, the Microsofties here want to just switch over to Win2003 and be done with it. I don't want to give up.

Thanks,

Unfortunately, it sounds like the system is "broken" - non-compliant in some way that is causing problems for the kernel.

You can try booting with the kernel options noacpi, noapic, mem=1000M to see if it helps. Edit your /etc/grub.conf and add the options to the kernel line. For example:

kernel /vmlinuz-2.6.xxx ro root=/dev/hda quiet noacpi noapic mem=1000M

Then reboot. You can also try updating your BIOS, and swapping the slots PCI cards are in (clear the BIOS PCI data if you do this).

Just as a followup, I had a brand new 80GB drive that was two years old that I have never used. I put that into the original sys replacing the used 120GB drive and it has been up for 6+ days now. The other sys that has the 160GB drive still needs to be rebooted once or twice a night to make it through the day.

So, the only thing I can think of is that the 120 & 160GB drives have a small incompatibility with the hardware and RH FC5/6. Both system exhibited the same symptoms, they would run for up to 2 days without a problem then "slow" down. I tried both FC5 & 6, and openSuse 10.2 and got the same symptoms.

All three drives are the cheap Maxtor drives from BBuy.

I guess I forgot that part also, on the 80GB drive is FC2. That was where we started from when the original drive died.

Thanks to all who helped, especially macemoneta.