Linux system randomly locks up and slows to a crawl
Linux - SoftwareThis forum is for Software issues.
Having a problem installing a new program? Want to know which application is best for the job? Post your question in this forum.
Notices
Welcome to LinuxQuestions.org, a friendly and active Linux Community.
You are currently viewing LQ as a guest. By joining our community you will have the ability to post topics, receive our newsletter, use the advanced search, subscribe to threads and access many other special features. Registration is quick, simple and absolutely free. Join our community today!
Note that registered members see fewer ads, and ContentLink is completely disabled once you log in.
If you have any problems with the registration process or your account login, please contact us. If you need to reset your password, click here.
Having a problem logging in? Please visit this page to clear all LQ-related cookies.
Get a virtual cloud desktop with the Linux distro that you want in less than five minutes with Shells! With over 10 pre-installed distros to choose from, the worry-free installation life is here! Whether you are a digital nomad or just looking for flexibility, Shells can put your Linux machine on the device that you want to use.
Exclusive for LQ members, get up to 45% off per month. Click here for more info.
Distribution: Mint 20.1 on workstation, Debian 11 on servers
Posts: 1,336
Rep:
Linux system randomly locks up and slows to a crawl
I noticed that sometimes my Linux server will randomly start to lag really badly, to the point where even a http request takes forever.
It is an Intel Core2Quad with 8GB of ram running FC9.
This is my "Everything" server so it does file, email, DNS, web (for local stuff only), VMs and so on.
There are about 5-6 VMs running on it at any given time. I manage it through a VNC session and have some SSH consoles within that session. This way if I reboot my PC I don't lose all my SSH consoles. If I need to SSH to any server I do it from there. I treat it kinda like a terminal server to some extent.
When this slowdown happens, top is not really useful because I also do F@H so that will always be to the top, but it's low priority. The VMs are also always near the top. This does not change whether it's slow or not, so when it's slow, I have nothing to go by on how to troubleshoot. The load does seem to skyrocket though. Right now it's doing the slowdown thing and it's at 8.09. Normally it's at around 3 which imo is good as it it's under 4. I have 4 cores so anything more than 4 means it's queuing. At least that's how I understand it.
The fact that half a gig of swap has been used indicates that at some point the system has run out of memory; that might be noticeable if it was a VM that had been swapped out. You would get a high load average during the time it was thrashing the drive. The iostat utility can be useful for monitoring the drive access.
I also observe that one of the VMs has used a surprisingly high amount of CPU time (ie, it is running about 75% of the time). Being a quad processor, that isn't going to cause much variation in the responsiveness, but it seems excessive.
You don't say how long the slowdown happens for, or what operating systems are running in the VMs.
Last edited by neonsignal; 06-06-2011 at 08:28 AM.
Normally swap would be at zero. It isn't critical on a desktop (since the application load can vary greatly), but things should be more predictable on a server.
How much base memory have the VMs each been allocated?
What is running in the VM (process 3033 in the above example) that would be using so much CPU (none of the applications you mentioned were CPU intensive)? You don't by any chance have a Win95/98/ME running in one of the VMs?
Last edited by neonsignal; 06-06-2011 at 06:59 PM.
Distribution: Mint 20.1 on workstation, Debian 11 on servers
Posts: 1,336
Original Poster
Rep:
Well two of the VMs are environments for a game server one is dev the other is test, and the game is somewhat intensive. So think it may just be vm activity that causes it?
So think it may just be vm activity that causes it?
No, but if several of the VMs are being used intensively, and since you are using the whole memory, then you can expect thrashing of the swap file. One step would be to limit the base memory allocated to each of the VMs (if you haven't already) so that the total is less than the 8 gig; that way if a particular VM is using excessive memory (or has a memory leak in an application), it won't impact the other VMs.
Last edited by neonsignal; 06-06-2011 at 07:45 PM.
Distribution: Mint 20.1 on workstation, Debian 11 on servers
Posts: 1,336
Original Poster
Rep:
I'm starting to wonder if it's disk I/O related. I noticed 3 mdadm errors where it failed a drive then it just rebuilt. This happened maybe 10 days ago.
Also, here's an iostat, it's doing it again, and I have a backup job running.
I'm guessing from the figures that you are running a RAID5 with the 5 drives. If you were actually getting disk errors, you would expect the read/write rates to drop off. Anyway, you can use smartdisk to check the status of the drives. Indeed, the current iostat shows the disks running heavily. The %system usage is also high.
Last edited by neonsignal; 06-06-2011 at 08:57 PM.
LinuxQuestions.org is looking for people interested in writing
Editorials, Articles, Reviews, and more. If you'd like to contribute
content, let us know.