[SOLVED] Huge CPU load due to high %sy time

fawaz25 · 03-19-2019, 02:14 AM

Dear All,

An application is running on my Red Hat Linux 5 server which is running in VSphere environment. My system has around 24 cores CPU. The CPU load has increased tremendously. I see load averages value to be more than 100. In the top command I see only around 2 % user time and 98-99 % system time. IO wait and %SI are zero.

The top and ps command shows processes consuming maximum CPU usage (which is the application process).
When I run vmstat command, I see more than 100 processes in runqueue. Upon checking the processes in run queue using this command:
#ps -eo pcpu,pid,user,args | sort -k 1 -r | head -10, I found it be application server processes (same as top).

I am trying to troubleshoot this issue. Any help from your side will be highly appreciated.

berndbausch · 03-19-2019, 03:02 AM

My guess is that a resource that the application expects to exist doesn't exist. It then desperately tries to access this resource rather than issuing an error message or otherwise handling the situation gracefully. The resource might be a file or an inter-process communication facility.

You could run strace to see what system calls contribute to the high CPU usage. Try strace -c, which creates a system call CPU utilization summary.

You could also use lsof -p <PROCESS_ID> to look at open files and try to draw conclusions.

If the above doesn't help, if the application doesn't log anything or doesn't have a debug mode that provides some information, and if you can't ask the author of that application, I am afraid you are out of luck.

TB0ne · 03-23-2019, 09:24 AM

Quote:

Originally Posted by fawaz25

Dear All,
An application is running on my Red Hat Linux 5 server which is running in VSphere environment. My system has around 24 cores CPU. The CPU load has increased tremendously. I see load averages value to be more than 100. In the top command I see only around 2 % user time and 98-99 % system time. IO wait and %SI are zero.

The top and ps command shows processes consuming maximum CPU usage (which is the application process). When I run vmstat command, I see more than 100 processes in runqueue. Upon checking the processes in run queue using this command:

#ps -eo pcpu,pid,user,args | sort -k 1 -r | head -10, I found it be application server processes (same as top).

I am trying to troubleshoot this issue. Any help from your side will be highly appreciated.

Haven't we been here before?
https://www.linuxquestions.org/quest...nd-4175643496/
https://www.linuxquestions.org/quest...er-4175648055/

Again, as you were told before, you're not telling us much of anything. What is this 'application'?? Version of VSphere? What have you actually done to diagnose the issue? Checked the logs? Run anything in debug mode? Anything??

You were also told previously that RHEL 5.x is VERY old, outdated, and totally unsupported. You are missing MANY YEARS of updates/patches/fixes, so if you're running on newer hardware, it's not surprising you're having trouble. And again, both RHEL and VSphere are commercial-PAY FOR products. If you're using them you need to pay, and contact/use the support you're entitled to.

fawaz25 · 03-26-2019, 02:49 AM

Quote:

Originally Posted by berndbausch

My guess is that a resource that the application expects to exist doesn't exist. It then desperately tries to access this resource rather than issuing an error message or otherwise handling the situation gracefully. The resource might be a file or an inter-process communication facility.

You could run strace to see what system calls contribute to the high CPU usage. Try strace -c, which creates a system call CPU utilization summary.

You could also use lsof -p <PROCESS_ID> to look at open files and try to draw conclusions.

If the above doesn't help, if the application doesn't log anything or doesn't have a debug mode that provides some information, and if you can't ask the author of that application, I am afraid you are out of luck.

Thank you very much for your response There was a bug in the application coz of which some of the application processes were spending max time in system calls that had resulted in high CPU usage.
Now the server and the application is working fine.

fawaz25 · 03-26-2019, 02:56 AM

Quote:

Originally Posted by TB0ne

Haven't we been here before?
https://www.linuxquestions.org/quest...nd-4175643496/
https://www.linuxquestions.org/quest...er-4175648055/

Again, as you were told before, you're not telling us much of anything. What is this 'application'?? Version of VSphere? What have you actually done to diagnose the issue? Checked the logs? Run anything in debug mode? Anything??

You were also told previously that RHEL 5.x is VERY old, outdated, and totally unsupported. You are missing MANY YEARS of updates/patches/fixes, so if you're running on newer hardware, it's not surprising you're having trouble. And again, both RHEL and VSphere are commercial-PAY FOR products. If you're using them you need to pay, and contact/use the support you're entitled to.

Thanks for your response. I have checked the cause of high CPU usage using top command, along with whats causing high system calls using lsof command. There was an issue with the application that has been resolved as of now.

Regarding upgrading the server, I have discussed this with my higher management. Hopefully will do soon.

Regards.