NMI watchdog: BUG: soft lockup

huyuhui · 08-11-2020, 10:07 PM

OS: SLES 15.1, which is a KVM guest machine
Kernel: 4.12.14-197.51-default
Issue: NMI watchdog: BUG: soft lockup

I tried below but issue is still there.

Code:

echo 20 > /proc/sys/kernel/watchdog_thresh

Please advise. Thanks

James

Ser Olmy · 08-11-2020, 11:33 PM

You've provided absolutely no useful information regarding the circumstances under which this problem occurs, so I don't see how anybody could provide a suggestion, much less an answer.

Now, I happened to see your near-identical post in the SuSE forum where you actually did provide some information: That you're experiencing the watchdog bug message while transferring large-ish files via SFTP. Also, you provided a link to the page from where you'd gotten the advice to increase the watchdog threshold.

Here's the deal: You're seeing this issue because your VM is starved for resources. While transferring the file, the kernel is stuck somewhere for longer than the watchdog timeout allows, so the watchdog considers it a soft lockup and kicks in.

How many virtual CPUs does this VM have? If the answer is "one", you should definitely try adding one more. But perhaps more importantly, what does top (or htop) report with regards to memory usage while the file transfer is occurring? If the VM starts swapping, that could explain why it becomes unresponsive to the point where the watchdog is triggered. If neither CPU nor memory usage seems abnormal, you should check the performance of the disk subsystem holding the filesystem onto which you're writing the file.

Of course, you could try just increasing the watchdog timeout value to 30 or 40 seconds or even longer until the watchdog is no longer triggered, but that would not address the underlying issue.

huyuhui · 08-12-2020, 05:30 AM

Quote:

Originally Posted by Ser Olmy

You've provided absolutely no useful information regarding the circumstances under which this problem occurs, so I don't see how anybody could provide a suggestion, much less an answer.

Now, I happened to see your near-identical post in the SuSE forum where you actually did provide some information: That you're experiencing the watchdog bug message while transferring large-ish files via SFTP. Also, you provided a link to the page from where you'd gotten the advice to increase the watchdog threshold.

Here's the deal: You're seeing this issue because your VM is starved for resources. While transferring the file, the kernel is stuck somewhere for longer than the watchdog timeout allows, so the watchdog considers it a soft lockup and kicks in.

How many virtual CPUs does this VM have? If the answer is "one", you should definitely try adding one more. But perhaps more importantly, what does top (or htop) report with regards to memory usage while the file transfer is occurring? If the VM starts swapping, that could explain why it becomes unresponsive to the point where the watchdog is triggered. If neither CPU nor memory usage seems abnormal, you should check the performance of the disk subsystem holding the filesystem onto which you're writing the file.

Of course, you could try just increasing the watchdog timeout value to 30 or 40 seconds or even longer until the watchdog is no longer triggered, but that would not address the underlying issue.

Thank you for your reply and comments。
There are some VMs in the host as well.

Below is the status of top command in the host.

Code:

top - 18:20:54 up 569 days,  7:14,  2 users,  load average: 1.13, 1.05, 1.07
Tasks: 979 total,   1 running, 978 sleeping,   0 stopped,   0 zombie
Cpu(s):  1.1%us,  0.3%sy,  0.0%ni, 98.5%id,  0.0%wa,  0.0%hi,  0.0%si,  0.0%st
Mem:   2068091M total,  1008766M used,  1059324M free,      195M buffers
Swap:     2053M total,        0M used,     2053M free,   737135M cached

I temporarily set below parameters to zero to see if it'd be helpful.

Code:

/proc/sys/kernel/tainted
/proc/sys/kernel/watchdog
/proc/sys/kernel/nmi_watchdog
/proc/sys/kernel/soft_watchdog
/proc/sys/kernel/softlockup_panic
/proc/sys/kernel/unknown_nmi_panic

About the performance of the disk subsystem that the VM guest resides, could you please share your thoughts more? I did not get the point of potential impact.