[SOLVED] Netdata reports 10min cpu iowait

Toadman · 08-11-2020, 11:08 AM

Netdata has been running since 1 Aug however since 4 Aug it's been reporting the below:

Code:

"3:26:49 AM",system.cpu,"10min cpu iowait",16.5%,CLEAR
"3:19:49 AM",system.cpu,"10min cpu iowait",41.2%,WARNING
"8/8/2020 3:32:49 AM",system.cpu,"10min cpu iowait",16.9%,CLEAR
"8/8/2020 3:19:49 AM",system.cpu,"10min cpu iowait",40.9%,WARNING
"8/7/2020 3:28:48 AM",system.cpu,"10min cpu iowait",16.8%,CLEAR
"8/7/2020 3:19:48 AM",system.cpu,"10min cpu iowait",41.1%,WARNING
"8/4/2020 3:31:46 AM",system.cpu,"10min cpu iowait",16.2%,CLEAR
"8/4/2020 3:19:46 AM",system.cpu,"10min cpu iowait",40.5%,WARNING

Some system info:

Code:

Kernel	Linux 5.4.0-42-generic (x86_64)
Version	#46~18.04.1-Ubuntu SMP Fri Jul 10 07:21:24 UTC 2020
Distribution	Ubuntu 18.04.5 LTS
Processors
Package Information
Intel(R) Core(TM)2 Duo CPU E8400 @ 3.00GHz	0	0:0	2731.74 MHz
Intel(R) Core(TM)2 Duo CPU E8400 @ 3.00GHz	1	0:1	2787.58 MHz
MemTotal Total Memory	15326184 KiB
SwapTotal	Virtual Memory	4089852 KiB

It seems to start at the same time within a few seconds. I see nothing odd in syslog. The only thing that is running during that time is an rsync backup to an external usb drive. It starts at 03:10am each morning and ends depending on how much data is backed up. For instance on 08/04 it started at 03:10 and ended 03:25. Other days noted in the log:

Code:

04/07 03:10 - 03:22
04/08 03:10 - 03:26
04/11 03:10 - 03:20

Honestly I'm not sure if this is even anything to be concerned about. If any more data is needed I'd be happy to provide it.

berndbausch · 08-11-2020, 02:38 PM

iowait normally means disk IO. The fact that it happens at the time the backup runs is not surprising.

Use top or vmstat to check if you have a large iowait figure now, and iostat to check your current IO activity. sar also reports these metrics.

Toadman · 08-11-2020, 03:40 PM

Ran iostat, it all looks good to me so I'll agree with you that it's more than likely the backup causing it. Just something I never noticed before running Netdata.

Code:

iostat -c 5 25
Linux 5.4.0-42-generic (localhost) 	08/11/2020 	_x86_64_	(2 CPU)

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
          12.78    0.03    4.02    1.38    0.00   81.80
avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           5.14    0.00    2.82    0.00    0.00   92.04
avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           4.77    0.00    2.03    0.30    0.00   92.90
avg-cpu:  %user   %nice %system %iowait  %steal   %idle
          15.28    0.00    5.93    1.71    0.00   77.09
avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           8.76    0.00    3.02    1.81    0.00   86.40
avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           5.34    0.00    2.62    0.20    0.00   91.83
avg-cpu:  %user   %nice %system %iowait  %steal   %idle
          15.94    0.00    3.83    0.20    0.00   80.02
avg-cpu:  %user   %nice %system %iowait  %steal   %idle
          13.69    0.00    4.70    0.90    0.00   80.72
avg-cpu:  %user   %nice %system %iowait  %steal   %idle
          14.21    0.00    4.74    0.71    0.00   80.34
avg-cpu:  %user   %nice %system %iowait  %steal   %idle
          35.77    0.00   12.58    0.09    0.00   51.55
avg-cpu:  %user   %nice %system %iowait  %steal   %idle
          30.41    0.00    8.69    4.44    0.00   56.47
avg-cpu:  %user   %nice %system %iowait  %steal   %idle
          43.50    0.00   14.00    3.02    0.00   39.48
avg-cpu:  %user   %nice %system %iowait  %steal   %idle
          14.45    0.00    2.64    0.00    0.00   82.91
avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           9.01    0.00    3.40    0.40    0.00   87.19
avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           5.84    0.00    3.22    0.00    0.00   90.95
avg-cpu:  %user   %nice %system %iowait  %steal   %idle
          17.60    0.00    6.20    0.40    0.00   75.80
avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           8.81    0.00    4.20    1.60    0.00   85.39
avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           5.99    0.00    3.69    0.30    0.00   90.02
avg-cpu:  %user   %nice %system %iowait  %steal   %idle
          15.43    0.00    4.51    0.30    0.00   79.76
avg-cpu:  %user   %nice %system %iowait  %steal   %idle
          17.20    0.00    5.30    1.90    0.00   75.60
avg-cpu:  %user   %nice %system %iowait  %steal   %idle
          10.01    0.00    3.70    0.50    0.00   85.79
avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           8.57    0.00    3.69    1.10    0.00   86.64
avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           5.34    0.00    3.12    0.60    0.00   90.94
avg-cpu:  %user   %nice %system %iowait  %steal   %idle
          20.44    0.00    6.91    2.00    0.00   70.64
avg-cpu:  %user   %nice %system %iowait  %steal   %idle
          12.54    0.00    3.78    3.78    0.00   79.90

I'll mark this thread as solved, appreciate the reply

syg00 · 08-11-2020, 06:45 PM

Quote:

Originally Posted by Toadman

Just something I never noticed before running Netdata.

Just another questionable metric used inappropriately. Get some users to log on around then and play some games. Problem solved.