[SOLVED] System freezes with high wa%

gandrusz · 06-08-2010, 02:30 PM

Hi,

Recently my system (Debian testing - up to date with the repositories, including the kernel) started freezing. top command shows high %wa:

Code:

ga@grzes:~$ top
top - 20:27:10 up  1:35,  3 users,  load average: 3.90, 2.47, 1.20
Tasks: 166 total,   1 running, 165 sleeping,   0 stopped,   0 zombie
Cpu(s):  2.0%us,  0.5%sy,  0.0%ni,  0.0%id, 97.4%wa,  0.2%hi,  0.0%si,  0.0%st
Mem:   3374768k total,  1400584k used,  1974184k free,    28940k buffers
Swap:        0k total,        0k used,        0k free,   810344k cached

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND                                                                                                                                                                         
 3257 lp        40   0  8244 3288 2720 S    2  0.1   0:09.88 pdftops                                                                                                                                                                         
 3076 ga        20   0  117m  22m  15m S    1  0.7   0:04.53 chrome                                                                                                                                                                          
 1945 root      40   0 56648  44m  10m S    0  1.3   1:44.14 Xorg                                                                                                                                                                            
 2863 ga        40   0 86544  21m  14m S    0  0.7   0:22.95 konsole                                                                                                                                                                         
 3084 ga        20   0  377m  63m  22m S    0  1.9   0:07.77 firefox-bin                                                                                                                                                                     
 3166 lp        40   0  5580 2140 1124 S    0  0.1   0:00.10 socket                                                                                                                                                                          
    1 root      40   0  2036  688  596 S    0  0.0   0:00.74 init

This behaviour is very annoying, because I use it as the primary desktop and I can't even browse the internet when the freeze happens, films freeze, etc.

Any idea how can I diagnose the real problem and fix it? The default kernel is not pre-emptive any more?

salasi · 06-08-2010, 02:45 PM

Quote:

Originally Posted by gandrusz

...top command shows high %wa:

...you can say that again...

Code:

97.4%wa,

Does iotop show anything out of the ordinary? presumably this is 'wait for disk access? (does that seem to be correct, does the disk light flash, does vmstat show anything about lots of disk traffic?...I'm guessing that its not, eg, network access, because its hard to see how you'd get that much iowait from the network, but that doesn't rule it out.)

Quote:

I can't even browse the internet when the freeze happens...

Of course you can't...you've got ~2.5% of your potential cpu throughput left, after acounting for what goes in waiting.

gandrusz · 06-09-2010, 03:35 AM

Hi,

thanks for your reply. In the weird cases iotop shows:

Code:

 
TID  PRIO  USER     DISK READ  DISK WRITE  SWAPIN     IO>    COMMAND  
1203 be/4 root        0.00 B/s    0.00 B/s  0.00 % 99.99 % [kjournald]

total disk writes are usually either 0 or somewhere below 300 kB/s. I have a new 1.5TB Caviar Green drive, so I am use it can handle this kind of load easily :/

To sum up, nothing happens but it uses all IO ??????

Quote:

Of course you can't...you've got ~2.5% of your potential cpu throughput left, after acounting for what goes in waiting.

The thing is though that I am not doing anything intensive on the box - just browsing the internet! My 4 year old X60 doesn't freeze like this new desktop machine.

gandrusz · 06-09-2010, 05:00 AM

Ah, I just realised I always had issues with disk freezing the system on this computer (I assembled it earlier this year), it kind of just got more annoying recently (this could be a subjective statement though).

salasi · 06-10-2010, 03:51 AM

Quote:

Originally Posted by gandrusz

...I have a new 1.5TB Caviar Green drive,...

Not a 4k sector drive, by any chance? The new 1t Caviar Greens are, and I'm not so sure about the 1.5T models. What is the model number?

gandrusz · 06-11-2010, 01:56 AM

Quote:

Originally Posted by salasi

Not a 4k sector drive, by any chance? The new 1t Caviar Greens are, and I'm not so sure about the 1.5T models. What is the model number?

It is one of these new 4k #@!%$$#^%$#&@#@$%$#@ sector models

I have aligned the partitions to the 4k sector boundaries though (cfdisk seems to do a good job here - with default everything)

syg00 · 06-11-2010, 03:06 AM

Quote:

Originally Posted by salasi

Of course you can't...you've got ~2.5% of your potential cpu throughput left, after acounting for what goes in waiting.

I think this indicates a mis-understanding of the wait%. I happen to think this is a terrible metric (but one of several in Linux) - here's a citation from the sysstat doco that explains it pretty well

Quote:

%iowait
Show the percentage of time that the CPU or CPUs were idle during which the system had an outstanding disk I/O request.

Even that is not strictly correct, and there is no implication that the task(s) waiting on I/O are in fact also waiting to subsequently use the CPU. It can indicate problems with the I/O farm (as appears the case here), but needn't.

salasi · 06-11-2010, 06:41 AM

Quote:

Originally Posted by gandrusz

It is one of these new 4k #@!%$$#^%$#&@#@$%$#@ sector models

I have aligned the partitions to the 4k sector boundaries though (cfdisk seems to do a good job here - with default everything)

So, your partition alignment now corresponds to that given here/here? Or even here? And what state is the 'backward compatability jumper' in?

Certainly reports of problems with these drives are spreading, but mostly they seem to go away once the fiddling with sector numbers has been done.

@syg00

Quote:

...Even that is not strictly correct, and there is no implication that the task(s) waiting on I/O are in fact also waiting to subsequently use the CPU. It can indicate problems with the I/O farm (as appears the case here), but needn't.

Errm, yes, sorry, that is correct. But performance is always poor when 'wa' is high. I don't know why it should be that bad (waiting for reads is always going to block the app that was waiting for that read, writes are a different issue) so you'd expect to be able to get on with other, unrelated stuff, without massive obstruction, even if wa is high, but that doesn't seem to be how it works out in practice.

(BTW I/O Farm? It is a single drive, isn't it?)

gandrusz · 06-14-2010, 03:39 AM

Hi,

thanks for your reply.

Quote:

Originally Posted by salasi

So, your partition alignment now corresponds to that given here/here? Or even here? And what state is the 'backward compatability jumper' in?

Now I got completely confused by what it should be set to. Let me just paste my partition table and hopefully someone will be able to check if it is ok:

Code:

grzes:/home/ga# fdisk -lu /dev/sda

Disk /dev/sda: 1500.3 GB, 1500301910016 bytes
255 heads, 63 sectors/track, 182401 cylinders, total 2930277168 sectors
Units = sectors of 1 * 512 = 512 bytes
Disk identifier: 0x24b4d5f6

   Device Boot      Start         End      Blocks   Id  System
/dev/sda1            2048      206847      102400    7  HPFS/NTFS
Partition 1 does not end on cylinder boundary.
/dev/sda2          206848   409599999   204696576    7  HPFS/NTFS
/dev/sda3   *   409624320   604909311    97642496   83  Linux
/dev/sda4       604909312  2930272064  1162681376+   5  Extended
/dev/sda5       604909375   624444351     9767488+  82  Linux swap / Solaris
/dev/sda6       624444415  2930272064  1152913825   83  Linux
grzes:/home/ga# parted
GNU Parted 2.2
Using /dev/sda
Welcome to GNU Parted! Type 'help' to view a list of commands.
(parted) unit kib                                                         
(parted) print                                                            
Model: ATA WDC WD15EARS-00Z (scsi)
Disk /dev/sda: 1465138584kiB
Sector size (logical/physical): 512B/512B
Partition Table: msdos

Number  Start         End            Size           Type      File system  Flags
 1      1024kiB       103424kiB      102400kiB      primary   ntfs
 2      103424kiB     204800000kiB   204696576kiB   primary   ntfs
 3      204812160kiB  302454656kiB   97642496kiB    primary   ext3         boot
 4      302454656kiB  1465136032kiB  1162681377kiB  extended
 5      302454688kiB  312222176kiB   9767489kiB     logical
 6      312222208kiB  1465136032kiB  1152913825kiB  logical   ext3

On some other page I read that the latter need to be align, but the former is now completely off

What is the best way to actually test performance and see if I get what I should? (I don't care about the first two windows partitions)

Regarding the jumper, its NOT inserted, so the drive shouldn't offset anything.

On the side note, it actually might be my ASUS motherboard causing the biggest pain! I tried to change the system properties:

Code:

echo 50 > /proc/sys/vm/vfs_cache_pressure
echo deadline > /sys/block/sda/queue/scheduler
echo 1 > /sys/block/sda/device/queue_depth
echo 1024 > /sys/block/sda/queue/nr_requests

and it seems to improve the situation a bit.

PS. I don't think I will buy another ASUS motherboard and WD disk anytime soon

gandrusz · 06-14-2010, 01:54 PM

Btw, where should I put these echo commands if I want them to run on every system startup? (I am using Debian testing)

gandrusz · 06-21-2010, 05:23 AM

Fixed now, indeed my partition was not aligned correctly. I found this article the most useful:
http://www.linuxconfig.org/linux-wd-...dvanced-format