is my hard drive really at 100% utilization?

sneakyimp · 09-11-2007, 12:43 AM

So I just got iostat installed on my machine. I didn't reboot it if that makes any difference (it's been up for 465 days) and I'm getting output which suggests my hard drives are over worked. Is that the case? Also, what is the difference between 'hda' and 'hda1', 'hda2', etc?

Code:

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
          16.50    0.00    2.40    0.00    0.00   81.10

Device:         rrqm/s   wrqm/s     r/s     w/s    rkB/s    wkB/s avgrq-sz avgqu-sz   await  svctm  %util
hda               0.00     8.60    0.00    9.60     0.00    73.70    15.35  3287.15   35.94 104.17 100.00
hda1              0.00     0.00    0.00    0.10     0.00     0.10     2.00     0.02  160.00 160.00   1.60
hda2              0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00   0.00   0.00
hda3              0.00     8.60    0.00    9.50     0.00    73.60    15.49     0.33   34.63   1.68   1.60
hdb               0.00     0.00    0.00    0.00     0.00     0.00     0.00 429476.73    0.00   0.00 100.00
hdb1              0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00   0.00   0.00

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           9.10    0.00    1.40    0.00    0.00   89.50

Device:         rrqm/s   wrqm/s     r/s     w/s    rkB/s    wkB/s avgrq-sz avgqu-sz   await  svctm  %util
hda               0.00     8.30    0.00    3.10     0.00    44.70    28.84 428488.57    0.00 322.58 100.00
hda1              0.00     0.10    0.00    0.20     0.00     0.30     3.00     0.00    0.00   0.00   0.00
hda2              0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00   0.00   0.00
hda3              0.00     8.20    0.00    2.90     0.00    44.40    30.62     0.00    0.00   0.00   0.00
hdb               0.00     0.00    0.00    0.00     0.00     0.00     0.00 429476.73    0.00   0.00 100.00
hdb1              0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00   0.00   0.00

Junior Hacker · 09-11-2007, 02:35 AM

Might be worth reading the man pages or info pages for iostat to see what all the numbers represent. Hda is the first hard drive as a whole, hda1 is the first partition on the first drive, hdb1 is the first partition on the second hard drive.
The df command will tell you flat out how much free space your partitions have, for example, I just ran the df command on my Debian installation in VMware in Windows and below is what it tells me:

Code:

yomama:/home/jo# df
Filesystem           1K-blocks      Used Available Use% Mounted on
/dev/sda2              7405720    952772   6452948  13% /
tmpfs                   229852         0    229852   0% /lib/init/rw
udev                     10240        84     10156   1% /dev
tmpfs                   229852         0    229852   0% /dev/shm
.host:/              112639744  77354064  35285680  69% /mnt/hgfs
yomama:/home/jo#

/dev/sda2 is my root file system (second partition) and only 13% of it is used. .host:/ is a partition used by my host operating system Windows that I allowed Debian to see, it is 110GB in size and is 69% used up.

Junior Hacker · 09-11-2007, 02:46 AM

If that's too hard to understand, you can use the -h option which is for "human readable format" as such:

Code:

mymama:/home/jo# df -h
Filesystem            Size  Used Avail Use% Mounted on
/dev/sda2             7.1G  931M  6.2G  13% /
tmpfs                 225M     0  225M   0% /lib/init/rw
udev                   10M   84K   10M   1% /dev
tmpfs                 225M     0  225M   0% /dev/shm
.host:/               108G   74G   34G  69% /mnt/hgfs
mymama:/home/jo#

A little discrepancy in the sizes reported, my Debian root partition is about 7.5GB and my data partition should be 110GB, not 108GB. It may be because it reports in MiB and GiB instead of MB & GB, or something like that, or it sees a Kilobyte as 1000 Bytes instead of the actual 1024B in a Kilobyte.

syg00 · 09-11-2007, 02:50 AM

Quote:

Originally Posted by Junior Hacker

Might be worth reading the man pages or info pages for iostat to see what all the numbers represent.

Excellent suggestion - like vmstat, the first range of numbers is basically meaningless - especially for a new install of iostat like that.
Run it with an interval for a while.

sneakyimp · 09-11-2007, 04:00 AM

df is an entirely different command and those used/available numbers refer to the total space available on the hard drive rather than the utilization of instantaneous throughput which is what util refers to. I'm familiar with df...my hard drives are only about half full:

Code:

[root@mydomain root]# df -h
Filesystem            Size  Used Avail Use% Mounted on
/dev/hda3              36G   16G   19G  45% /
/dev/hdb1              37G   21G   15G  58% /backup
/dev/hda1             145M  6.3M  131M   5% /boot
none                  374M     0  374M   0% /dev/shm
[root@myplan root]#

iostat, on the other hand, reports a bunch of information about instantaneous usage of your hard drive to help you determine if your server is I/O bound. From the man pages for iostat:

Code:

              await
                     The  average  time  (in  milliseconds)  for  I/O requests
                     issued to the device to be served. This includes the time
                     spent by the requests in queue and the time spent servic-
                     ing them.

              svctm
                     The  average  service  time  (in  milliseconds)  for  I/O
                     requests that were issued to the device.

              %util
                     Percentage  of  CPU  time  during which I/O requests were
                     issued to  the  device  (bandwidth  utilization  for  the
                     device).  Device  saturation  occurs  when  this value is
                     close to 100%.

given the long await and svctm values for hda it might make sense that it's at 100% utilization. On the other hand, it's only writing 73KB per second over the first interval and 44KB/sec over the 2nd interval. Also, hdb reports 100% utilization despite having read and written nothing.

Junior Hacker · 09-11-2007, 02:46 PM

I'm having problems finding out what the "avgqu-sz" is all about, I can guess, but would rather not. I was going to say I don't have iostat installed and can't read the man pages, but that's a crock, google can pull up any man page, just... I don't think the ones I'm finding are as recent or robust as yours.
Based on the first paragraph of iostat's description, I would tend to think you may be interpreting the first line wrong. To me, it would seem the intervals of the partitions are the lines that tell the story as to how much bandwidth saturation your drives have. And I don't think the await and svctm figures are all that bad, probably normal because hard drives are near the bottom of the memory pyramid, and not all drives have the same access time, cheaper drives can have an access time three times that of an expensive line. According to the intervals, the drive's partitions activity are quite minimal:

Quote:

The iostat utility iteratively reports terminal, disk, and
tape I/O activity, as well as CPU utilization. The first
line of output is for all time since boot; each subsequent
line is for the prior interval only.

Is /dev/hda3 a /home partition, or swap? Because if it is only doing "read ahead", the nice level may be low as it's only stacking data into ram for future use, not for immediate use, which could explain the 73 and 44KB/sec.