iostat output

donneo · 08-10-2009, 04:25 AM

We have complaints from the users that there are read write issues in our VPS server. This is the sample output of the iostat command.

Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await svctm %util
sda 0.14 6.86 0.54 2.48 0.01 0.04 32.02 0.63 206.76 6.24 1.89
sdb 56.84 41.72 73.37 133.98 1.56 1.52 30.42 3.25 15.66 1.65 34.19
dm-0 0.00 0.00 0.00 0.00 0.00 0.00 9.05 0.00 25.89 24.28 0.00

where dm-0 indicated an LVM partition in the server. Each VPS server is created as a separate LVM. There is a LVM partition which always shows more than 90% disk usage.

dm-21 0.00 0.00 416.00 0.00 8.25 0.00 40.60 15.53 36.72 2.35 97.60

Is there any way to find out the details of this dm-21? I want to know which VPS is causing the problem. The lvdisplay command does not show id dm-21. So I am unable to track the exact VPS causing the problem. Any ideas?

tommylovell · 08-10-2009, 09:50 PM

I don't really know if this'll work, but try "cat /sys/class/block/dm-21/dev", then "dmsetup ls" to see what LV it is.

An example

Code:

[root@athlonz devices]# cd /sys/class/block/

[root@athlonz block]# ls
dm-0  dm-2  dm-4  fd0  md1   ram1   ram11  ram13  ram15  ram3  ram5  ram7  ram9  sda1  sda3  sdb   sdb2  sdb5
dm-1  dm-3  dm-5  md0  ram0  ram10  ram12  ram14  ram2   ram4  ram6  ram8  sda   sda2  sda5  sdb1  sdb3  sr0

[root@athlonz block]# cat /sys/class/block/dm-0/dev 
253:0

[root@athlonz block]# dmsetup ls
vgz01-t (253, 5)
vgz01-hfsplus   (253, 3)
vgz00-lvz01     (253, 0)
vgz00-lvz00     (253, 1)
vgz01-maclv     (253, 4)
vgz01-lvz00     (253, 2)

donneo · 08-11-2009, 07:43 AM

Thank you for the details.. it really helped. Can you please let me know what does (253, 3) indicate? I checked the man pages and could not find any helping details.

tommylovell · 08-11-2009, 04:59 PM

(253,3) are the major and minor number of the fourth LVM Logical Volume that you created.

The explanation that I've seen is that the major number is associated with a software driver for a certain type of device and the minor numbers represent the actual devices under it.

(If you Google "linux major device numbers" you can see quite a few articles trying to explain it. It's a simple concept, but few of these articles give a simple explanation, and I can't either...)

If you cat /proc/devices you can see under the block devices (disks are block devices) that '8' is 'sd' (scsi device) and '253' is device-mapper (which LVM uses -- see the Wikipedia article on "device mapper").

Your real /dev/sda device is probably 8,0; /dev/sda1 is probably 8,1; etc. While your LVM logical volumes are 253,nn.

If you had a hardware RAID array controller, you'd have a different major number and different names for the devices.)

Here is a display from a simple home x86 machine with IDE drives, software RAID (MD devices), and LVM2. You can see the major and minor numbers with an 'ls -l'.

Code:

[root@athlonz ~]# ls -l /dev/sd*
brw-rw---- 1 root disk 8,  0 2009-07-28 12:40 /dev/sda
brw-rw---- 1 root disk 8,  1 2009-07-28 12:41 /dev/sda1
brw-rw---- 1 root disk 8,  2 2009-07-28 12:40 /dev/sda2
brw-rw---- 1 root disk 8,  3 2009-07-28 12:40 /dev/sda3
brw-rw---- 1 root disk 8,  5 2009-07-28 12:40 /dev/sda5
brw-rw---- 1 root disk 8, 16 2009-07-28 12:40 /dev/sdb
brw-rw---- 1 root disk 8, 17 2009-07-28 12:41 /dev/sdb1
brw-rw---- 1 root disk 8, 18 2009-07-28 12:40 /dev/sdb2
brw-rw---- 1 root disk 8, 19 2009-07-28 12:40 /dev/sdb3
brw-rw---- 1 root disk 8, 21 2009-07-28 12:40 /dev/sdb5

[root@athlonz ~]# ls -l /dev/md*
brw-rw---- 1 root disk 9, 0 2009-07-28 12:40 /dev/md0
brw-rw---- 1 root disk 9, 1 2009-07-28 12:41 /dev/md1

[root@athlonz ~]# ls -l /dev/mapper/*
crw-rw---- 1 root root  10, 60 2009-07-28 12:40 /dev/mapper/control
brw-rw---- 1 root disk 253,  1 2009-07-28 12:40 /dev/mapper/vgz00-lvz00
brw-rw---- 1 root disk 253,  0 2009-07-28 12:41 /dev/mapper/vgz00-lvz01
brw-rw---- 1 root disk 253,  3 2009-07-28 12:41 /dev/mapper/vgz01-hfsplus
brw-rw---- 1 root disk 253,  2 2009-07-28 12:41 /dev/mapper/vgz01-lvz00
brw-rw---- 1 root disk 253,  4 2009-07-28 12:41 /dev/mapper/vgz01-maclv
brw-rw---- 1 root disk 253,  5 2009-07-28 12:41 /dev/mapper/vgz01-t

And to make it more interesting, more than one device name can have the same major/minor numbers and therefore represent the same device.

Code:

[root@athlonz ~]# ls -l /dev/root 
brw------- 1 root root 253, 0 2009-07-28 12:40 /dev/root

[root@athlonz ~]# ls -l /dev/dm-0
brw-rw---- 1 root disk 253, 0 2009-07-28 12:40 /dev/dm-0

[root@athlonz ~]# ls -l /dev/mapper/vgz00-lvz01
brw-rw---- 1 root disk 253, 0 2009-07-28 12:41 /dev/mapper/vgz00-lvz01

Hope that helps.

donneo · 08-11-2009, 10:58 PM

Hello tommylovell,

Thanx for the explanation and the tips to get more details. I managed to use the minor number details and find out the exact VPS :-). thanx a lot again :-)