Linux Server - Buffered disk read speed just 3.98MB/s Very Slow!

robotronik · 07-12-2010, 06:07 AM

Hi, I am having a problem with my CentOS based server where the hard drive speeds are incredibly slow for both of the hard drives installed -

Code:

/dev/hda:
 Timing cached reads:   18016 MB in  2.00 seconds = 9028.56 MB/sec
 Timing buffered disk reads:    6 MB in  3.01 seconds =   2.00 MB/sec
[root@freedom ~]# hdparm -tT /dev/hda2

/dev/hda2:
 Timing cached reads:   16512 MB in  2.00 seconds = 8272.92 MB/sec
 Timing buffered disk reads:   12 MB in  3.01 seconds =   3.98 MB/sec

I have looked around but do not understand the complexities of rebuilding kernel etc so would need some assistance. I have checked IO_support and it is set to 0

Code:

[root@freedom ~]# hdparm -v /dev/hda

/dev/hda:
 multcount    = 16 (on)
 IO_support   =  0 (default 16-bit)
 unmaskirq    =  0 (off)
 using_dma    =  0 (off)
 keepsettings =  0 (off)
 readonly     =  0 (off)
 readahead    = 256 (on)
 geometry     = 30401/255/63, sectors = 488397168, start = 0

Here are the results for hardware installed after a quick lspci -v

Code:

[root@freedom ~]# hdparm -d1 /dev/hda

/dev/hda:
 setting using_dma to 1 (on)
 HDIO_SET_DMA failed: Operation not permitted
 using_dma    =  0 (off)
[root@freedom ~]# lspci -v
00:00.0 Host bridge: Intel Corporation 82G35 Express DRAM Controller (rev 03)
        Subsystem: Intel Corporation Unknown device d701
        Flags: bus master, fast devsel, latency 0
        Capabilities: [e0] Vendor Specific Information

00:02.0 VGA compatible controller: Intel Corporation 82G35 Express Integrated Gr                                                                                                                        aphics Controller (rev 03) (prog-if 00 [VGA controller])
        Subsystem: Intel Corporation Unknown device d701
        Flags: bus master, fast devsel, latency 0, IRQ 11
        Memory at e0100000 (32-bit, non-prefetchable) [size=1M]
        Memory at d0000000 (64-bit, prefetchable) [size=256M]
        I/O ports at 2440 [size=8]
        Capabilities: [90] Message Signalled Interrupts: 64bit- Queue=0/0 Enable                                                                                                                        -
        Capabilities: [d0] Power Management version 2

00:02.1 Display controller: Intel Corporation 82G35 Express Integrated Graphics                                                                                                                         Controller (rev 03)
        Subsystem: Intel Corporation Unknown device d701
        Flags: bus master, fast devsel, latency 0
        Memory at e0000000 (32-bit, non-prefetchable) [size=1M]
        Capabilities: [d0] Power Management version 2

00:19.0 Ethernet controller: Intel Corporation 82566DC Gigabit Network Connectio                                                                                                                        n (rev 02)
        Subsystem: Intel Corporation Unknown device d701
        Flags: bus master, fast devsel, latency 0, IRQ 98
        Memory at e0200000 (32-bit, non-prefetchable) [size=128K]
        Memory at e0220000 (32-bit, non-prefetchable) [size=4K]
        I/O ports at 20c0 [size=32]
        Capabilities: [c8] Power Management version 2
        Capabilities: [d0] Message Signalled Interrupts: 64bit+ Queue=0/0 Enable                                                                                                                        +

00:1a.0 USB Controller: Intel Corporation 82801H (ICH8 Family) USB UHCI Controll                                                                                                                        er #4 (rev 02) (prog-if 00 [UHCI])
        Subsystem: Intel Corporation Unknown device d701
        Flags: bus master, medium devsel, latency 0, IRQ 177
        I/O ports at 20a0 [size=32]

00:1a.1 USB Controller: Intel Corporation 82801H (ICH8 Family) USB UHCI Controll                                                                                                                        er #5 (rev 02) (prog-if 00 [UHCI])
        Subsystem: Intel Corporation Unknown device d701
        Flags: bus master, medium devsel, latency 0, IRQ 233
        I/O ports at 2080 [size=32]

00:1a.7 USB Controller: Intel Corporation 82801H (ICH8 Family) USB2 EHCI Control                                                                                                                        ler #2 (rev 02) (prog-if 20 [EHCI])
        Subsystem: Intel Corporation Unknown device d701
        Flags: bus master, medium devsel, latency 0, IRQ 185
        Memory at e0221400 (32-bit, non-prefetchable) [size=1K]
        Capabilities: [50] Power Management version 2
        Capabilities: [58] Debug port

00:1c.0 PCI bridge: Intel Corporation 82801H (ICH8 Family) PCI Express Port 1 (r                                                                                                                        ev 02) (prog-if 00 [Normal decode])
        Flags: bus master, fast devsel, latency 0
        Bus: primary=00, secondary=01, subordinate=01, sec-latency=0
        Capabilities: [40] Express Root Port (Slot+) IRQ 0
        Capabilities: [80] Message Signalled Interrupts: 64bit- Queue=0/0 Enable                                                                                                                        +
        Capabilities: [90] #0d [0000]
        Capabilities: [a0] Power Management version 2

00:1c.1 PCI bridge: Intel Corporation 82801H (ICH8 Family) PCI Express Port 2 (r                                                                                                                        ev 02) (prog-if 00 [Normal decode])
        Flags: bus master, fast devsel, latency 0
        Bus: primary=00, secondary=02, subordinate=02, sec-latency=0
        Capabilities: [40] Express Root Port (Slot+) IRQ 0
        Capabilities: [80] Message Signalled Interrupts: 64bit- Queue=0/0 Enable                                                                                                                        +
        Capabilities: [90] #0d [0000]
        Capabilities: [a0] Power Management version 2

00:1c.2 PCI bridge: Intel Corporation 82801H (ICH8 Family) PCI Express Port 3 (r                                                                                                                        ev 02) (prog-if 00 [Normal decode])
        Flags: bus master, fast devsel, latency 0
        Bus: primary=00, secondary=03, subordinate=03, sec-latency=0
        I/O behind bridge: 00001000-00001fff
        Prefetchable memory behind bridge: 00000000e0300000-00000000e0300000
        Capabilities: [40] Express Root Port (Slot+) IRQ 0
        Capabilities: [80] Message Signalled Interrupts: 64bit- Queue=0/0 Enable                                                                                                                        +
        Capabilities: [90] #0d [0000]
        Capabilities: [a0] Power Management version 2

00:1d.0 USB Controller: Intel Corporation 82801H (ICH8 Family) USB UHCI Controll                                                                                                                        er #1 (rev 02) (prog-if 00 [UHCI])
        Subsystem: Intel Corporation Unknown device d701
        Flags: bus master, medium devsel, latency 0, IRQ 225
        I/O ports at 2060 [size=32]

00:1d.1 USB Controller: Intel Corporation 82801H (ICH8 Family) USB UHCI Controll                                                                                                                        er #2 (rev 02) (prog-if 00 [UHCI])
        Subsystem: Intel Corporation Unknown device d701
        Flags: bus master, medium devsel, latency 0, IRQ 50
        I/O ports at 2040 [size=32]

00:1d.2 USB Controller: Intel Corporation 82801H (ICH8 Family) USB UHCI Controll                                                                                                                        er #3 (rev 02) (prog-if 00 [UHCI])
        Subsystem: Intel Corporation Unknown device d701
        Flags: bus master, medium devsel, latency 0, IRQ 185
        I/O ports at 2020 [size=32]

00:1d.7 USB Controller: Intel Corporation 82801H (ICH8 Family) USB2 EHCI Control                                                                                                                        ler #1 (rev 02) (prog-if 20 [EHCI])
        Subsystem: Intel Corporation Unknown device d701
        Flags: bus master, medium devsel, latency 0, IRQ 225
        Memory at e0221000 (32-bit, non-prefetchable) [size=1K]
        Capabilities: [50] Power Management version 2
        Capabilities: [58] Debug port

00:1e.0 PCI bridge: Intel Corporation 82801 PCI Bridge (rev f2) (prog-if 01 [Sub                                                                                                                        tractive decode])
        Flags: bus master, fast devsel, latency 0
        Bus: primary=00, secondary=04, subordinate=04, sec-latency=32
        Capabilities: [50] #0d [0000]

00:1f.0 ISA bridge: Intel Corporation 82801HB/HR (ICH8/R) LPC Interface Controll                                                                                                                        er (rev 02)
        Subsystem: Intel Corporation Unknown device d701
        Flags: bus master, medium devsel, latency 0
        Capabilities: [e0] Vendor Specific Information

00:1f.2 IDE interface: Intel Corporation 82801H (ICH8 Family) 4 port SATA IDE Co                                                                                                                        ntroller (rev 02) (prog-if 8a [Master SecP PriP])
        Subsystem: Intel Corporation Unknown device d701
        Flags: bus master, 66MHz, medium devsel, latency 0, IRQ 50
        I/O ports at <ignored>
        I/O ports at <ignored>
        I/O ports at <ignored>
        I/O ports at <ignored>
        I/O ports at 2410 [size=16]
        I/O ports at 2400 [size=16]
        Capabilities: [70] Power Management version 3

00:1f.3 SMBus: Intel Corporation 82801H (ICH8 Family) SMBus Controller (rev 02)
        Subsystem: Intel Corporation Unknown device d701
        Flags: medium devsel, IRQ 233
        Memory at e0221800 (32-bit, non-prefetchable) [size=256]
        I/O ports at 2000 [size=32]

00:1f.5 IDE interface: Intel Corporation 82801H (ICH8 Family) 2 port SATA IDE Co                                                                                                                        ntroller (rev 02) (prog-if 85 [Master SecO PriO])
        Subsystem: Intel Corporation Unknown device d701
        Flags: bus master, 66MHz, medium devsel, latency 0, IRQ 50
        I/O ports at 2428 [size=8]
        I/O ports at 244c [size=4]
        I/O ports at 2420 [size=8]
        I/O ports at 2448 [size=4]
        I/O ports at 20f0 [size=16]
        I/O ports at 20e0 [size=16]
        Capabilities: [70] Power Management version 3

03:00.0 IDE interface: JMicron Technologies, Inc. JMB368 IDE controller (prog-if                                                                                                                         85 [Master SecO PriO])
        Subsystem: JMicron Technologies, Inc. JMB368 IDE controller
        Flags: bus master, fast devsel, latency 0, IRQ 185
        I/O ports at 1018 [size=8]
        I/O ports at 1024 [size=4]
        I/O ports at 1010 [size=8]
        I/O ports at 1020 [size=4]
        I/O ports at 1000 [size=16]
        Expansion ROM at e0300000 [disabled] [size=64K]
        Capabilities: [68] Power Management version 2
        Capabilities: [50] Express Legacy Endpoint IRQ 1
        Capabilities: [70] Message Signalled Interrupts: 64bit- Queue=0/0 Enable                                                                                                                        -

I really need to know if there are any noticeable incompatibilities here and how I could work to speed this up. I only have SSH access and could possibly get a KVM on it as it is currently locked up in a DataCenter.

Help would be greatly appreciated. Thanks very much.

business_kid · 07-12-2010, 06:38 AM

Quote:

Subsystem: Intel Corporation Unknown device d701

Hmmm.
There is a script /usr/sbin/update-pciids to keep the pci.ids file up to date. I've just grabbed the latest and it doesn't list d701. Is it this?
ftp://download.intel.com/support/mot...2/stl2_mem.pdf
Your problem is that you need your module for that southbridge (82801, I suspect)& sata compiled into the kernel, or at least in the initrd.
First try would be sticking the module into an initrd (man mkinitrd hopefully)
You can compile a the same kernel. The config is in /boot, and presuming /usr/src/linux for the kernel source, run
cp /boot/config-version /usr/src/linux/.config (notice the '.')
make menuconfig, change your driver, set generic as a module, and compile in your one.
You have the appaling vista before you of having to update to dodgy beta drivers and read (and perhaps write) stuff on LKML if this doesn't work

I would also bleat on the Centos lists, or no kernel update will have it. And tell the guy maintaining pci.ids what a d701 is

robotronik · 07-12-2010, 01:48 PM

In the end I found an easy solution! So no need to send in any replies.

If anyone else experiences this problem you can modify the existing grub menu.lst (/boot/grub/menu.lst) and add onto the end of the kernel line after LABEL=/

Code:

hda=noprobe hdc=noprobe

Works great and has since increased my speeds from 4MB/s to 80+

salmanucit · 07-13-2010, 10:53 PM

Hi Robotronic, i am facing the same kind of problem and IO rate for me is 17 MB/Sec. Can you please explain what doe's this mean what you suggested to add in "/boot/grub/menu.lst" file and any official document about this that what does it mean?
By the way, i have following setting in my menu.lst file, i have multiple entries on LABEL/=, where to add your suggested string

Code:

# grub.conf generated by anaconda
#
# Note that you do not have to rerun grub after making changes to this file
# NOTICE:  You do not have a /boot partition.  This means that
#          all kernel and initrd paths are relative to /, eg.
#          root (hd0,0)
#          kernel /boot/vmlinuz-version ro root=/dev/cciss/c0d0p1
#          initrd /boot/initrd-version.img
#boot=/dev/cciss/c0d0
# default=0 # commented out by Proliant HBA install script
# default=3 # commented out by Proliant HBA install script
default=4
fallback=3
# fallback=0 # commented out by Proliant HBA install script
timeout=5
splashimage=(hd0,0)/boot/grub/splash.xpm.gz
hiddenmenu
title Red Hat Enterprise Linux ES (2.6.9-22.ELsmp)
        root (hd0,0)
        kernel /boot/vmlinuz-2.6.9-22.ELsmp ro root=LABEL=/ rhgb quiet
        initrd /boot/initrd-2.6.9-22.ELsmp.img
title Red Hat Enterprise Linux ES-up (2.6.9-22.EL)
        root (hd0,0)
        kernel /boot/vmlinuz-2.6.9-22.EL ro root=LABEL=/ rhgb quiet
        initrd /boot/initrd-2.6.9-22.EL.img

# This entry (no. 2) added by Proliant HBA install script
# in package cpq_cciss-2.6.12-7.rhel4
title HP-2.6.9-22.EL
        root (hd0,0)
        kernel /boot/vmlinuz-2.6.9-22.EL ro root=LABEL=/ rhgb quiet
        initrd /boot/HP-initrd-2.6.9-22.EL.img


# This entry (no. 3) added by Proliant HBA install script
# in package cpq_cciss-2.6.12-7.rhel4
title HP-2.6.9-22.ELsmp
        root (hd0,0)
        kernel /boot/vmlinuz-2.6.9-22.ELsmp ro root=LABEL=/ rhgb quiet
        initrd /boot/HP-initrd-2.6.9-22.ELsmp.img


# This entry (no. 4) added by Proliant HBA install script
# in package mptlinux-3.12.14.00-5.rhel4
title HP-2.6.9-22.ELsmp-0
        root (hd0,0)
        kernel /boot/vmlinuz-2.6.9-22.ELsmp ro root=LABEL=/ rhgb quiet
        initrd /boot/HP-initrd-2.6.9-22.ELsmp.img-0

Thanks

Salman

robotronik · 07-14-2010, 03:02 AM

Quote:

Originally Posted by salmanucit

Hi Robotronic, i am facing the same kind of problem and IO rate for me is 17 MB/Sec. Can you please explain what doe's this mean what you suggested to add in "/boot/grub/menu.lst" file and any official document about this that what does it mean?

Hi Salman, what are your hard disk drives reporting as, are they sda or hda? HDA normally suggests that the problem is what I was facing somewhere along the line.

Anyway, if you want to test this fix, add the entry to all of the kernels displayed in menu.lst. So for example -

Code:

title Red Hat Enterprise Linux ES (2.6.9-22.ELsmp)
        root (hd0,0)
        kernel /boot/vmlinuz-2.6.9-22.ELsmp ro root=LABEL=/ rhgb quiet hda=noprobe hdc=noprobe
        initrd /boot/initrd-2.6.9-22.ELsmp.img
title Red Hat Enterprise Linux ES-up (2.6.9-22.EL)
        root (hd0,0)
        kernel /boot/vmlinuz-2.6.9-22.EL ro root=LABEL=/ rhgb quiet hda=noprobe hdc=noprobe
        initrd /boot/initrd-2.6.9-22.EL.img

Then after this you will want to reboot. Every time you update your kernel ensure this setting remains. But if this does not increase your speed, look to taking it off of the menu.lst file.

salmanucit · 07-14-2010, 05:07 AM

Thanks for your response.
I have actually SATA internal disk and i have HP MSA storage array which has SAS or SCSI i think, in this scenario, can you please suggest anything?
Thanks

robotronik · 07-16-2010, 04:32 AM

Afraid I couldn't. I am not that great with servers. Only know the basics. This method was recommended to me by a tech and all I did was relay the information on.