Hi everyone,
This weekend I migrated my home server from an old machine to a much newer one. The old machine performs at an acceptable speed, but I've decided to move to the newer one because it's silly not to use it.
This is your typical do-everything home server: nat, web, email, mysql, ssh. Some other little things, too.
The issue I am having is that disk writes are absolutely dismal in speed.
I am using a custom 2.4.22 kernel. Loadble module support is disabled as I prefer to do on servers. The machine has 2 IDE hard drives. Here is the output of 'lspci':
00:00.0 Host bridge: Silicon Integrated Systems [SiS] 650 Host (rev 01)
00:01.0 PCI bridge: Silicon Integrated Systems [SiS] SiS 530 Virtual PCI-to-PCI bridge (AGP)
00:02.0 ISA bridge: Silicon Integrated Systems [SiS] 85C503/5513
00:02.2 USB Controller: Silicon Integrated Systems [SiS] SiS7001 USB Controller (rev 07)
00:02.3 USB Controller: Silicon Integrated Systems [SiS] SiS7001 USB Controller (rev 07)
00:02.5 IDE interface: Silicon Integrated Systems [SiS] 5513 [IDE] (rev d0)
00:07.0 Ethernet controller: Digital Equipment Corporation DECchip 21142/43 (rev 41)
00:08.0 Ethernet controller: National Semiconductor Corporation DP83815 (MacPhyter) Ethernet Controller
01:00.0 VGA compatible controller: ATI Technologies Inc Radeon VE QY
Pretty simple machine. 'hdparm -I' gives:
/dev/hda:
ATA device, with non-removable media
Model Number: WDC WD307AA
Serial Number: WD-WMA111213665
Firmware Revision: 05.05B05
Standards:
Supported: 4 3 2 1
Likely used: 5
Configuration:
Logical max current
cylinders 16383 16383
heads 16 16
sectors/track 63 63
--
bytes/track: 57600 bytes/sector: 600
CHS current addressable sectors: 16514064
LBA user addressable sectors: 60074784
device size with M = 1024*1024: 29333 MBytes
device size with M = 1000*1000: 30758 MBytes (30 GB)
Capabilities:
LBA, IORDY(can be disabled)
Buffer size: 2048.0kB bytes avail on r/w long: 40 Queue depth: 1
Standby timer values: spec'd by Standard, with device specific minimum
R/W multiple sector transfer: Max = 16 Current = 16
DMA: mdma0 mdma1 mdma2 udma0 udma1 udma2 udma3 *udma4
Cycle time: min=120ns recommended=120ns
PIO: pio0 pio1 pio2 pio3 pio4
Cycle time: no flow control=120ns IORDY flow control=120ns
Commands/features:
Enabled Supported:
* READ BUFFER cmd
* WRITE BUFFER cmd
* Host Protected Area feature set
* Look-ahead
* Write cache
* Power Management feature set
* SMART feature set
* DOWNLOAD MICROCODE cmd
HW reset results:
CBLID- above Vih
Device num = 0 determined by the jumper
and 'hdparm -I /dev/hdd' gives:
/dev/hdd:
ATA device, with non-removable media
Model Number: Maxtor 6Y160P0
Serial Number: Y427V78E
Firmware Revision: YAR41BW0
Standards:
Supported: 7 6 5 4
Likely used: 7
Configuration:
Logical max current
cylinders 16383 16383
heads 16 16
sectors/track 63 63
--
CHS current addressable sectors: 16514064
LBA user addressable sectors: 268435455
LBA48 user addressable sectors: 320173056
device size with M = 1024*1024: 156334 MBytes
device size with M = 1000*1000: 163928 MBytes (163 GB)
Capabilities:
LBA, IORDY(can be disabled)
Queue depth: 1
Standby timer values: spec'd by Standard, no device specific minimum
R/W multiple sector transfer: Max = 16 Current = 16
Advanced power management level: unknown setting (0x0000)
Recommended acoustic management value: 192, current value: 254
DMA: mdma0 mdma1 mdma2 udma0 udma1 udma2 udma3 udma4 *udma5 udma6
Cycle time: min=120ns recommended=120ns
PIO: pio0 pio1 pio2 pio3 pio4
Cycle time: no flow control=120ns IORDY flow control=120ns
Commands/features:
Enabled Supported:
* NOP cmd
* READ BUFFER cmd
* WRITE BUFFER cmd
* Host Protected Area feature set
* Look-ahead
* Write cache
* Power Management feature set
Security Mode feature set
* SMART feature set
* FLUSH CACHE EXT command
* Mandatory FLUSH CACHE command
* Device Configuration Overlay feature set
* 48-bit Address feature set
* Automatic Acoustic Management feature set
SET MAX security extension
Advanced Power Management feature set
* DOWNLOAD MICROCODE cmd
* SMART self-test
* SMART error logging
Security:
Master password revision code = 65534
supported
not enabled
not locked
not frozen
not expired: security count
not supported: enhanced erase
HW reset results:
CBLID- above Vih
Device num = 1 determined by the jumper
Checksum: correct
When I am copying files, either over the network, or with network disabled and just copying locally, the machine's speed becomes bursty. Watching 'vmstat 1' in one window , the IO/bytes-in column is nice and even, but the IO/bytes-out column will remain at 0 for roughly 10 iterations (seconds) and then have a massive burst, freezing the machine to user input or screen output while doing this massive write. It does eventually finish, but it takes far too long.
The load average spikes to about 4 every time it copies data for longer than a minute or so. (It takes about that long to climb to that point, but then it remains at that load average). I have one processor and it does not support hyperthreading (1.7ghz celeron) Although the load average is 4.0, the system averages about 80% idle during the process.
This happens when I copy many hundreds of thousands of small files, or when I copy a few multi hundred megabyte files (mailboxes). It happens locally, or over the network.
Here's an interesting thing to consider, though: The stock 2.4.20 kernel that comes with Red Hat 9.0 works! Performance is excellent.
I tried dropping back to 2.4.20 myself but still encoutered the same bursty issue.
My system drive, hda, is an ext3 partition, and my data drive, hdd, is a reiserfs filesystem. My initial thought was that it was reiser, but upon commenting it out in the fstab file and rebooting, user data to the system drive, ext3, was still bursty. Very confusing. I am very fortunate that I am encountering this on a personal machine instead of a production work server for the first time, though!!!
Oh yes, one more thing. While copying the data and monitoring 'vmstat 1', the screen will post about 3 lines of random cyrillic characters. This is obviously some sort of corrupted output, and I have no other way to elaborate on it.
What other tools should I be using to help me get some more clues? Does this stick out to anybody? And did I neglect to mention anything that should have been mentioned about the system? I'm perhaps incorrectly forced to assume that it has to be software related as there is a working software configuration in the stock kernel which I don't care to use.
/etc/fstab:
LABEL=/ / ext3 defaults 1 1
/dev/hdd1 /data reiserfs defaults 1 2
none /dev/pts devpts gid=5,mode=620 0 0
none /proc proc defaults 0 0
none /dev/shm tmpfs defaults 0 0
/dev/hda2 swap swap defaults 0 0
/dev/fd0 /mnt/floppy auto noauto,owner,kudzu 0 0
dmesg:
Linux version 2.4.22 (root@soy) (gcc version 3.2.2 20030222 (Red Hat Linux 3.2.2-5)) #10 Mon Sep 22 01:02:41 EDT 2003
BIOS-provided physical RAM map:
BIOS-e820: 0000000000000000 - 00000000000a0000 (usable)
BIOS-e820: 00000000000f0000 - 0000000000100000 (reserved)
BIOS-e820: 0000000000100000 - 000000000fff0000 (usable)
BIOS-e820: 000000000fff0000 - 000000000fff3000 (ACPI NVS)
BIOS-e820: 000000000fff3000 - 0000000010000000 (ACPI data)
BIOS-e820: 00000000fec00000 - 00000000fec01000 (reserved)
BIOS-e820: 00000000fee00000 - 00000000fee01000 (reserved)
BIOS-e820: 00000000ffff0000 - 0000000100000000 (reserved)
255MB LOWMEM available.
On node 0 totalpages: 65520
zone(0): 4096 pages.
zone(1): 61424 pages.
zone(2): 0 pages.
Kernel command line: auto BOOT_IMAGE=2.4.22 ro BOOT_FILE=/boot/2.4.22.8
Initializing CPU#0
Detected 1693.133 MHz processor.
Console: colour VGA+ 80x25
Calibrating delay loop... 3381.65 BogoMIPS
Memory: 255964k/262080k available (1928k kernel code, 5732k reserved, 459k data, 288k init, 0k highmem)
Dentry cache hash table entries: 32768 (order: 6, 262144 bytes)
Inode cache hash table entries: 16384 (order: 5, 131072 bytes)
Mount cache hash table entries: 512 (order: 0, 4096 bytes)
Buffer cache hash table entries: 16384 (order: 4, 65536 bytes)
Page-cache hash table entries: 65536 (order: 6, 262144 bytes)
CPU: Trace cache: 12K uops, L1 D cache: 8K
CPU: L2 cache: 128K
Intel machine check architecture supported.
Intel machine check reporting enabled on CPU#0.
CPU: After generic, caps: 3febfbff 00000000 00000000 00000000
CPU: Common caps: 3febfbff 00000000 00000000 00000000
CPU: Intel(R) Celeron(R) CPU 1.70GHz stepping 03
Enabling fast FPU save and restore... done.
Enabling unmasked SIMD FPU exception support... done.
Checking 'hlt' instruction... OK.
POSIX conformance testing by UNIFIX
PCI: PCI BIOS revision 2.10 entry at 0xfb430, last bus=1
PCI: Using configuration type 1
PCI: Probing PCI hardware
PCI: Probing PCI hardware (bus 00)
PCI: Using IRQ router default [1039/0650] at 00:00.0
Linux NET4.0 for Linux 2.4
Based upon Swansea University Computer Society NET3.039
Initializing RT netlink socket
Starting kswapd
Journalled Block Device driver loaded
Installing knfsd (copyright (C) 1996
okir@monad.swb.de).
pty: 256 Unix98 ptys configured
Serial driver version 5.05c (2001-07-08) with MANY_PORTS SHARE_IRQ SERIAL_PCI enabled
ttyS00 at 0x03f8 (irq = 4) is a 16550A
ttyS01 at 0x02f8 (irq = 3) is a 16550A
Floppy drive(s): fd0 is 1.44M
FDC 0 is a post-1991 82077
eth0: DC21143 at 0xe000 (PCI bus 0, device 7), h/w address 00:c0:f0:4d:c4:33,
and requires IRQ12 (provided by PCI BIOS).
de4x5.c:V0.546 2001/02/22
davies@maniac.ultranet.com
SLIP: version 0.8.4-NET3.019-NEWTTY (dynamic channels, max=256).
Linux Tulip driver version 0.9.15-pre12 (Aug 9, 2002)
PCI: Unable to reserve I/O region #1:80@e000 for device 00:07.0
PCI: Unable to reserve I/O region #1:80@e000 for device 00:07.0
natsemi dp8381x driver, version 1.07+LK1.0.17, Sep 27, 2002
originally by Donald Becker <becker@scyld.com>
url deleted to post as newbie
2.4.x kernel port by Jeff Garzik, Tjeerd Mulder
eth1: NatSemi DP8381[56] at 0xd0800000, 00:a0:cc:75:95:6d, IRQ 11.
PPP generic driver version 2.4.2
PPP Deflate Compression module registered
PPP BSD Compression module registered
Universal TUN/TAP device driver 1.5 (C)1999-2002 Maxim Krasnyansky
Linux agpgart interface v0.99 (c) Jeff Hartmann
agpgart: Maximum main memory to use for agp memory: 203M
agpgart: Detected SiS 650 chipset
agpgart: AGP aperture is 64M @ 0xe0000000
[drm] Initialized tdfx 1.0.0 20010216 on minor 0
[drm] AGP 0.99 on SiS @ 0xe0000000 64MB
[drm] Initialized radeon 1.1.1 20010405 on minor 1
[drm] AGP 0.99 on SiS @ 0xe0000000 64MB
[drm] Initialized i810 1.2.0 20010920 on minor 2
Uniform Multi-Platform E-IDE driver Revision: 7.00beta4-2.4
ide: Assuming 33MHz system bus speed for PIO modes; override with idebus=xx
hda: WDC WD307AA, ATA DISK drive
hdd: Maxtor 6Y160P0, ATA DISK drive
ide0 at 0x1f0-0x1f7,0x3f6 on irq 14
ide1 at 0x170-0x177,0x376 on irq 15
hda: attached ide-disk driver.
hda: host protected area => 1
hda: 60074784 sectors (30758 MB) w/2048KiB Cache, CHS=3739/255/63
hdd: attached ide-disk driver.
hdd: host protected area => 1
hdd: 320173056 sectors (163929 MB) w/7936KiB Cache, CHS=19929/255/63
Partition check:
hda: hda1 hda2
hdd: hdd1
es1371: version v0.32 time 01:05:12 Sep 22 2003
usb.c: registered new driver hub
host/uhci.c: USB Universal Host Controller Interface driver v1.1
NET4: Linux TCP/IP 1.0 for NET4.0
IP Protocols: ICMP, UDP, TCP, IGMP
IP: routing cache hash table of 2048 buckets, 16Kbytes
TCP: Hash tables configured (established 16384 bind 32768)
ip_conntrack version 2.1 (2047 buckets, 16376 max) - 292 bytes per conntrack
ip_tables: (C) 2000-2002 Netfilter core team
ipt_recent v0.3.1: Stephen Frost <sfrost@snowman.net>. url deleted to post as newbie
arp_tables: (C) 2002 David S. Miller
NET4: Unix domain sockets 1.0/SMP for Linux NET4.0.
kjournald starting. Commit interval 5 seconds
EXT3-fs: mounted filesystem with ordered data mode.
VFS: Mounted root (ext3 filesystem) readonly.
Freeing unused kernel memory: 288k freed
EXT3 FS 2.4-0.9.19, 19 August 2002 on ide0(3,1), internal journal
Adding Swap: 522104k swap-space (priority -1)
reiserfs: found format "3.6" with standard journal
reiserfs: checking transaction log (device ide1(22,65)) ...
for (ide1(22,65))
ide1(22,65):Using r5 hash to sort names
eth0: media is TP.
eth1: link up.
eth1: Setting full-duplex based on negotiated link capability.
