[SOLVED] ZFS copying stalls and then resumes

bucovaina78 · 08-24-2013, 05:11 AM

Hi

I've got this weird problem with ZFS.

I've got a few TB of data to copy from a desktop computer to a server as backup. I've used rsync before and now I'm working with midnight commander to copy the data. Weirdly the setup seems to read about 20MB of data at reasonable speed, then the progress bar in mc completely comes to a stop for a second or 2 then the data gets written and the process starts all over again. I get throughput of about 8MB/s. I am on a gigabit SOHO network with very little load so speed should be much and much higher.

It seems to me like some buffer needs to be filled up first, some checksums are being done and then the data gets written and it starts all over again.

I do get the same symtoms when I'm copying data from one ZFS volume to another (different HD's) within the same PC. So I guess it's not the network nor the server that is the bottleneck.

amani · 08-24-2013, 09:51 AM

Does it work fine with xfs, ext4, btrfs?

zfs options being used?

bucovaina78 · 09-11-2013, 08:33 AM

I've only used ext4 frequently with all the hardware I have so far. I only have this problem when I'm using ZFS. No special ZFS options are in use, it's a "vanilla" zfs(-fuse) install.

chrism01 · 09-13-2013, 12:22 AM

Tried the kernel version http://zfsonlinux.org/ ?

bucovaina78 · 09-16-2013, 08:39 AM

@chrism01: you are my hero! I can see a DRAMATIC increase in transfer speed, it went from 8MB/s to 133MB/S! (133MB/s is from device to device, not over the network)

However during the migrate from zfs-fuse to native zfs, my pool got degraded somehow, see terminal output below.

I did the following:
1. zpool offline tank
2. follow instructions on http://zfslinux.org
3. zpool online tank
4. zpool status

Code:

root@cosmos:~# zpool status
  pool: tank
 state: DEGRADED
status: One or more devices could not be used because the label is missing or
        invalid.  Sufficient replicas exist for the pool to continue
        functioning in a degraded state.
action: Replace the device using 'zpool replace'.
   see: http://zfsonlinux.org/msg/ZFS-8000-4J
  scan: none requested
config:

        NAME                                           STATE     READ WRITE CKSUM
        tank                                           DEGRADED     0     0     0
          raidz3-0                                     DEGRADED     0     0     0
            scsi-2001b4d2003019300                     ONLINE       0     0     0
            scsi-2001b4d2015387800                     ONLINE       0     0     0
            scsi-2001b4d2003202900                     ONLINE       0     0     0
            scsi-2001b4d2015486000                     ONLINE       0     0     0
            13730364079218549963                       UNAVAIL      0     0     0  was /dev/disk/by-i/ata-WDC_WD20EARS-00J2GB0_WD-WCAYY0084767
            ata-WDC_WD20EARX-00PASB0_WD-WMAZA7958042   ONLINE       0     0     0
            ata-WDC_WD20EARX-00PASB0_WD-WCAZA8279660   ONLINE       0     0     0
        logs
          ata-OCZ-AGILITY3_OCZ-5BUDCX5J7U6GGMP1-part2  ONLINE       0     0     0
        cache
          sdi1                                         ONLINE       0     0     0

errors: No known data errors
root@cosmos:~# zpool offline tank /dev/disk/by-id/ata-WDC_WD20EARS-00J2GB0_WD-WCAYY0084767
root@cosmos:~# zpool status
  pool: tank
 state: DEGRADED
status: One or more devices has been taken offline by the administrator.
        Sufficient replicas exist for the pool to continue functioning in a
        degraded state.
action: Online the device using 'zpool online' or replace the device with
        'zpool replace'.
  scan: none requested
config:

        NAME                                           STATE     READ WRITE CKSUM
        tank                                           DEGRADED     0     0     0
          raidz3-0                                     DEGRADED     0     0     0
            scsi-2001b4d2003019300                     ONLINE       0     0     0
            scsi-2001b4d2015387800                     ONLINE       0     0     0
            scsi-2001b4d2003202900                     ONLINE       0     0     0
            scsi-2001b4d2015486000                     ONLINE       0     0     0
            13730364079218549963                       OFFLINE      0     0     0  was /dev/disk/by-i/ata-WDC_WD20EARS-00J2GB0_WD-WCAYY0084767
            ata-WDC_WD20EARX-00PASB0_WD-WMAZA7958042   ONLINE       0     0     0
            ata-WDC_WD20EARX-00PASB0_WD-WCAZA8279660   ONLINE       0     0     0
        logs
          ata-OCZ-AGILITY3_OCZ-5BUDCX5J7U6GGMP1-part2  ONLINE       0     0     0
        cache
          sdi1                                         ONLINE       0     0     0

errors: No known data errors
root@cosmos:~# zpool replace tank /dev/disk/by-id/ata-WDC_WD20EARS-00J2GB0_WD-WCAYY0084767
the kernel failed to rescan the partition table: 16
cannot label 'sdk': try using parted(8) and then provide a specific slice: -1
root@cosmos:~# zpool replace tank /dev/sdk
invalid vdev specification
use '-f' to override the following errors:
/dev/sdk1 is part of exported pool 'tank'
root@cosmos:~# zpool replace -f tank /dev/sdk
the kernel failed to rescan the partition table: 16
cannot label 'sdk': try using parted(8) and then provide a specific slice: -1
root@cosmos:~# mc

root@cosmos:~#

bucovaina78 · 09-16-2013, 09:20 AM

Holy Moly!

I had to do a bit of "dangerous" stuff to my disks and pool but my zpool is resilvering now.

I've done:

Code:

$ su
# dd if=/dev/zero of=/dev

I tried to add the disk to the zpool but got the message the EFI label was broken (duh), then tried to make a new gpt label with parted but that didn't work, I had to reboot and try again.

Code:

# reboot
# parted /dev/disk/by-id/pathtomydisk
# (parted) mklabel gpt
# (parted) quit
# zpool replace tank /dev/disk/by-id/ata-WDC_WD20EARS-00J2GB0_WD-WCAYY0084767
# nmon #--> then pressed d to monitor disk activity and yes it is writing to /dev/sdk