LVM: pvcreate fails on 5TB drive with "pe_align (2048 sectors) must not be less than pe_align_offset (8388607 sectors)"

furryspider · 09-10-2016, 02:30 PM

Dear all,

I was trying to make a new 5 TB hard drive into a backup disk by first encrypting it with LUKS and then creating a bunch of logical volumes on it. Alas, I'm already stopped by pvcreate:

Code:

# pvcreate -v /dev/mapper/luks5tb

Wiping internal VG cache
Wiping cache of LVM-capable devices
Wiping signatures on new PV /dev/mapper/luks5tb.
/dev/mapper/luks5tb: pe_align (2048 sectors) must not be less than pe_align_offset (8388607 sectors)
Format-specific initialisation of physical volume /dev/mapper/luks5tb failed.
Failed to setup physical volume "/dev/mapper/luks5tb".

I did some research, which yielded little but hinted at a bug that apparently was closed quite a while ago.

I tried to partition the drive into smaller portions (2TB + rest), but pvcreate gave the same error.

So, with nothing to go on, I grabbed the first live disk I could find (System Rescue CD 4.2.0), booted it and tried pvcreate there. And it worked (which is also the reason I'm posting this in the Slackware forums), even with the much older version of LVM running there (Slackware 14.2: LVM version: 2.02.154(2) (2016-05-14) / Library version: 1.02.125 (2016-05-14) / Driver version: 4.34.0; SysRescueCD 4.2.0: LVM version: 2.02.103(2) (2013-10-04) / Library version: 1.02.82 (2013-10-04) / Driver version: 4.24.0).

I'm running Slackware 14.2 32bit, but I would rule this out as the problem because it worked with both 32bit and 64bit kernels using SysRescueCD.

After pvcreate ran successfully in SysRescCD, Slackware actually recognizes the PVs with pvscan, and pvck doesn't give any errors.

Any ideas why I can't use pvcreate for this in Slackware? Any similar experiences?
Please let me know if further information is required. Thanks!

syg00 · 09-10-2016, 06:58 PM

You might find this thread interesting.

furryspider · 09-11-2016, 08:02 AM

Thanks syg00, that thread is indeed interesting. The min/opt io size values of my disk (Seagate) are the same as for brobr.

I'm not sure though that this is at the root of what I'm seeing here. I wouldn't even have thought of checking disk alignment with parted, as neither gdisk, nor cryptsetup, nor mkfs gave any complaints. It was only when I decided to go the LUKS+LVM way that problems began.

Anyway, the info in the thread made me experiment with partition alignment. I first changed to a 4096 boundary (as opposed to the 2048 boundary used by gdisk). Then I tried 8192. Then 65535 (which gdisk refused), then 65536. In all cases, I first had pvcreate run directly on the partition (which always worked, even on standard partition start at 2048), and then I ran luksFormat on them (after which pvcreate always failed).

I tried to play with --align-payload (4096 and 65536) in luksFormat, not really understanding what this does, but also to no avail.

Out of curiosity, I tried another, more up-to-date live distro for comparison (Fedora 24). It shows the same behaviour as Slackware 14.2.

Not really sure yet what to make of all of this.

---

A little supplemental:
I experimented with LVM's --dataalignment and --dataalignmentoffset switches, and this line successfully created a PV:

Code:

# pvcreate -v --dataalignmentoffset 1024k /dev/mapper/luks5tb

Question is: Am I doing some damage by manually setting the offset (eg. to the LUKS volume or the future LVs)? luksClose and luksOpen worked fine afterwards.

kjhambrick · 09-12-2016, 08:49 AM

furryspider ( and brobr if you're reading this ) --

This is something that I am sure we are going to eventually see in the field at our Customer Sites ( they tend to buy whatever they find at Office Depot or ... ).

I would sure like to try to find a work-around in 'the lab' before we run into it out in the world.

The false misalignment reports seem to be a function of running 'Advanced Format' Drives in USB Enclosures.

If you don't mind my asking ...

What Makes / Models of USB Enclosure are you running ?

Or if you're working with an all-in-one, Integrated Device which one(s) are you running ?

Thank you !

-- kjh

furryspider · 09-12-2016, 11:41 AM

kjhambrick, my drive is a Seagate Expansion Desktop 5TB (STEB5000200). As mentioned, none of my 'regular' tools complained, only pvcreate, so most people who buy this kind of drive will probably never be aware of any trouble.

Concerning my original problem: I searched for information concerning LUKS, and as far as I understand it, LVM should never be able to touch anything that isn't exposed by the device mapper (= header and key slots), ie. pvcreate shouldn't be able to overwrite my LUKS header even if I pass it an ill conceived --dataalignmentoffset value. Can anybody confirm that this is a correct assumption?

Following this, if pvck doesn't report any problems after creating a PV in the abovementioned way, I would assume that the creation was successful and I have a correctly working PV. Again, is this feasible, or am I missing something here?

Sorry for my ignorance, and thanks in advance!

kjhambrick · 09-12-2016, 01:06 PM

furryspider --

Thanks for the good news about pvcreate !

But unfortunately, I don't have any answers, only questions which is why I need to 'play with' one of these Drives in the Lab.

Given the number of hits on google, I imagine we'll eventually run into this issue out in the field on our Customer's Systems.

We provide support for three types of Systems -- 1) Data Conversion Appliances, 2) Integrated Application Appliances and 3) Installations of #1 or #2 on Customer-Provided Hardware.

The Integrated Application Systems almost always require Authentication against an Active Directory Domain Controller but we're not ready to ship Slackware or install Slackware remotely on these Machines yet ( first we need to figure out PAM + KRB5 + SAMBA ).

We will stick with CentOS 6.x until we figure out how to do AD Authentication for these Systems.

OTOH, we are now ready to ship our first Slackware 14.2 Data Conversion Appliance ( !!! woohoo !!! ).

In addition to { 1,2,3 }, we also support External Drive(s) for BackUps ( presently either USB or eSATA External Enclosures ) and if we don't provide the Hardware, we will format the drive(s) on an External Device that our Customers have acquired for themselves.

Today, USB 3 seems the only way to go for 'low end' External Drive Enclosures ( low-end is something other than an existing NAS or Tape Unit ) and these days eSATA has been replaced by USB 3 ( too bad about that ).

Encryption is essential these days for ANY device holding sensitive data that could 'up and walk away' and you're the second Slackware User and about the 'zillionth' Linux User reporting 'oddities' with AF Drives in USB Enclosures.

Anyhow ... I found this: Seagate Expansion 5TB (STEB5000100) at Amazon.

The model is slightly different than yours, but does it look somewhat similar ?

If so, I will order one for the Lab and start testing and post any findings, good or bad here on LQ.

And thank you for the feedback !

-- kjh

furryspider · 09-12-2016, 01:56 PM

kjhambrick, appearance and description of the linked drive are the same as for mine. Price tag is even lower... storage really is getting cheaper these days.

brobr · 09-12-2016, 04:52 PM

@furryspider: the fact that I realized that the sectors on my disc were not properly aligned was when I created a swap partition in a lvm volume. That raised an error that lead to the thread syg00 referred to; and was also highlighted by "(parted) align-check opt". No errors were raised by gdisk and all the start-settings I tried with gdisk did not bypass the parted warning, i.e. the same experience as you described.

@kjhambrick: I was using an "Inateck 2.5 Inch USB 3.0 Hard Drive Disk HDD Aluminum External Enclosure Case with usb 3.0 Cable for 9.5mm 7mm 2.5" SATA" bought in 2014; visible in dmesg as:

Code:

[42385.301896] usb 4-2: new SuperSpeed USB device number 2 using xhci_hcd
[42385.313845] usb 4-2: New USB device found, idVendor=174c, idProduct=55aa
[42385.313853] usb 4-2: New USB device strings: Mfr=2, Product=3, SerialNumber=1
[42385.313856] usb 4-2: Product: ASMT1051
[42385.313858] usb 4-2: Manufacturer: asmedia
[42385.313861] usb 4-2: SerialNumber: 1234567892DC
[42385.327290] usbcore: registered new interface driver usb-storage

and with lsusb -v

Code:

Bus 004 Device 002: ID 174c:55aa ASMedia Technology Inc. ASM1051E SATA 6Gb/s bridge, ASM1053E SATA 6Gb/s bridge, ASM1153 SATA 3Gb/s bridge
Device Descriptor:
  bLength                18
  bDescriptorType         1
  bcdUSB               3.00
  bDeviceClass            0 
  bDeviceSubClass         0 
  bDeviceProtocol         0 
  bMaxPacketSize0         9
  idVendor           0x174c ASMedia Technology Inc.
  idProduct          0x55aa ASM1051E SATA 6Gb/s bridge, ASM1053E SATA 6Gb/s bridge, ASM1153 SATA 3Gb/s bridge
  bcdDevice            1.00
  iManufacturer           2 asmedia
  iProduct                3 ASMT1051
  iSerial                 1 1234567892DC
  bNumConfigurations      1
  Configuration Descriptor:
    bLength                 9
    bDescriptorType         2
    wTotalLength          121
    bNumInterfaces          1
    bConfigurationValue     1
    iConfiguration          0 
    bmAttributes         0xc0
      Self Powered
    MaxPower                0mA
    Interface Descriptor:
      bLength                 9
      bDescriptorType         4
      bInterfaceNumber        0
      bAlternateSetting       0
      bNumEndpoints           2
      bInterfaceClass         8 Mass Storage
      bInterfaceSubClass      6 SCSI
      bInterfaceProtocol     80 Bulk-Only
      iInterface              0 
      Endpoint Descriptor:
        bLength                 7
        bDescriptorType         5
        bEndpointAddress     0x81  EP 1 IN
        bmAttributes            2
          Transfer Type            Bulk
          Synch Type               None
          Usage Type               Data
        wMaxPacketSize     0x0400  1x 1024 bytes
        bInterval               0
        bMaxBurst              15
      Endpoint Descriptor:
        bLength                 7
        bDescriptorType         5
        bEndpointAddress     0x02  EP 2 OUT
        bmAttributes            2
          Transfer Type            Bulk
          Synch Type               None
          Usage Type               Data
        wMaxPacketSize     0x0400  1x 1024 bytes
        bInterval               0
        bMaxBurst              15
    Interface Descriptor:
      bLength                 9
      bDescriptorType         4
      bInterfaceNumber        0
      bAlternateSetting       1
      bNumEndpoints           4
      bInterfaceClass         8 Mass Storage
      bInterfaceSubClass      6 SCSI
      bInterfaceProtocol     98 
      iInterface              0 
      Endpoint Descriptor:
        bLength                 7
        bDescriptorType         5
        bEndpointAddress     0x81  EP 1 IN
        bmAttributes            2
          Transfer Type            Bulk
          Synch Type               None
          Usage Type               Data
        wMaxPacketSize     0x0400  1x 1024 bytes
        bInterval               0
        bMaxBurst              15
        MaxStreams             16
        Data-in pipe (0x03)
      Endpoint Descriptor:
        bLength                 7
        bDescriptorType         5
        bEndpointAddress     0x02  EP 2 OUT
        bmAttributes            2
          Transfer Type            Bulk
          Synch Type               None
          Usage Type               Data
        wMaxPacketSize     0x0400  1x 1024 bytes
        bInterval               0
        bMaxBurst              15
        MaxStreams             16
        Data-out pipe (0x04)
      Endpoint Descriptor:
        bLength                 7
        bDescriptorType         5
        bEndpointAddress     0x83  EP 3 IN
        bmAttributes            2
          Transfer Type            Bulk
          Synch Type               None
          Usage Type               Data
        wMaxPacketSize     0x0400  1x 1024 bytes
        bInterval               0
        bMaxBurst              15
        MaxStreams             16
        Status pipe (0x02)
      Endpoint Descriptor:
        bLength                 7
        bDescriptorType         5
        bEndpointAddress     0x04  EP 4 OUT
        bmAttributes            2
          Transfer Type            Bulk
          Synch Type               None
          Usage Type               Data
        wMaxPacketSize     0x0400  1x 1024 bytes
        bInterval               0
        bMaxBurst               0
        Command pipe (0x01)
Binary Object Store Descriptor:
  bLength                 5
  bDescriptorType        15
  wTotalLength           22
  bNumDeviceCaps          2
  USB 2.0 Extension Device Capability:
    bLength                 7
    bDescriptorType        16
    bDevCapabilityType      2
    bmAttributes   0x00000002
      HIRD Link Power Management (LPM) Supported
  SuperSpeed USB Device Capability:
    bLength                10
    bDescriptorType        16
    bDevCapabilityType      3
    bmAttributes         0x00
    wSpeedsSupported   0x000e
      Device can operate at Full Speed (12Mbps)
      Device can operate at High Speed (480Mbps)
      Device can operate at SuperSpeed (5Gbps)
    bFunctionalitySupport   1
      Lowest fully-functional device speed is Full Speed (12Mbps)
    bU1DevExitLat          10 micro seconds
    bU2DevExitLat        2047 micro seconds
can't get debug descriptor: Resource temporarily unavailable
Device Status:     0x000d
  Self Powered
  U1 Enabled
  U2 Enabled

I have a couple of other enclosures so, ideally I could test these for you but I am a bit tired of all the formatting and I need to get this drive inside my computer. In preparation for that, I have already transferred non-sensitive stuff (like music etc) that I do not want to put within the encrypted part. At the moment I am waiting for a gap in my work to get this transfer done.

Rob

kjhambrick · 09-12-2016, 05:43 PM

Thanks brobr.

No need to do any more testing for me

And I've got a feeling that things may change for the better after you install that HDD into your Laptop.

I've got a couple ( similar ) OYEN Digital Pro 2.5 inch USB 3 / eSATA Enclosures and a few spare HGST EA Drives here in the Lab.

As I mentioned in the other post, we connect via eSATA if possible but eSATA ports are less popular than they were before USB 3 was introduced so we're using USB 3 more often now.

Interestingly, I've never had an alignment issue on CentOS 6.x with the USB Interface ( and I wouldn't expect one with eSATA ) ...

I'll plug one of the OYEN's via USB 3 into my Slackware64 14.2 LapTop and see what I can see.

In the meantime, I ordered the Seagate STEB5000200 which should be here in time for a little testing next week, depending on my 'work load'.

Thanks for the info Y'All !

-- kjh

brobr · 09-12-2016, 06:23 PM

Yeah, eSATA is a great interface; it created a tough, shock-proof connection with the cable and allowed booting from an external drive, something I haven't been able to accomplish with uefi/usb3. Be warned with that Seagate you ordered; it comes with a micro-B usb connector; the cable comes out too easily from these which can be a big risk for data-transfer; a couple of disconnections will destroy the disc by accumulating I/O errors, as happened with my external HGST drive. Your external drive should not be able to move around easily and the cable should not be disrupted too. If you're used to eSATA, the design of the micro-B usb-connector will seem absurdly bad. Best would be if they had used the new USB3-C type connector; that creates a more solid connection that is not so prone to breakage.

furryspider · 09-18-2016, 04:18 AM

For what it's worth: I just tried brobr's method using parted, and indeed parted itself reports the partition as aligned on sector 65535. gdisk doesn't complain when I set the partition type, neither does cryptsetup on making it a luks volume. However, pvcreate still fails (without having manually set --dataalignmentoffset). It still says "pe_align (2048 sectors) must not be less than pe_align_offset (8388607 sectors)".

furryspider · 10-11-2016, 12:33 PM

Since no-one (here or elsewhere) seems to really know what's going on, I decided to ignore parted's sector alignment misgivings and go with what worked for me in the past. I set up a single partition using gdisk, starting on sector 4096. This was formatted with luks, and the luks volume was made into a PV with a manually set data alignment offset of 2048s. Afterwards, creation of VG and several LV worked without problems. Accessibility and data retrieval were tested on three different machines with five different systems (just to be on the safe side, flying half-blind as I'm doing here...), all without problems. In worst case, I may lose some performance by not having physical and logical sector boundaries aligned (if this really be the case). But as it's a backup disk, I guess I'll be fine.

As the original issue / the root cause is still unclear, I'm not marking this solved for the time being.

almope · 10-14-2016, 05:21 PM

Quote:

Originally Posted by furryspider

5 TB hard drive into a backup disk by first encrypting it with LUKS

I seem to recall something about encrypting volumes larger than 2TB could weaken the encryption because of a repeating pattern. This was some time ago though and might have been a truecrypt issue. While not related to your lvm issue it might be relevant for you to look into.