[SOLVED] Handling NVMe devices in Slackware : Call for testing.

Didier Spaier · 01-15-2017, 01:01 PM

Hi there,

I have modified the Slackware installer (scripts /usr/sbin/probe and /usr/lib/setup/SeTEFI) and /sbin/eliloconfig so that Slackware be able to:

Install Slackware on an NVMe device's partition
Install a bootloader in an ESP on a NVMe device
Install a boot entry for Slackware in the firmware's boot menu in case where the boot image is in an ESP on a NVMe device.

I have made limited testing in a VirtualBox VM but do not own a NVMe device thus this is a call for testers on bare metal.

In that aim I provide a mini-iso here, "mini" meaning that this ISO includes no package, only the installer itself.

To test you need:

A computer equipped with a NVMe device, with Slackware installed on it or not, set to boot in UEFI mode.
The ISO written with dd on an USB stick

WARNING: when asked, do *not* accept to format any partition, unless you wan to make a real installation and have at hand a Slackware installation media that you will use as source of packages.

TIP: if you need to partition your NVMe device, before running setup do this:

type lsblk to know how it is named (probably nvme0n1)
to format it, type "cfdisk /dev/nvme0n1" for instance. Of course set a GPT partition, not a DOS one.
Your partitions will be named nvme0n1p<partition number>
As usual you will need an EFI partition (100M) and a Linux partition. You can also set a swap partition if you want.

TESTING PROCEDURE

If you really install Slackware, you will need a media (DVD or USB stick or on a network) with the Slackware packages, that you will select when asked at the SOURCE step. Other than that, proceed to the installation as usual.
Else (Slackware already installed), do not format any existing partition and skip the INSTALL step but perform the CONFIGURE step so that you can check if you are proposed to install elilo on the ESP and a boot entry, and check that succeeds.

QUESTIONS:

Is your swap partition (if any) on NVMe recognized by the installer?
Is your Linux partition on NVMe recognized by the installer?
Is elilo properly installed in the ESP on the NVMe?
Is the elilo boot entry successfully written in the firmware's boot menu?

I attach to this post the patches used to build the ISO and the patch to eliloconfig.

NOTE You will need to apply the patch to /usr/sbin/eliloconfig in the installed system yourself, *before* performing the CONFIGURE step. To do that:

Alt+F2 to start another virtual terminal
cd /mnt/usr/sbin
either download the patch with wget or get it from another USB stick where you will have stored it, mounted as /floppy. Assuming that you have copied it in /mnt/usr/sbin, do:
Code:
```
patch -p0 < eliloconfig.diff.txt
```

Thanks for reading and in advance for your test reports.

PS Some links about NVMe:
Overview
An explanation, including partitions layout
The current specification

The kernel module:

Code:

root[/home/didier]# modinfo nvme
filename:       /lib/modules/4.4.38/kernel/drivers/nvme/host/nvme.ko
version:        1.0
license:        GPL
author:         Matthew Wilcox (email hidden by Didier)
srcversion:     DBC9558E0C13F0D5DCDD26D
alias:          pci:v0000106Bd00002001sv*sd*bc*sc*i*
alias:          pci:v*d*sv*sd*bc01sc08i02*
depends:        
intree:         Y
vermagic:       4.4.38 SMP mod_unload 
parm:           admin_timeout:timeout in seconds for admin commands (byte)
parm:           io_timeout:timeout in seconds for I/O (byte)
parm:           shutdown_timeout:timeout in seconds for controller shutdown (byte)
parm:           nvme_major:int
parm:           nvme_char_major:int
parm:           use_threaded_interrupts:int
parm:           use_cmb_sqes:use controller's memory buffer for I/O SQes (bool)

PPS I realize now that the possible devices' list in SeTEFI is probably wrong. But this shouldn't matter if you have only one, in other words for preliminary testing.

Didier Spaier · 01-16-2017, 03:10 PM

More references:

NVM Express Drives and Digital Forensics. Good presentation including pics of the physical connectors and how to query the devices. Also from the same author (Bruce Nikkel) there is a book: "Practical Forensic Imaging: Securing Digital Evidence with Linux Tools".
Implementing NVMe Drives on Lenovo Servers brings both a practical presentation of this kind of devices with pictures and the usable Linux command to manage them. Very handy.
Intel® Rapid Storage Technology enterprise for NVMe Drives in Linux OS is not a must read but presents examples of the mdadm command for NVMe devices.

Let's borrow an explanation to Bruce Nikkel:

Quote:

NVME devices are not SATA or SCSI, and therefore not represented as /dev/sd* devices under the Linux device directory. They have an alternate file naming convention beginning with /dev/nvme*. The naming convention allows for representation of multiple devices, which may contain multiple namespaces, which in turn may contain multiple partitions. For example, a host with a single NVME drive containing one namespace with three partitions appears as follows:

Code:

# ls /dev/nvme*
/dev/nvme0  /dev/nvme0n1  /dev/nvme0n1p1  /dev/nvme0n1p2  /dev/nvme0n1p3

Here "nvme0" refers to the character device of the nvme drive, "n1" refers to the raw block device of the namespace, and "p*" refer to the three partition devices (normal partitions, created with fdisk)

So, what?

In nvme0n1, nvme is the type of device, 0 is the volume number (if you have three devices they will be named nvme0n1, nvme1n1 and nvme 2n1) and 1 the namespace number. If the first controller had several namespaces we could see n2, n3 etc. instead of n1. But as says also Bruce :

Quote:

Consumer NVME devices on the market today typically only support a single namespace.

Namespaces are mentioned in the current specification under 6.1 Namespaces, page 143.

So I will modify at least SetEFI to handle at most 10 NVMe SSDs, but with only one namespace as that should be enough for now, build another ISO with that and post again here when that will be ready to grab.

I will also try to build nvme-cli and use it in a VM.

Stay tuned (if you want).

PS I think that my modified eliloconfig was wrong too. Will re-check.

Didier Spaier · 01-17-2017, 05:39 AM

To make a long story short, you will find here:

A new ISO image (only the modified installer)
The fixed patches in patches/
A nvme-cli package, and its source in nvme-cli/

I now understand that a controller can have as many namespaces as storage units attached to it.

So, I modified SeTEFI to be able to handle at most 10 controllers, each with at most 4 storage units attached.

I have also modified /sbin/probe to avoid duplicate entries in $TMP/SeTfdisk

I have also modified eliloconfig to fix the nvme partitions' names length including /dev/ (14 characters).

Everything should work now as expected. That is the case in a VM at least, see attached pics.

PS Feedback for testing on bare metal is of course still welcome. Be careful with the "nvme" command. Read the man pages: begin with typing just "nvme" as root then "nvme help <argument>". They are also provided as html pages. Just type <your preferred browser> /usr/doc/nvme-cli-1.1/nvme.html to read them.

Didier Spaier · 01-20-2017, 02:53 PM

No answer... Nevertheless I applied these patches to Slint and will let them in the same place for a while in case Patrick wants to apply them to Slackware current.

AlleyTrotter · 02-24-2017, 10:28 AM

Hello Didier
I am currently looking at a new system, NVM would be included. It will probably be a while until I get it put together. However your work is appreciated and when the hardware is available to me I will definitely test your work on bare metal.
Thanks
John

AlleyTrotter · 03-03-2017, 02:15 PM

Purchased a Plextor 512GB PCIe NVMe memory board
Newegg product description 512GB | PLEXTOR PX-512M8PEY RTL
P/N:3C07140090
S/N P0265110157

power on with the Plextor inserted in the
PCIe express 3.0 x 16 slot on the motherboard
which is an ASRock Z77 Extreme 4

ASROCK UEFI SETUP UTILITY
shows

Code:

PLEXTOR PX-512MBPeY

at this point I attempet to boot from the NVMe and got a screen of 99's
(expected this of course as the nvme board was unitialized)

booted my full slackware64-14.2 using 4.10.1 kernel

dmesg
[code [ 4.451869] nvme nvme0: pci function 0000:01:00.0 [/code]

tryed fdisk -l
...the nvme part...

Code:

Disk /dev/nvme0n1: 477 GiB, 512110190592 bytes, 1000215216 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes

tryed

Code:

gdisk -l /dev/nvme0n1
GPT fdisk (gdisk) version 1.0.0

Partition table scan:
  MBR: not present
  BSD: not present
  APM: not present
  GPT: not present

Creating new GPT entries.
Disk /dev/nvme0n1: 1000215216 sectors, 476.9 GiB
Logical sector size: 512 bytes
Disk identifier (GUID): DB3139E4-84A8-4721-BC52-52B1817E4087
Partition table holds up to 128 entries
First usable sector is 34, last usable sector is 1000215182
Partitions will be aligned on 2048-sector boundaries
Total free space is 1000215149 sectors (476.9 GiB)

Number  Start (sector)    End (sector)  Size       Code  Name

tryed

Code:

lspci -v
...
01:00.0 Non-Volatile memory controller: Lite-On Technology Corporation Device 22f1 (rev 01) (prog-if 02 [NVM Express])
        Subsystem: Marvell Technology Group Ltd. Device 1093
        Flags: bus master, fast devsel, latency 0, IRQ 16
        Memory at f7b20000 (64-bit, non-prefetchable) [size=16K]
        Expansion ROM at f7b00000 [disabled] [size=128K]
        Capabilities: [40] Power Management version 3
        Capabilities: [50] MSI: Enable- Count=1/1 Maskable+ 64bit+
        Capabilities: [70] Express Endpoint, MSI 00
        Capabilities: [b0] MSI-X: Enable+ Count=19 Masked-
        Capabilities: [100] Advanced Error Reporting
        Capabilities: [148] Device Serial Number 00-00-00-00-00-00-00-00
        Capabilities: [158] Power Budgeting <?>
        Capabilities: [168] Alternative Routing-ID Interpretation (ARI)
        Capabilities: [178] #19
        Capabilities: [2b8] Latency Tolerance Reporting
        Capabilities: [2c0] L1 PM Substates
        Kernel driver in use: nvme
        Kernel modules: nvme

Sorry for the tease, but that is all I have time for right know.
Next I will try your fake installer to see if it sees the NVMe board.
From the above I think the standard UEFI installer will see it.
We shall see
john

PS on the box
">Supports Operating System
Windows OS / Linux OS @ PC Platform
"

Didier Spaier · 03-03-2017, 02:26 PM

Thanks for the follow up John.

You could also try a Slint64-14.2 ISO that includes a modified installer.

Oh, and for your information: now cfdisk (and also fdisk I assume) can handle GUID partition tables (GPT) as do cgdisk/gdisk.

PS Looking at your output I could be wrong about fdisk. But no issue with cfdisk and GPT.

AlleyTrotter · 03-04-2017, 05:54 PM

Quote:

Originally Posted by Didier Spaier

Hi there,

Is your swap partition (if any) on NVMe recognized by the installer?
Is your Linux partition on NVMe recognized by the installer?
Is elilo properly installed in the ESP on the NVMe?
Is the elilo boot entry successfully written in the firmware's boot menu?

Hi Didier
didn't use a swap on the NVMe
Linux partition was recognized
elilo was installed (not sure how properly)
firmware's boot memory got all screwed up!

firstly installing an NVMe board changed the UEFI memory map it became FS0
FS0 was previously /dev/sda
FS1 was previously /dev/sdb
everything was shifted up one, took a while for me to figure that out

I took the leap and used your installer to install slack from a hard drive partition. It appeared to install correctly including the efi partition, however it would not boot. Nothing would boot took me all afternoon to get things straightened out had to restore things using efibootmgr and Shellx64.efi. The recovered system sees the NVMe board and can read and write fine. I have a few more tricks up my sleeve to try yet but that is all the time I have for now.
Please provide any suggestions or additional tests you would like me to try.
John

AlleyTrotter · 03-05-2017, 07:26 AM

I did another install and tried to install lilo no good it did not install.
The third time is the charm sort of. On the third install using your fakeDVD and a partition for the Slackware goodies. I did not install lilo or elilo but choose to make a boot stick. Surprise surprise my system will UEFI boot from a boot stick to an NVMe installed Slackware64-14.2.
You are definitely on the right path Didier, probably setting your efibootmgr in the installer to use a different name beside slackware would keep it from overwriting the working UEFI nvm ram entries or shifting them around.
Thanks
John

Off to Church for now.

AlleyTrotter · 03-05-2017, 01:01 PM

I am able to boot the nvme partition by adding the last stanza to my Slackware elilo.conf located on /dev/sda1.

Code:

chooser=simple
delay=50
timeout=50
default=4101
#
image=bzImage-4.10.1
        label=4101
        read-only
        append="root=/dev/sda2 vga=normal ro"
#
image=bzImage-4.4.52
        label=4452
        read-only
        append="root=/dev/sda2 vga=normal ro"
#
image=bzImage-4.9.13
  label=last
  read-only
  root=/dev/sda2
#
image=vmlinuz-3.13.3-lfs-7.5
  label=lfs
  read-only
  append="root=/dev/sdb8 vga=normal ro"
#
# was image=bzImage-rescue
image=bzImage-4.6.3
  label=rescue
  read-only
  append="root=/dev/sdb2 vga=normal ro"
#
image=bzImage-4.4.15
  label=build
  read-only
  append="root=/dev/sdb4 vga=normal ro"
#
image=bzImage-4.10.0
  label=build10
  read-only
  append="root=/dev/sdb4 vga=normal ro"
#
image=vmlinuz
        label=nvm
        read-only
        append="root=/dev/nvme0n1p2 vga=normal ro"

So as I see it efibootmgr should write a new UEFI entry for the nvme without over writing the Slackware entry or shifting the other entries. I think it just needs to have a different label perhaps "-L NVMe" for testing purposes or even ask user for the label.
just my 2 cents
Thanks
John

PS: I used to think my ssd was fast. The NVMe makes it seem like a floppy LOL!

Didier Spaier · 03-05-2017, 03:06 PM

I am glad you can use your NVMe device.

To better understand your situation (as I infer from your last post that you have at least five systems on three mass storage devices), please give the output of this command

Code:

lsblk -o model,name,size,fstype,mountpoint
fdisk -l|grep ^/dev #this one typed as root

And tell what is the usage of all the listed partitions

Hopefully this will help me give you a sound advice and/or ask relevant questions.

AlleyTrotter · 03-05-2017, 06:32 PM

Attached your two requested items.
Always something new to learn
Glad to help/test anything.
Fixing my mistakes is what makes life fun.
John

AlleyTrotter · 03-06-2017, 02:51 PM

Quote:

Originally Posted by Didier Spaier

Hi there,
...
I have modified the Slackware installer (scripts /usr/sbin/probe and /usr/lib/setup/SeTEFI) and /sbin/eliloconfig so that Slackware be able to:

Is your swap partition (if any) on NVMe recognized by the installer?
Is your Linux partition on NVMe recognized by the installer?
Is elilo properly installed in the ESP on the NVMe?
Is the elilo boot entry successfully written in the firmware's boot menu?

I attach to this post the patches used to build the ISO and the patch to eliloconfig.

NOTE You will need to apply the patch to /usr/sbin/eliloconfig in the installed system yourself, *before* performing the CONFIGURE step. To do that:

Alt+F2 to start another virtual terminal
cd /mnt/usr/sbin
either download the patch with wget or get it from another USB stick where you will have stored it, mounted as /floppy. Assuming that you have copied it in /mnt/usr/sbin, do:
Code:
```
patch -p0 < eliloconfig.diff.txt
```

Thanks for reading and in advance for your test reports.

YOU MAY KINDLY DISREGARD ALL MY PREVIOUS COMMUNICATIONS ON THIS SUBJECT

I get very excited when I get a new piece of hardware to play with, now that things have settled down here in the shack I can state

EVERYTHING WORKED AS YOU INDICATED

Seems I did not read your first post very carefully and missed the eliloconfig patch. This time after the install was complete and before rebooting I applied the eliloconfig.diff and ran eliloconfig. One additional (and very important) adjustment to /usr/sbin/eliloconfig I did an edit to change "-L Slackware" to "-L nvme". This prevented eliloconfig from over writing my Slackware entry in the UEFI NVRAM. I now have both my Slackware entry and a nvme entry the latter boots into the Plextor memory card.

The answer to all four of your questions is a resounding YES

Again Sorrowfully Yours
John
PS hope I did not cause you too much trouble

Didier Spaier · 03-06-2017, 03:03 PM

I am glad it worked and truly thankful that you took the time to test on bare metal and report the outcome.

No sorrow: be happy and have fun!

Cheers,

laprjns · 03-06-2017, 06:56 PM

Quote:

Originally Posted by AlleyTrotter

One additional (and very important) adjustment to /usr/sbin/eliloconfig I did an edit to change "-L Slackware" to "-L nvme". This prevented eliloconfig from over writing my Slackware entry in the UEFI NVRAM. I now have both my Slackware entry and a nvme entry the latter boots into the Plextor memory card.

Changing the label(-L) is not doing what you think it is. efibootmgr, called by elilconfig, doesn't overwrite UEFI NVRAM entries, but rather creates a new one. You can have multiple entries with the same labels each pointing to a different ESP bootloader or even the same one. All you would get is a list of menu entries with the same label when you invoke the UEFI firmware boot screen. Here an example of having NVRAM boot entries with the same lablel

Code:

rich[~]$ sudo efibootmgr -v
Password: 
BootCurrent: 000C
Timeout: 1 seconds
BootOrder: 000C,0005,0006,0004,0008,0007,0000,0001,0000,0002,0003
Boot0000* Unknown Device 	Vendor(99e275e7-75a0-4b37-a2e6-c5385e6c00cb,)AMBO
Boot0004* Salix	HD(1,800,96000,dccc0346-6342-49cc-a509-c22a6ff7ae0d)File(\EFI\Slint\elilo.efi)
Boot0005* Salix	HD(1,800,96000,dccc0346-6342-49cc-a509-c22a6ff7ae0d)File(\EFI\Salix-Xfce-14.2\elilo.efi)
Boot0006* Windows Boot Manager	HD(1,800,96000,dccc0346-6342-49cc-a509-c22a6ff7ae0d)File(\EFI\Microsoft\Boot\bootmgfw.efi)WINDOWS
Boot0007* UEFI: IP4 Realtek Ethernet Controller	ACPI(a0341d0,0)PCI(1c,2)PCI(0,0)MAC(08606e1a7626,0)IPv4(0.0.0.0:0<->0.0.0.0:0,0, 0AMBO
Boot0008* UEFI SHell	HD(1,800,96000,dccc0346-6342-49cc-a509-c22a6ff7ae0d)File(\EFI\tools\Shellx64.efi)
Boot000A  CD/DVD Drive 	BIOS(3,0,00)AMGOAMNO........,.P.2.:. .M.A.T.S.H.I.T.A.D.V.D.-.R.A.M. .U.J.8.B.2. . . . . .....(.....P2: MATSHITADVD-RAM
Boot000C* rEFInd	HD(1,800,96000,dccc0346-6342-49cc-a509-c22a6ff7ae0d)File(\EFI\refind\refind_x64.efi)

Booth Boot0004 and Boot005 have the same label, Salix, with one pointing to a boot loader for Slint and the other to Salix. So the real problem is when eliloconfig creates the Slackware directory on the ESP (EFI/Slackware) and copies the kernel, initrd and bootloader to it. If there is already a Slackware directory on the ESP, then it will be overwritten, but I do believe the installer warns about this.