[SOLVED] Difficulty seeing NVMe

andygoth · 02-04-2021, 12:00 PM

Hi, trying to install Slackware64-current on an ASUS VivoBook 15 which uses NVMe, but the installer can't see it. It shows up in the output of lspci, but nowhere else that I can find.

Interestingly, I succeeded in installing Slackware64-current on an ASUS M570DD, which as far as I can tell is basically the same thing as the VivoBook 15. NVMe was no trouble at all.

Even more interestingly, CloneZilla Live 20201102 is able to access NVMe on the ASUS VivoBook 15.

What's the difference between Slackware64-current and CloneZilla Live? Why can the latter see NVMe but not the former on this laptop? Slackware64-current definitely has NVMe support:

Code:

CONFIG_NVME_CORE=y
CONFIG_BLK_DEV_NVME=y
CONFIG_NVME_MULTIPATH=y
CONFIG_NVME_HWMON=y
CONFIG_NVME_FABRICS=m
CONFIG_NVME_RDMA=m
CONFIG_NVME_FC=m
CONFIG_NVME_TCP=m
CONFIG_NVME_TARGET=m
CONFIG_NVME_TARGET_PASSTHRU=y
CONFIG_NVME_TARGET_LOOP=m
CONFIG_NVME_TARGET_RDMA=m
CONFIG_NVME_TARGET_FC=m
CONFIG_NVME_TARGET_FCLOOP=m
CONFIG_NVME_TARGET_TCP=m

What more is there?

Hmm, just took a look around, and I see there's a newer version of CloneZilla Live available (20210127). I'll give it a try to check if it also can access my NVMe. If not, then maybe my issue is due to some recent kernel bug or incompatibility.

My ChangeLog.txt's date is Tue Feb 2 20:16:43 UTC 2021.

eduardr · 02-04-2021, 05:27 PM

Have never had issues seeing NVMe's on servers (Intel & Samsung) or installing Slackware on them. Maybe some kind of laptop specific issue not sure.

A few things to check:

- Make sure BIOS config for the laptop is correct, NVMe drive enabled

- Check if the device shows up, for ex. as /dev/nvme0n1

- Make sure to partition and format the NVMe before running the Slackware installer. For ex:

Code:

$ gdisk /dev/nvme0n1

(or whatever the appropriate nvme dev name is)

After correct partitioning you should see something like the following:

Code:

/dev/nvme0    
/dev/nvme0n1  
/dev/nvme0n1p1
/dev/nvme0n1p2

Emerson · 02-04-2021, 05:32 PM

I have noticed for SoC platforms it is mandatory to turn on GPIO options in kernel or many things won't work.

When you boot with Clonezilla, do you see GPIO modules loaded?

enorbet · 02-05-2021, 09:40 PM

I've seen bios/uefi image upgrades as late as December 2020 for that machine. It's probably wise to be certain you have the newest.

andygoth · 02-07-2021, 11:26 AM

Quote:

Originally Posted by Emerson

I have noticed for SoC platforms it is mandatory to turn on GPIO options in kernel or many things won't work.

When you boot with Clonezilla, do you see GPIO modules loaded?

In Clonezilla, there are no GPIO modules loaded, no mention of GPIO in dmesg. The Clonezilla initrd has plenty of GPIO stuff, including specific to Asus computers.

The newer Clonezilla works just as well as the old one, so there's not been a kernel regression. The problem is isolated to Slackware itself. Apparently there's something it's not doing at boot time that's needed in order to enable NVMe.

Clonezilla does load these modules with Asus in the name:

asus_nb_wmi
asus_wmi
pegasus (haha)

None of these modules are present in the Slackware installer.

andygoth · 02-07-2021, 08:25 PM

Quote:

Originally Posted by eduardr

Have never had issues seeing NVMe's on servers (Intel & Samsung) or installing Slackware on them. Maybe some kind of laptop specific issue not sure.

A few things to check:

- Make sure BIOS config for the laptop is correct, NVMe drive enabled

- Check if the device shows up, for ex. as /dev/nvme0n1

- Make sure to partition and format the NVMe before running the Slackware installer. For ex:

Code:

$ gdisk /dev/nvme0n1

(or whatever the appropriate nvme dev name is)

After correct partitioning you should see something like the following:

Code:

/dev/nvme0    
/dev/nvme0n1  
/dev/nvme0n1p1
/dev/nvme0n1p2

NVMe is definitely enabled. I'm able to boot Clonezilla and see the NVMe. I'm also able to boot Windows which came preinstalled. Clonezilla sees NVMe just fine, and it shows up in both /dev and lsblk. Yet, Slackware does not, so naturally I can't partition it either.

andygoth · 02-07-2021, 08:27 PM

Experimenting to figure out how best to add a module to the installer...

First, I got the latest slackware64-current to be the baseline for my local customizations. This was a minor update to the mirror I already maintain for my home network. I have scripts for all this, but in this post I'll just spell out the underlying commands for parsimony.

Code:

cd /home/ftp
rsync -havyP --delete-delay --no-o --no-g --no-l -L --exclude source/ rsync://mirror.slackbuilds.org/slackware/slackware64-current/ slackware64

Next, I updated my host system to the latest, giving me easy access to the right version of the kernel modules. I already have file://home/ftp/slackware64/ in /etc/slackpkg/mirrors. This particular update didn't involve installing or removing any packages, though I checked anyway.

Code:

slackpkg update
slackpkg install-new
slackpkg upgrade-all
slackpkg clean-system

I could have rebooted at this point to use the new kernel, but it wasn't really necessary since I can just tell depmod what kernel version will be used.

I proceeded to unpack the initrd.img file, to which I'll be adding modules.

Code:

mkdir /tmp/initrd-tree
cd /tmp/initrd-tree
xzcat /home/ftp/slackware64/EFI/BOOT/initrd.img | cpio -i

To make things simpler for myself, I then decompress all the *.ko.xz files. I have no clue why they're compressed in the first place, since the entire initrd will be compressed anyway.

Code:

find -name '*.ko.xz' | xargs unxz

Then, to add the Asus modules (not Pegasus) and their dependencies:

Code:

sed -n '/[^:]*\<asus\>.*: /s///;T;s/ /\n/g;p' /lib/modules/5.10.13/modules.dep |
    sort -u | tar -C /lib/modules/5.10.13 -T - -c | tar -C lib/modules/5.10.13 -x -v
zcat /home/ftp/slackware64/kernels/huge.s/System.map.gz | depmod -F /dev/stdin -b . 5.10.13

This complains about there being no modules.order or modules.builtin, but I guess they're not needed just for running the installer. Also I notice that all the other kernel modules are compressed with xz, but that seems like a waste since the entire initrd.img is going to be compressed anyhow.

With that, I recreated initrd.img with the additional modules:

Code:

find | cpio -o -H newc | lzma > /home/ftp/slackware64/isolinux/initrd.img
cp /home/ftp/slackware64/{isolinux/initrd.img,EFI/BOOT}

Maybe they won't automatically get loaded at boot, but who cares? I'll just modprobe them. Anyway, with all of the above done, the next step is to create the disc image. I used the procedure in isolinux/README.TXT, slightly modified. Because my disc burner seems to be having an issue finalizing DVDs, I instead wrote this image to an SD card.

Code:

xorriso -as mkisofs \
  -iso-level 3 \
  -full-iso9660-filenames \
  -R -J -A "Slackware Install" \
  -hide-rr-moved \
  -v -d -N \
  -eltorito-boot isolinux/isolinux.bin \
  -eltorito-catalog isolinux/boot.cat \
  -no-emul-boot -boot-load-size 4 -boot-info-table \
  -isohybrid-mbr /usr/share/syslinux/isohdpfx.bin \
  -eltorito-alt-boot \
  -e isolinux/efiboot.img \
  -no-emul-boot -isohybrid-gpt-basdat \
  -volid SlackDVD \
  -output /dev/mmcblk0 \
  /home/ftp/slackware64

I then booted the installer and logged in as root. It automatically loaded asus-wmi and asus-nb-wmi. Alas, this was not enough for NVMe to work.

I'm considering comparing the contents of dmesg between Clonezilla and Slackware to see what's different. This could get annoying to do with no network and no common storage. I will have to hunt up a USB memory stick for each one to write to.

andygoth · 02-19-2021, 11:58 AM

I finally got a chance to compare the CloneZilla and Slackware dmesg outputs. The only USB memory stick I could find turned out to be totally dead, so I scrounged up one of those weird combination USB/SD memory cards which did the job.

Here's CloneZilla's dmesg, in part:

Code:

i2c i2c-0: 2/8 memory slots populated (from DMI)
i2c i2c-0: Systems with more than 4 memory slots not supported yet, not instantiating SPD
intel-lpss 0000:00:15.0: enabling device (0004 -> 0006)
idma64 idma64.0: Found Intel integrated DMA 64-bit 
nvme nvme0: pci function 10000:e1:00.0
pcieport 10000:e0:1d.0: can't derive routing for PCI INT A
nvme 10000:e1:00.0: PCI INT A: not connected
nvme nvme0: 8/0/0 default/read/poll queues
 nvme0n1: p1 p2 p3 p4

Here's Slackware:

Code:

i2c i2c-0: 2/8 memory slots populated (from DMI)
i2c i2c-0: Systems with more than 4 memory slots not supported yet, not instantiating SPD
drm_kms_helper: Unknown symbol fb_sys_write (err -2)
drm_kms_helper: Unknown symbol sys_imageblit (err -2)
drm_kms_helper: Unknown symbol sys_fillrect (err -2)
drm_kms_helper: Unknown symbol sys_copyarea (err -2)
drm_kms_helper: Unknown symbol fb_sys_read (err -2)

The drm_kms_helper errors are entirely unrelated. I only put them there to show the gaping void after initializing i2c. The Slackware kernel messages don't mention intel-lpss, idma64, or nvme. I know Slackware includes nvme support, but that doesn't help when the storage device isn't accessible in the first place.

Procedure version 2.0:

Code:

# Extract Pat's initrd
mkdir /tmp/initrd-tree
cd /tmp/initrd-tree
xzcat /home/ftp/slackware64/EFI/BOOT/initrd.img | cpio -i

# Decompress *.ko.xz
find -name '*.ko.xz' -exec unxz {} +

# Import modules with lpss and idma64 in their names, plus everything they require
sed -En '/([^:]*\<(lpss|idma64)\>.*): /s//\1 /;T;s/ /\n/g;p' /lib/modules/5.10.17/modules.dep |
    sort -u | tar -C /lib/modules/5.10.17 -T - -c | tar -C lib/modules/5.10.17 -x -v
zcat /home/ftp/slackware64/kernels/huge.s/System.map.gz | depmod -F /dev/stdin -b . 5.10.17

# Build the new initrd
find | cpio -o -H newc | lzma > /home/ftp/slackware64/isolinux/initrd.img
cp /home/ftp/slackware64/{isolinux/initrd.img,EFI/BOOT}

# "Burn" to SD card
xorriso -as mkisofs \
  -iso-level 3 \
  -full-iso9660-filenames \
  -R -J -A "Slackware Install" \
  -hide-rr-moved \
  -v -d -N \
  -eltorito-boot isolinux/isolinux.bin \
  -eltorito-catalog isolinux/boot.cat \
  -no-emul-boot -boot-load-size 4 -boot-info-table \
  -isohybrid-mbr /usr/share/syslinux/isohdpfx.bin \
  -eltorito-alt-boot \
  -e isolinux/efiboot.img \
  -no-emul-boot -isohybrid-gpt-basdat \
  -volid SlackDVD \
  -output /dev/mmcblk0 \
  /home/ftp/slackware64

This wasn't enough to get my nvme to show up, though I did confirm that lpss and idma64 are now being initialized the same in Slackware as in Clonezilla. I'll write a follow-up post as I continue to investigate.

andygoth · 02-19-2021, 01:36 PM

This round, I tried to use a big hammer and see if I got results. I went ahead and put every *.ko file I have into the initrd.

Code:

# Import all modules
(cd /lib/modules/5.10.17; find -name '*.ko') | tar -C /lib/modules/5.10.17 -T - -c |
    tar -C lib/modules/5.10.17 -x -v
zcat /home/ftp/slackware64/kernels/huge.s/System.map.gz | depmod -F /dev/stdin -b . 5.10.17
# Build the new initrd
# [...]
# "Burn" to SD card
# [...]

Wonder of wonders, that did it! Now I have /dev/nvme*.

Next, I will try to track down which modules are actually required to make this work, since we should probably include them in the installer by default.

bassmadrigal · 02-19-2021, 01:52 PM

Can you check lsmod in the installer and see what's loaded? That might give you a starting point that you can then use to tweak your initrd.

eduardr · 02-19-2021, 02:29 PM

Can try looking at the output of `lsmod` and comparing with output of lsmod when something was missing. That should help id which modules were additionally loaded when it does work.

andygoth · 02-19-2021, 09:36 PM

I compared the stock Slackware dmesg with my Christmas Tree-edition Slackware (i.e. includes every module), and the main thing that jumped out at me was vmd.

modules.dep reveals that vmd.ko doesn't depend on anything, so the command to put it into the initrd is much simpler than any of the above.

Tested it, and yup! vmd.ko is the only thing I needed to add in order to get /dev/nvme*.

I'm writing a much longer procedure to show exactly how I set up the laptop. I'll post it later (when everything possible is working).

dr.s · 08-12-2021, 03:06 PM

Quote:

Originally Posted by andygoth

I compared the stock Slackware dmesg with my Christmas Tree-edition Slackware (i.e. includes every module), and the main thing that jumped out at me was vmd.

modules.dep reveals that vmd.ko doesn't depend on anything, so the command to put it into the initrd is much simpler than any of the above.

Tested it, and yup! vmd.ko is the only thing I needed to add in order to get /dev/nvme*.

I'm writing a much longer procedure to show exactly how I set up the laptop. I'll post it later (when everything possible is working).

Thanks for this. The latest -current installer had no problems seeing my PCIe SSD but rebooting with the huge kernel failed with kernel panics, solved by recompiling huge with nvme target and vmd support built in.

Code:

CONFIG_NVME_CORE=y
...
CONFIG_NVME_TARGET=y
CONFIG_NVME_TARGET_PASSTHRU=y
CONFIG_NVME_TARGET_LOOP=y
...
CONFIG_VMD=y

Aeterna · 08-13-2021, 12:06 PM

Quote:

Originally Posted by dr.s

Thanks for this. The latest -current installer had no problems seeing my PCIe SSD but rebooting with the huge kernel failed with kernel panics, solved by recompiling huge with nvme target and vmd support built in.

Code:

CONFIG_NVME_CORE=y
...
CONFIG_NVME_TARGET=y
CONFIG_NVME_TARGET_PASSTHRU=y
CONFIG_NVME_TARGET_LOOP=y
...
CONFIG_VMD=y

Very interesting. I installed Slackware-current with (then) latest iso on NVMe

Quote:

cat config-generic-5.12.12.x64 | grep VMD
CONFIG_VMD=m
CONFIG_MVMDIO=m
catt config-huge-5.12.12.x64 | grep VMD
CONFIG_VMD=m
CONFIG_MVMDIO=m

without any issues on the laptop as seen above from kernel config.

dr.s · 08-13-2021, 12:39 PM

Quote:

Originally Posted by Aeterna

Very interesting. I installed Slackware-current with (then) latest iso on NVMe

without any issues on the laptop as seen above from kernel config.

Yeah got some inconsistent results here as well.
The unmodified huge kernel had no problems booting up an older laptop with an nvme/PCIe ssd
but would panic on the new laptop.
I've tested with 5.13.x and 5.14-rc, haven't tried 5.12.x though.