Pinebook Pro boot from NVME - drop requirement for booting from uSD card

difdif · 01-01-2024, 05:42 PM

Following the Slackware Pinebook Pro installation instructions it has a section "Boot from NVME - drop requirement for booting from uSD card" noting that this is not yet fully worked out.

After a lot of trial and error I now have this working. First I wrote the Slackware bootloader to the SPI flash storage as described in my previous post. This allowed the SD card with the Slackware installer to boot. I formatted the NVME drive as a swap partition (/dev/nvme0n1p1) and a single data partition with ext4 format, /dev/nvme0n1p2 (maybe a separate /boot partition would work, I didn't try that). After completing the installation to the NVME storage the installer places the kernel and initrd on the SD card. The EMMC card was present in the computer but switched to disable it.

I copied the files from the boot partition on the SD card over to the /boot directory on the ext4 root partition. Editing /boot/extlinux/extlinux.conf on the NVME drive was where all the trial and error was then needed. I found the following entry worked and boots up from the NVME without the SD card present.

Code:

LABEL System
   MENU LABEL Boot NVME
   KERNEL /boot/Image-armv8
   INITRD /boot/initrd-armv8
   FDTDIR /boot/dtb
   APPEND earlyprintk rw console=tty1 root=/dev/nvme0n1p2 rootfstype=ext4 rootwait

Changing the PROMPT value to 90 gave enough time to see the menu and make a choice - it otherwise moves too quickly when running from the NVME drive, much faster then when running from the SD card.

Using root=LABEL=SLKroot in the APPEND line works too as the installer already labels up the partition - that would be more easily generalisable than the /dev/nvme0n1p2 version. Including console=tty1 ensured some errors were written to the screen to help understand what was happening before I got the APPEND options right. I'm not sure the combination above is optimal, but it works! Including rootfstype was crucial to the kernel being able to reach the root partition as it boots up.

drmozes · 01-02-2024, 03:41 AM

I appreciate it! I've revised the installation guide to include a reference to this post. You can adhere to the standard Boot Loader installation instructions and implement this alteration during the post-installation phase by choosing 'Shell' from the installer's exit menu.

glorsplitz · 01-03-2024, 06:24 PM

Quote:

Originally Posted by difdif

...boots up from the NVME without the SD card present.

AND with the system up and running, what happens when you insert sd card?

Is it accessible to mount? like a flash drive, floppy drive, whatever external mountable device.

mralk3 · 01-03-2024, 06:43 PM

Quote:

Originally Posted by glorsplitz

AND with the system up and running, what happens when you insert sd card?

Is it accessible to mount? like a flash drive, floppy drive, whatever external mountable device.

I have a mixed opinion about this feature.

1. If for some reason the internal /boot partition or / partition do not allow the system to boot, you still need a SD card to rescue the system. I find having the installer available on the SD card convenient for this reason. It is more convenient while running -current.

2. With that said, my NVMe drive boots the system extremely fast when compared to the SD card, as expected. I plan to boot from the NVMe drive once we have a 15.1 release of Aarch64. I also encrypted my NVMe drive with LUKS + LVM as described here: https://docs.slackware.com/slackware...ckware_aarch64

The SD card becomes a "floppy" or "flash" drive and can be mounted for whatever use case. I plan to store my LUKS encryption key on my SD card. If you wanted to do so, you could put another OS on the SD Card and dual boot, sort of. There are many other options. You may wish to leave the boot order within u-boot alone so that it looks for the SD card first and contains a rescue image.

difdif · 01-04-2024, 02:30 PM

I have also found that the built in SD card reader continues to work normally and an SD card can be mounted there.

Yes, the case of the NVME drive holding the system but not booting can be managed if you keep the bootable SD card available. I used this method multiple times while I worked out the contents of extlinux.conf on the NVME drive.

One thing I found was you can get the case that the NVME drive takes priority over the internal SD card reader. If the internal drive doesn't boot properly that can leave you in a mess. There's probably U-Boot commands to handle this. But I also found that putting the Slackware installer SD card in an external card reader plugged into the USB3 port (left side of PinebookPro) can overcome this. The external USB card reader was found and enabled booting ahead of the NVME and internal SD card reader. This was using the standard Slackware U-boot install copied to the SPI flash where I expect this order was configured, but I didn't look further at that.

glorsplitz · 01-04-2024, 07:31 PM

Quote:

Originally Posted by difdif

But I also found that putting the Slackware installer SD card in an external card reader plugged into the USB3 port (left side of PinebookPro) can overcome this.

It's been little while since I installed, pretty sure usb3 port/card reader is what I did along with installing towboot.

I'm thinking I'll lose being able to boot if I flash to new slackware u-boot.

glorsplitz · 01-04-2024, 07:38 PM

Thank you for replying, you both say it can be done, I'd like to see it done.

Like what does messages say after sd card is inserted, what /dev/??? sd card is identified as.

x86_64 slackware, messages shows the following when plugging in sd card in card reader

Quote:

Jan 4 19:41:50 kernel: scsi 6:0:0:0: Direct-Access Generic STORAGE DEVICE 0902 PQ: 0 ANSI: 6
Jan 4 19:41:51 kernel: sd 6:0:0:0: [sdc] 124735488 512-byte logical blocks: (63.9 GB/59.5 GiB)
Jan 4 19:41:51 kernel: sd 6:0:0:0: [sdc] Write Protect is off
Jan 4 19:41:51 kernel: sd 6:0:0:0: [sdc] Write cache: disabled, read cache: enabled, doesn't support DPO or FUA
Jan 4 19:41:51 kernel: sd 6:0:0:0: [sdc] Attached SCSI removable disk

plugging sd card in sd card slot on pbp messages says

Quote:

Jan 4 19:45:28 kernel: mmc_host mmc1: Bus speed (slot 0) = 300000Hz (slot req 300000Hz, actual 300000HZ div = 0)
Jan 4 19:45:29 kernel: mmc_host mmc1: Bus speed (slot 0) = 200000Hz (slot req 200000Hz, actual 200000HZ div = 0)
Jan 4 19:45:30 kernel: mmc_host mmc1: Bus speed (slot 0) = 100000Hz (slot req 100000Hz, actual 100000HZ div = 0)

and I don't see any /dev/ device

drmozes · 01-05-2024, 05:57 AM

Quote:

Originally Posted by glorsplitz

x86_64 slackware, messages shows the following when plugging in sd card in card reader

The naming difference arises from how the system assigns device names based on the interface they are connected through (USB, PCI, SATA, etc.).
USB-connected devices are identified as generic 'mass storage devices' irrespective of the physical media they're hosting.

Quote:

plugging sd card in sd card slot on pbp messages says

and I don't see any /dev/ device

As it's a direct MMC device, not connected through a USB interface it'll be /dev/mmcblk1 or /dev/mmcblk0

Code:

root@dastardly:~# lsblk -M /dev/mmcblk*
NAME        MAJ:MIN RM  SIZE RO TYPE MOUNTPOINTS
mmcblk1     179:0    0   58G  0 disk
└─mmcblk1p1 179:1    0  3.7G  0 part /boot
mmcblk1p1   179:1    0  3.7G  0 part /boot

difdif · 01-05-2024, 05:05 PM

There's more to this than it seemed. Some cards are not readable in the internal SD slot but work fine in an external reader. Another one works perfectly in the internal reader.

The one that now does not read is the one I installed Slackware from - so I know this did work in the internal reader. There's some software issue here now the machine has booted off the installed system.

Details are below - if anyone has some ideas what I could try to help explore this please let me know.

Some searching which is aligned with the dmesg output below is that there's a speed set for the SD card. Working cards give messages such as "dwmmc_rockchip fe320000.mmc: Successfully tuned phase to 220". Cards that won't read don't have this message. The posts linked below suggest this might be about a speed set in the device tree - is the tree used during installation different to the one booting after install? These posts seem to point towards this as an issue - probably nothing to do with the original subject of this post!

eMMC doesn't work after kernel start - Does this apply to the internal SD too on the mmcblk1 device?

Kernel doesn't work with eMMC after start

Here's some output for cards that do/don't work, in the internal and an external SD card reader.

(1) Slackware installer SD card in internal reader - not readable. dmesg output:

Code:

[  197.280320] mmc_host mmc1: Bus speed (slot 0) = 400000Hz (slot req 400000Hz, actual 400000HZ div = 0)
[  197.417110] mmc_host mmc1: Bus speed (slot 0) = 50000000Hz (slot req 50000000Hz, actual 50000000HZ div = 0)
[  197.417780] mmc1: new high speed SDHC card at address 0001
[  197.418868] mmcblk1: mmc1:0001 00000 29.2 GiB 
[  197.429268]  mmcblk1: p1 p2
[  197.780018] mmc_host mmc1: Bus speed (slot 0) = 400000Hz (slot req 400000Hz, actual 400000HZ div = 0)
[  197.816905] mmc_host mmc1: Bus speed (slot 0) = 50000000Hz (slot req 50000000Hz, actual 50000000HZ div = 0)
[  197.817843] I/O error, dev mmcblk1, sector 9764866 op 0x0:(READ) flags 0x80700 phys_seg 1 prio class 2
[  198.053277] I/O error, dev mmcblk1, sector 6176 op 0x0:(READ) flags 0x80700 phys_seg 1 prio class 2
[  198.059554] I/O error, dev mmcblk1, sector 9764866 op 0x0:(READ) flags 0x0 phys_seg 1 prio class 2
[  198.061382] Buffer I/O error on dev mmcblk1p2, logical block 0, async page read
[  198.075327] mmc_host mmc1: Bus speed (slot 0) = 400000Hz (slot req 400000Hz, actual 400000HZ div = 0)
[  198.109004] mmc_host mmc1: Bus speed (slot 0) = 50000000Hz (slot req 50000000Hz, actual 50000000HZ div = 0)
[  198.109432] I/O error, dev mmcblk1, sector 6176 op 0x0:(READ) flags 0x0 phys_seg 1 prio class 2
[  198.110236] Buffer I/O error on dev mmcblk1p1, logical block 8, async page read
[  198.229110] I/O error, dev mmcblk1, sector 0 op 0x0:(READ) flags 0x80700 phys_seg 1 prio class 2
[  198.348672] I/O error, dev mmcblk1, sector 0 op 0x0:(READ) flags 0x0 phys_seg 1 prio class 2
[  198.350538] Buffer I/O error on dev mmcblk1, logical block 0, async page read
[  198.465657] I/O error, dev mmcblk1, sector 0 op 0x0:(READ) flags 0x0 phys_seg 1 prio class 2
[  198.467805] Buffer I/O error on dev mmcblk1, logical block 0, async page read
[  198.470138] I/O error, dev mmcblk1, sector 0 op 0x0:(READ) flags 0x0 phys_seg 1 prio class 2
[  198.472389] Buffer I/O error on dev mmcblk1, logical block 0, async page read

Chcking using lsblk

Code:

NAME        MAJ:MIN RM   SIZE RO TYPE MOUNTPOINTS
mtdblock0    31:0    0    16M  0 disk 
mmcblk1     179:0    0  29.2G  0 disk 
├─mmcblk1p1 179:1    0   3.7G  0 part 
└─mmcblk1p2 179:2    0   5.3G  0 part 
nvme0n1     259:0    0 931.5G  0 disk 
├─nvme0n1p1 259:1    0     4G  0 part [SWAP]
└─nvme0n1p2 259:2    0 927.5G  0 part /

Card details via fdisk /dev/mmcblk1

Code:

Disk /dev/mmcblk1: 29.24 GiB, 31393316864 bytes, 61315072 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: dos
Disk identifier: 0xfc1d096e

Device         Boot   Start      End  Sectors  Size Id Type
/dev/mmcblk1p1 *       6144  7812499  7806356  3.7G 83 Linux
/dev/mmcblk1p2      9764864 20971519 11206656  5.3G 83 Linux

(2) Slackware installer SD card in external USB reader. Perfectly readable, dmesg gives

Code:

[ 1266.559649] usb 4-1.1: new high-speed USB device number 4 using ehci-platform
[ 1266.845608] usb-storage 4-1.1:1.0: USB Mass Storage device detected
[ 1266.848125] scsi host0: usb-storage 4-1.1:1.0
[ 1267.872447] scsi 0:0:0:0: Direct-Access     Generic  Compact Flash    0.00 PQ: 0 ANSI: 2
[ 1267.873624] sd 0:0:0:0: Attached scsi generic sg0 type 0
[ 1267.874668] sd 0:0:0:0: [sda] Media removed, stopped polling
[ 1267.876236] sd 0:0:0:0: [sda] Attached SCSI removable disk
[ 1268.602935] scsi 0:0:0:1: Direct-Access     Generic  SD/MMC           0.00 PQ: 0 ANSI: 2
[ 1268.603813] sd 0:0:0:1: Attached scsi generic sg1 type 0
[ 1268.605628] sd 0:0:0:1: [sdb] Media removed, stopped polling
[ 1268.606829] sd 0:0:0:1: [sdb] Attached SCSI removable disk
[ 1268.606917] scsi 0:0:0:2: Direct-Access     Generic  microSD          0.00 PQ: 0 ANSI: 2
[ 1268.607526] sd 0:0:0:2: Attached scsi generic sg2 type 0
[ 1268.615138] sd 0:0:0:2: [sdc] 61315072 512-byte logical blocks: (31.4 GB/29.2 GiB)
[ 1268.615748] scsi 0:0:0:3: Direct-Access     Generic  MS/MS-PRO        0.00 PQ: 0 ANSI: 2
[ 1268.616442] sd 0:0:0:2: [sdc] Write Protect is off
[ 1268.616457] sd 0:0:0:2: [sdc] Mode Sense: 03 00 00 00
[ 1268.617133] sd 0:0:0:2: [sdc] No Caching mode page found
[ 1268.617143] sd 0:0:0:2: [sdc] Assuming drive cache: write through
[ 1268.617343] sd 0:0:0:3: Attached scsi generic sg3 type 0
[ 1268.619171] sd 0:0:0:3: [sdd] Media removed, stopped polling
[ 1268.622638] scsi 0:0:0:4: Direct-Access     Generic  SM/xD-Picture    0.00 PQ: 0 ANSI: 2
[ 1268.627497] sd 0:0:0:3: [sdd] Attached SCSI removable disk
[ 1268.630198] sd 0:0:0:4: Attached scsi generic sg4 type 0
[ 1268.636538]  sdc: sdc1 sdc2
[ 1268.638807] sd 0:0:0:4: [sde] Media removed, stopped polling
[ 1268.641186] sd 0:0:0:4: [sde] Attached SCSI removable disk
[ 1268.649730] sd 0:0:0:2: [sdc] Attached SCSI removable disk

lsblk gives:

Code:

NAME        MAJ:MIN RM   SIZE RO TYPE MOUNTPOINTS
sda           8:0    1     0B  0 disk 
sdb           8:16   1     0B  0 disk 
sdc           8:32   1  29.2G  0 disk 
├─sdc1        8:33   1   3.7G  0 part 
└─sdc2        8:34   1   5.3G  0 part 
sdd           8:48   1     0B  0 disk 
sde           8:64   1     0B  0 disk 
mtdblock0    31:0    0    16M  0 disk 
nvme0n1     259:0    0 931.5G  0 disk 
├─nvme0n1p1 259:1    0     4G  0 part [SWAP]
└─nvme0n1p2 259:2    0 927.5G  0 part /

fdisk /dev/sdc output:

Code:

Disk /dev/sdc: 29.24 GiB, 31393316864 bytes, 61315072 sectors
Disk model: microSD         
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: dos
Disk identifier: 0xfc1d096e

Device     Boot   Start      End  Sectors  Size Id Type
/dev/sdc1  *       6144  7812499  7806356  3.7G 83 Linux
/dev/sdc2       9764864 20971519 11206656  5.3G 83 Linux

(3) dmesg for an alternative SD card in the internal reader which reads perfectly:

Code:

[  833.875491] mmc_host mmc1: Bus speed (slot 0) = 400000Hz (slot req 400000Hz, actual 400000HZ div = 0)
[  833.983970] mmc_host mmc1: Bus speed (slot 0) = 100000000Hz (slot req 100000000Hz, actual 100000000HZ div = 0)
[  834.347404] dwmmc_rockchip fe320000.mmc: Successfully tuned phase to 220
[  834.347440] mmc1: new ultra high speed SDR50 SDHC card at address 0001
[  834.348537] mmcblk1: mmc1:0001 ASTC 14.6 GiB 
[  834.350256]  mmcblk1: unable to read partition table
[  834.443311] mmc1: card 0001 removed
[  835.071299] mmc_host mmc1: Bus speed (slot 0) = 400000Hz (slot req 400000Hz, actual 400000HZ div = 0)
[  835.183853] mmc_host mmc1: Bus speed (slot 0) = 100000000Hz (slot req 100000000Hz, actual 100000000HZ div = 0)
[  835.552347] dwmmc_rockchip fe320000.mmc: Successfully tuned phase to 220
[  835.552392] mmc1: new ultra high speed SDR50 SDHC card at address 0001
[  835.553608] mmcblk1: mmc1:0001 ASTC 14.6 GiB 
[  835.556704]  mmcblk1: p1

lsblk output:

Code:

NAME        MAJ:MIN RM   SIZE RO TYPE MOUNTPOINTS
mtdblock0    31:0    0    16M  0 disk 
mmcblk1     179:0    0  14.6G  0 disk 
└─mmcblk1p1 179:1    0  14.4G  0 part 
nvme0n1     259:0    0 931.5G  0 disk 
├─nvme0n1p1 259:1    0     4G  0 part [SWAP]
└─nvme0n1p2 259:2    0 927.5G  0 part /

Card details from fdisk /dev/mmcblk1

Code:

Disk /dev/mmcblk1: 14.57 GiB, 15640559616 bytes, 30547968 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: dos
Disk identifier: 0x10fa9616

Device         Boot Start      End  Sectors  Size Id Type
/dev/mmcblk1p1      32768 30212096 30179329 14.4G 83 Linux

difdif · 01-06-2024, 06:11 PM

After lots of reading and trying things out I think I've worked out what's happening and found a work-around.

SD cards have a range of operating frequencies depending on their capabilities.

If you boot from the SD the Rockchip hardware tunes to the capability of the card in place and it all works. If no SD card is present (as when booting using the NVME drive) the current behaviour is to set operation of the SD card socket for the highest frequency available in the host hardware - and this then seems to be locked in once the machine is running. The hardware should recognise a card plugged in later and if necessary drop frequency to match, but it does not. Hence some cards work when used as the boot drive, but fail if plugged in later following NVME boot. Other cards that can handle the maximum frequency of the socket work fine both ways. Some clues about all this are here.

This problem can be avoided by setting a lower maximum operating frequency for the SD socket. This can be done my modifying the device tree using this (as root):

Code:

fdtput /boot/dtb-6.1.67/rockchip/rk3399-pinebook-pro.dtb mmc1 max-frequency 12500000

That reduces the SD operating frequency to 12.5MHz, down from its standard value of 50MHz. After a reboot the SD card that previously failed to mount then works perfectly. Probably there's a performance penalty for data transfer to the SD card, but I didn't notice anything major. It might be possible to go higher (e.g. 25MHz) without causing the card to fail - I've not tried that.

The real solution would be for the RK3399 to adapt frequency properly, but the above is a work-around which allows more (all?) SD cards to mount even if they are not particularly high spec cards. Rather than modifying the device tree it should also be possible to change this frequency from uboot (extlinux.conf file) using something like

Code:

fdt set /soc/apb@d0000000/mmc@fe320000 max-frequency <0xBEBC20>

But I've not worked that out yet.

glorsplitz · 01-06-2024, 07:47 PM

@drmozes

I was not posting about what 'mass storage devices', I was posting what messages acknowledges when something system related occurs.

I was also indicating there was no new /dev/ device of any kind after sd card was inserted.

I will try to follow up difdif investigations soon.

enorbet · 01-19-2024, 01:08 AM

I'm curious as to how you guys are connecting M.2 NVME drives.

It's pretty obvious how one would mount one drive with a simple PCIe to NVME sled or dock. In my case my Pine64 is used for NAS so I would need multiple M.2 drives and slots. I'm using an actual PSU instead of the little wall wart supply but have doubts about the current ability to experience any real bandwidth gains should I go to a 4 x M.2 sled or dock. Any comments?

mralk3 · 01-19-2024, 10:39 AM

The pinebook pro has an addon that enables a NVMe drive via a M.2 slot. A small ribbon cable and a mounting board interface the drive to the main board.

The Rockpro64 PCIe lane most likely doesn't have enough bandwidth to use more that one or two NVMe drives. I may be wrong about that.

Most users drop in a SATA or Network card in the PCIe slot of the Rockpro64. Pine64 does sell a NVMe card, but it only has input for one drive. The pine64 wiki hardware compatibility page shows only two tested nvme drives.