Kernel does not recognize MMC controller... but u-boot does!

I'm trying to get a custom board to boot from an SD card, and it's very close to working. U-boot is able to see the SD card, it loads the kernel, and the kernel starts booting. However, when it tries to mount the filesystem on the SD card, it fails to find the /dev/mmcblk0 device.

There's a lot of content related to that on the internet, but I can confirm that the exact same SD card, when inserted into a dev board (i.e., same SoC, similar reference circuit for the SD card, etc.), CAN properly boot.

So I'm struggling to understand what the problem is. The custom board's circuit between the SoC and the SD card slot must be ok because u-boot has no problem using it. The kernel must contain all the right drivers for the SoC's MMC controller because the dev board can use the same one successfully. And the device tree must be ok because the dev board and the custom board have the same SoC. However, on the custom board, the kernel loads only up until it tries to find a file system.

I've tried disabling the card-detect for the MMC in the device tree (i.e., by using "broken-cd"), but that did not change the behavior. I also tried setting the kernel boot parameter "root=/dev/mmcblk0" to mmcblk1 and mmcblk2, just in case the MMCs were being enumerated in a way I didn't expect. Still nothing.

I'm stumped here, because it seems like, if the dev board works and the custom board doesn't, then it's probably a problem with the circuit. But u-boot on the custom board finds and uses the MMC just fine.

If anyone out there is deeply familiar with u-boot and how it handles MMC controllers, please let me know. There's got to be some difference between u-boot's handling of them and the Linux kernel's, but I'm not familiar enough with u-boot's innards.

For reference, it's a sun8i SoC, kernel 4.20, and u-boot 2019.04.

Thanks!

Hi,

A few years ago I spent some time experimenting with installing Linux onto a range of tablets (All 64bit Bay Trail/Cherry Trail/Celeron).
Initial problems with some of them was that the eMMC was not always detected. In every case, this particular problem was (eventually) solved by upgrading the installation kernel to the newest available at the time. It took a wait of about six months before one particular tablet was supported properly.

In every case, Grub2 could see the eMMC.

I would guess that your hardware is perfectly O.K. but you are missing specific kernel support for something important..

Can you boot your board from an external device that has a newer kernel? This would at least show which kernel versions work with your board.

Good luck.

Bodge99

this particular problem was (eventually) solved by upgrading the installation kernel to the newest available at the time.

In this case, though, the same build of the kernel and the same device tree behave differently on two different boards with the same SoC. Could that still be a potential solution?

That's what's so weird. I know the kernel contains all the dependencies for the MMC to work correctly, because it DOES work on this SoC, just on a different board. And I know the circuit on the custom board must be OK because u-boot can use it. So I don't know what else could go wrong.

Hi,

I've only recently started playing with Arm type kit, so I think that you are way ahead of me..

I've been collecting docs and further info from any source that I can as I would like to learn this stuff as thoroughly as possible.

For anyone reading this who is just starting off or just for general interest...
One doc that I've got gives the following in its general guidance section:

Device Tree overlays Most in-circuit and GPIO based interfaces (SPI, I2C, I2S, UART, …) don’t have a mechanism for detecting and identifying devices connected to the bus, so the Linux kernel has to be told explicitly about the device and its configuration details. While a Device Tree is a way of describing hardware configuration to the kernel, Device Tree overlays are a way for modifying the DT in order to provide the kernel and kernel drivers with details about external devices or to activate interfaces disabled by default. From the mainline Linux kernel perspective all unused in-circuit type interfaces should be disabled by default and all pins muxed with those interfaces should be available as standard GPIOs.

I think that you're correct and you'll find that something required is missing in your specific Device Tree overlay that your particular board needs.

I'll see if I can find anything relevant in my docs..

Do you have any specific details of what is exactly on the board?

Bodge99

Originally Posted by MALDATA However, when it tries to mount the filesystem on the SD card, it fails to find the /dev/mmcblk0 device.

If you have a boot issue, then posting the boot log is expected when asking for help instead of only providing your interpretation of events.

Originally Posted by MALDATA I'm stumped here, because it seems like, if the dev board works and the custom board doesn't, then it's probably a problem with the circuit.

That's seriously flawed logic.
These hardware devices require software and configuration.
You're comparing a working system (hardware plus software plus a configuration) with another system that fails a particular task.
How can you assert one part of a system is at fault with such little information?

Originally Posted by MALDATA If anyone out there is deeply familiar with u-boot and how it handles MMC controllers, please let me know. There's got to be some difference between u-boot's handling of them and the Linux kernel's, but I'm not familiar enough with u-boot's innards.

Your focus on U-Boot makes no sense at all.
The boot failure occurs in the kernel, so that is where to look.
Apparently you have not reviewed the boot log other than at the end when the boot fails.
The kernel "fails to find the /dev/mmcblk0 device" probably because the mmc device driver failed its probe/initialization, and you haven't bothered to look for that salient error notification.

Regards

Originally Posted by MALDATA In this case, though, the same build of the kernel and the same device tree behave differently on two different boards with the same SoC.

That is the expected result.
A Device Tree is meant to describe just one board, i.e. the specific configuration of one board.
When you have a similar but different (circuit-wise) board, then that other board needs it own Device Tree file.
That's why each Device Tree file (i.e. .dts or .dtb) is named for a board and not a SoC.

Using the same SoC on two boards does not permit you to reuse the same DT for both boards unless they are identical electronically.

Your focus on U-Boot makes no sense at all. The boot failure occurs in the kernel, so that is where to look.

Yes, but for my own understanding, I was hoping to get more insight into how the two mechanisms work. Obviously u-boot has some way of accessing hardware without the kernel or the device tree. And the fact that u-boot would work while the kernel didn't is curious to me. I would love to understand this more, so I came here to find someone to learn from, and to engage in a dialog about it so I can sort out the issue AND learn a little more.

You sound like you have some experience with this technology, so I appreciate your thoughts.

Apparently you have not reviewed the boot log other than at the end when the boot fails.

You are correct, as I am assisting in the debugging of the device, and had not yet been provided with a complete boot log myself.

How can you assert one part of a system is at fault with such little information?

Because I'm trying to figure out what's going on with such little information, so I'm trying to motivate the discussion by explaining with the information I have and hoping a kind internet stranger could help me figure out a fruitful path.

A Device Tree is meant to describe just one board, i.e. the specific configuration of one board. When you have a similar but different (circuit-wise) board, then that other board needs it own Device Tree file. That's why each Device Tree file (i.e. .dts or .dtb) is named for a board and not a SoC.

This was definitely a misconception on my part. I was under the impression that the kernel used the device tree to know what devices were present on the SoC. Thank you for pointing this out, as it could definitely be a root cause of the issue.

Nevertheless, I am curious. In the absence of a kernel or device tree, how is it possible for the same build of u-boot to get access to the same hardware (SD card) on both boards? If they are electronically different, shouldn't I need two different builds of u-boot?

Thanks

Originally Posted by MALDATA Obviously u-boot has some way of accessing hardware without the kernel or the device tree. And the fact that u-boot would work while the kernel didn't is curious to me.

U-Boot has its own version of device drivers to accomplish the booting task.
U-Boot device drivers rarely use interrupts, whereas Linux kernel device drivers certainly do use interrupts.
U-Boot device drivers may or may not use DMA, whereas Linux kernel device drivers typically do.

Also be aware that SD cards have several operating modes and configurations.
Some boot programs use just a low-speed, one-bit data path to accomplish its booting task.
The Linux kernel, depending on what is specified in the DT, will typically try to use a high-speed transfer mode, and the maximum data width available (e.g. 8 bits). Unless the driver is designed to use a degraded/fallback mode, the driver might fail its initialization, fail to install, and its device nodes in /dev will not be created.

Originally Posted by MALDATA You are correct, as I am assisting in the debugging of the device, and had not yet been provided with a complete boot log myself.

That would be like hunting with blindfold and earplugs.
IMO trying to debug a boot problem without the console output is a waste of time and effort.

Originally Posted by MALDATA In the absence of a kernel or device tree, how is it possible for the same build of u-boot to get access to the same hardware (SD card) on both boards? If they are electronically different, shouldn't I need two different builds of u-boot?

Depends on how similar or different the boards are.
Most "custom" boards are derived from a reference design from the SoC manufacturer, i.e. a board design that is known to function. Board designers that stray from the reference design by substituting different components (e.g. different oscillators and NAND) can create a project & software impact to boot programs and kernel.
Since the early stages of the boot task only needs a subset of the board's functionality to complete, it can be possible for similar boards (based on the same reference design) to utilize some common boot programs.

Regards

U-Boot has its own version of device drivers to accomplish the booting task. U-Boot device drivers rarely use interrupts, whereas Linux kernel device drivers certainly do use interrupts. U-Boot device drivers may or may not use DMA, whereas Linux kernel device drivers typically do. Also be aware that SD cards have several operating modes and configurations. Some boot programs use just a low-speed, one-bit data path to accomplish its booting task. The Linux kernel, depending on what is specified in the DT, will typically try to use a high-speed transfer mode, and the maximum data width available (e.g. 8 bits). Unless the driver is designed to use a degraded/fallback mode, the driver might fail its initialization, fail to install, and its device nodes in /dev will not be created.

This is all very interesting, thank you for sharing! We have started digging deeper, and this helps explain how it is possible for the boards to behave this way. Much appreciated.