[SOLVED] Handling of amd u-code

3rensho · 02-25-2021, 03:38 AM

Starting with kernel-5.10.xx (running 5.11.1 presently) every couple of days I get an error like the following in the system log -

Code:

kernel	mce: [Hardware Error]: CPU 0: Machine Check: 0 Bank 2: 9c254010000a0145
kernel	mce: [Hardware Error]: TSC 0 ADDR 2c0 MISC c008001c00000000 
kernel	mce: [Hardware Error]: PROCESSOR 2:600f20 TIME 1614231277 SOCKET 0 APIC 0 microcode 6000822
kernel	mce: [Hardware Error]: CPU 0: Machine Check: 0 Bank 2: dc25408000040136
kernel	mce: [Hardware Error]: TSC 0 ADDR 2b763fb38 MISC c008001b00000000 
kernel	mce: [Hardware Error]: PROCESSOR 2:600f20 TIME 1614228656 SOCKET 0 APIC 0 microcode 6000822

Never happened with earlier kernel versions. Doesn't seem to cause any problems and occurs only when system is just idling with only normal background processes running. Not under heavy load and not every day. I am running an AMD FX-9590 cpu. Doing some searches revealed that some AMD FX series processors are prone to these errors. Got to thinking that it could be a cpu u-code issue. Seems like the u-code is updated these days via the kernel firmware and mine is up to date as evidenced by entries in /lib/firmware/amd-ucode from 22.2.21.

Code:

-rw-r--r-- 1 root root 12684 Feb 22 05:50 microcode_amd.bin
-rw-r--r-- 1 root root   490 Feb 22 05:50 microcode_amd.bin.asc
-rw-r--r-- 1 root root  7876 Feb 22 05:50 microcode_amd_fam15h.bin
-rw-r--r-- 1 root root   473 Feb 22 05:50 microcode_amd_fam15h.bin.asc
-rw-r--r-- 1 root root  3510 Feb 22 05:50 microcode_amd_fam16h.bin
-rw-r--r-- 1 root root   473 Feb 22 05:50 microcode_amd_fam16h.bin.asc
-rw-r--r-- 1 root root  6476 Feb 22 05:50 microcode_amd_fam17h.bin
-rw-r--r-- 1 root root   488 Feb 22 05:50 microcode_amd_fam17h.bin.asc

I would like to know how and when cpu microcode is actually loaded. When booting, the very early output says it is loading amd ucode 6000822 for each of the eight cores. That has never changed in the last several years so I am wondering if the new ucode from the firmware is being loaded at all?? How to tell???

Found an old entry in my /boot directory from 1999 named amd-ucode.cpio. It apparently never gets updated. Searched and found a script to generate a new one, did so but still these occasional errors continue. Not sure /boot/amd-ucode.cpio is ever read.

So now, after that I have a couple of questions as I am thoroughly confused.

How,when and if cpu u-code is updated. If it is how to determine actual version?

Thanks in advance for any clarification after this long winded question.

business_kid · 02-25-2021, 05:09 AM

IIRC, You should be able to see a line or two from the kernel bootup on microcode. There is a script that checks for Meltdown & spectre and that may be useful if it checks microcode versions. I found microcode in the past on the Intel site during this Spectre/Meltdown business, which went on the next release to bleeding edge kernels. But it's a while before that filters down to distros. You might try there.

3rensho · 02-25-2021, 05:27 AM

Quote:

Originally Posted by business_kid

IIRC, You should be able to see a line or two from the kernel bootup on microcode. There is a script that checks for Meltdown & spectre and that may be useful if it checks microcode versions. I found microcode in the past on the Intel site during this Spectre/Meltdown business, which went on the next release to bleeding edge kernels. But it's a while before that filters down to distros. You might try there.

Thanks for that but doesn't get me any further along. As stated in OP I see ucode loaded on boot but mine hasn't changed in years. I'm interested in the ucode from the firmware, specifically how it gets loaded, at what point it is loaded and how to confirm that it is indeed loaded and functioning. Does the /boot/amd-ucode.cpio do anything at all or is it a remnant from a bygone era? The processing of firmware ucode is a big, opaque black box for me.

mickski56 · 02-25-2021, 06:00 AM

Things to check
Is /boot/amd-ucode.cpio updated when you run the script to generate the updated microcode cpio?
CONFIG_MICROCODE_AMD=y and CONFIG_BLK_DEV_INITRD=Y should be set in the kernel config I'm pretty sure that's standard.
For early loading /boot/amd-ucode.cpio must be the first initrd loaded (how depends on your boot loader)
I use lilo so in my lilo.conf the relevant bits are

initrd=/boot/amd-ucode.cpio

image = /boot/vmlinuz
label = Linux
append = ""
read-only # Partitions should be mounted read-only for checking
# Linux bootable partition config ends

LILO and potentially other old bootloaders do not support multiple initrd images.

You can also cat the cpio and the initrd you use together. Like so cat /boot/amd-ucode.cpio >/boot/initrd_ucode.gz; cat /boot/initrd.gz >> /boot/initrd_ucode.gz (order is important here)

A good read is https://wiki.archlinux.org/index.php/microcode

Make sure you have an known working fall back for your bootloader of choice before trying this.
HTH

3rensho · 02-25-2021, 07:51 AM

Quote:

Originally Posted by mickski56

Things to check
Is /boot/amd-ucode.cpio updated when you run the script to generate the updated microcode cpio?
CONFIG_MICROCODE_AMD=y and CONFIG_BLK_DEV_INITRD=Y should be set in the kernel config I'm pretty sure that's standard.
For early loading /boot/amd-ucode.cpio must be the first initrd loaded (how depends on your boot loader)
I use lilo so in my lilo.conf the relevant bits are

initrd=/boot/amd-ucode.cpio

image = /boot/vmlinuz
label = Linux
append = ""
read-only # Partitions should be mounted read-only for checking
# Linux bootable partition config ends

LILO and potentially other old bootloaders do not support multiple initrd images.

You can also cat the cpio and the initrd you use together. Like so cat /boot/amd-ucode.cpio >/boot/initrd_ucode.gz; cat /boot/initrd.gz >> /boot/initrd_ucode.gz (order is important here)

A good read is https://wiki.archlinux.org/index.php/microcode

Make sure you have an known working fall back for your bootloader of choice before trying this.
HTH

Thank you mickski56 for the information. Clearly I've been missing a lot. Lucky this thing even boots most likely. I use grub now instead of lilo. My /boot/amd-ucode.cpio never updates when I build a new kernel so will amend my build scripts to allow for updating. The same old one had been in /boot since a couple of years. Not even sure how it got there. I will read over the wiki you link to. When you talk about catting /boot/amd-ucode.cpio >/boot/initrd_ucode.gz; cat /boot/initrd.gz >> /boot/initrd_ucode.gz I'm not sure what to do with the resulting initrd_ucode.gz. I've only gotten booting with an initrd to work fairly recently after a kind soul pointed out that the mkinitrd command needs a -R to allow for a raid system. So what I do now is run -

mkinitrd -c -R -k 5.11.1 -m ext4 and that produces the initrd-tree and initrd.gz respectively. Not sure where the resultant /boot/initrd_ucode.gz would fit into my procedure. That is, does the /boot/initrd_ucode.gz get picked up when I run the mkinitrd -c -R -k 5.11.1 -m ext4 command?? I run grub-mkconfig after generating the initrd bits. Appreciate any further tips you might be able to give. Off to peruse the wiki now. Thanks again

mickski56 · 02-25-2021, 08:28 AM

@3rensho If the output of grub-mkconfig doesn't mention the microcode cpio ,you may need to edit /etc/default/grub as per
https://wiki.gentoo.org/wiki/Microco...rocode_loading
or edit your microcode generating script to generate one of the supported microcode cpio names.

You will only need to regenerate the microcode cpio when the is an update to the kernel-firmware package, not necessarily on every kernel update.
Don't worry about concatenating the two files together that's a lilo thing as lilo doesn't support multiple initrd's.

drumz · 02-25-2021, 08:38 AM

Quote:

Originally Posted by 3rensho

When you talk about catting /boot/amd-ucode.cpio >/boot/initrd_ucode.gz; cat /boot/initrd.gz >> /boot/initrd_ucode.gz I'm not sure what to do with the resulting initrd_ucode.gz. I've only gotten booting with an initrd to work fairly recently after a kind soul pointed out that the mkinitrd command needs a -R to allow for a raid system. So what I do now is run -

mkinitrd -c -R -k 5.11.1 -m ext4 and that produces the initrd-tree and initrd.gz respectively. Not sure where the resultant /boot/initrd_ucode.gz would fit into my procedure. That is, does the /boot/initrd_ucode.gz get picked up when I run the mkinitrd -c -R -k 5.11.1 -m ext4 command?? I run grub-mkconfig after generating the initrd bits. Appreciate any further tips you might be able to give. Off to peruse the wiki now. Thanks again

mkinitrd in Slackware can do this automatically for you. I have an Intel CPU, but the process should be the same for AMD. Here is my /etc/mkinitrd.conf:

Code:

# mkinitrd.conf.sample
# See "man mkinitrd.conf" for details on the syntax of this file
#
#SOURCE_TREE="/boot/initrd-tree"
#CLEAR_TREE="0"
#OUTPUT_IMAGE="/boot/initrd.gz"
#KERNEL_VERSION="$(uname -r)"
#KEYMAP="us"
#MODULE_LIST="ext4"
#LUKSDEV="/dev/sda2"
#LUKSTRIM="/dev/sda2" # verify support with 'hdparm -I $dev | grep TRIM'
#LUKSKEY="LABEL=TRAVELSTICK:/keys/alienbob.luks"
#ROOTDEV="/dev/sda1"
#ROOTFS="ext4"
RESUMEDEV="/dev/sda2"
#RAID="0"
#LVM="0"
#UDEV="1"
#MODCONF="0"
MICROCODE_ARCH="/boot/intel-ucode.cpio"
#WAIT="1"

Note the MICROCODE_ARCH line.

And here is my script /boot/prepare_boot_files.sh that I run when upgrading kernels (which has to be piped to "sh"):

Code:

#!/bin/sh
# prepare_boot_files.sh
#
# Integrate Intel microcode into initrd and run eliloconfig

# First run /usr/share/mkinitrd/mkinitrd_command_generator.sh and create an
# initrd. Also ensure /boot/vmlinuz points to the desired kernel.
#KVER=4.19.88-etr
KVER=5.10.18-etr
#KVER=5.4.89-etr
#KVER=4.4.240
/usr/share/mkinitrd/mkinitrd_command_generator.sh -k $KVER -a "-F -s /boot/initrd-tree-$KVER -o /boot/initrd-$KVER.gz"

# No longer needed:
#mv /boot/initrd.gz /boot/initrd.orig.gz
#cat /boot/intel-ucode.cpio /boot/initrd.orig.gz > /boot/initrd.gz
#eliloconfig

# Copy bzImage/vmlinuz and initrd to efi directory and update elilo.conf

# Copy vmlinuz and initrd.gz to USB boot disk

Note the old commented-out style of using cat, which is now handled by mkinitrd itself.

3rensho · 02-25-2021, 08:51 AM

Ausgezeichnet!!!!!! Now it becomes clear. Thank you guys so much for clarifying the situation. I will sleep tonight. That's what I love about Slackware-current + my foggy brain = endless learning opportunities. Especially during these times. Off to modify some scripts.

bassmadrigal · 02-25-2021, 10:13 AM

Just a note, AMD processors are supposed to be able to do late loading of microcode by simply having it installed (in Slackware, via the kernel-firmware package), however, this function doesn't seem to be working, so you'll need to load it manually when booting (which is what was accomplished in this thread). Maybe one day they'll fix it...