early x86 microcode loading

[Mark Johnston]

I've been working on support for early loading of microcode updates and
wanted to solicit feedback on the approach before starting to get any
code changes reviewed.

Currently we support microcode updates via cpuctl(4), where
cpucontrol(8) passes microcode blobs to the kernel via an ioctl
interface. Updates are distributed by the sysutils/devcpu-data port.
The scheme has a few shortcomings:
- Microcode updates may introduce new CPU features, but since we load
microcode from userland, updates are performed well after the kernel
has done CPU feature detection.
- Updates need to be reapplied after an ACPI suspend/resume, and there's
currently no mechanism to automatically reapply the update after a
resume.
- Updates aren't applied until userspace starts running, so there exists
a window in which the kernel is running without vulnerability
mitigations provided by microcode updates.

The aim of this work is to instead use the boot loader to load microcode
updates into kernel memory, and modify the kernel to apply the updates
as the first step in BSP and AP initialization, as well as after an ACPI
resume. To configure an update, one would then just need to add the
following lines to loader.conf:

cpu_ucode_load="YES"
cpu_ucode_name="/boot/firmware/microcode.bin"
cpu_ucode_type="cpu_ucode"

The kernel would then automatically load the update during processor
initialization in the subsequent boot-up.

A given Intel microcode update applies only to CPUs of a specific
<family, model, stepping> tuple, while AMD releases a single update per
processor family. My plan is to extend cpucontrol(8) to determine the
correct microcode update for the running system, and have the devcpu-data
port install the corresponding file to /boot/firmware. The port could
then add the following to loader.conf.local:

devcpu_data_load="YES"
devcpu_data_name="/boot/firmware/<update file>"
devcpu_data_type="cpu_ucode"

Currently, the port doesn't automatically enable microcode updates; one
needs to enable the microcode_update rc script after installing the
port. I'm not yet sure how this should be integrated with early
loading. For example, if "service microcode_update onestart" enables
early loading, how should early loading be disabled? Would it be
reasonable for the port to automatically enable updates when it is
installed?

For at least Intel, my intention is to support loading of multiple
update files concatenated together, and have the kernel determine the
correct update to apply. This is to make it easier to support a cluster
of mixed systems which boot from some shared environment; rather than
needing to select and configure the correct update for each machine, one
can supply all updates as a single file and have each machine select the
right one automatically. This would come at the expense of some wasted
memory: the combined set of Intel microcode files is currently about
1.8MB in size.

A small patch to the loader is needed to guarantee the 16 byte-alignment
of the loaded microcode update required by Intel. This will require a
bootcode update on GPT-based systems in order for early loading to work
reliably.

I'm interested in any feedback on the above, and especially any
suggestions on how this feature should be integrated with the
devcpu-data port. Thanks in advance.

[John Baldwin]

Post by Mark Johnston
I'm interested in any feedback on the above, and especially any
suggestions on how this feature should be integrated with the
devcpu-data port. Thanks in advance.

In general this sounds good to me. I'm not quite sure how to manage
setting the lines in loader.conf. Using 'service foo start/stop'
seems to be a bit of an abuse of what service is used for as opposed
to just including a standalone script in the devcpu-data port with a
suitable pkg-message. sysrc seems to be a closer analog than service
in theory, but shoehorning this into sysrc would seem to be quite
ugly and doesn't seem like a good idea to me either.

[Eric van Gyzen]

Post by John Baldwin

Post by Mark Johnston
I'm interested in any feedback on the above, and especially any
suggestions on how this feature should be integrated with the
devcpu-data port. Thanks in advance.

In general this sounds good to me. I'm not quite sure how to manage
setting the lines in loader.conf. Using 'service foo start/stop'
seems to be a bit of an abuse of what service is used for as opposed
to just including a standalone script in the devcpu-data port with a
suitable pkg-message. sysrc seems to be a closer analog than service
in theory, but shoehorning this into sysrc would seem to be quite
ugly and doesn't seem like a good idea to me either.

I hesitate to suggest this, due to the necessary change in /two/ boot
loaders, but... The foo.d approach is very convenient for packages. An
activation script installed by the port/package could create a new
/boot/loader.conf.d/devcpu-data file containing these lines. A
pkg-message would instruct the user to run it.

Eric

[Simon J. Gerraty]

Post by Eric van Gyzen
I hesitate to suggest this, due to the necessary change in /two/ boot
loaders, but... The foo.d approach is very convenient for packages. An
activation script installed by the port/package could create a new
/boot/loader.conf.d/devcpu-data file containing these lines. A
pkg-message would instruct the user to run it.

The ability for packages to add to the boot env is indeed nice,
(we do that in Junos) but a loader.conf.d/ would be problematic for
anyone whating verification (at least using my implementation) during
boot.

Allowing subdirs such that a loader.conf snippet, the module (or
whatever) and associated signatures could all co-exist without
interference with other packages works nicely.

[Shawn Webb]

Hey Mark,

Thank you very much for working on this and opening the discussion.
What you've drafted sounds reasonable to me, but perhaps with one
edge case.

Post by Mark Johnston
I've been working on support for early loading of microcode updates and
wanted to solicit feedback on the approach before starting to get any
code changes reviewed.
Currently we support microcode updates via cpuctl(4), where
cpucontrol(8) passes microcode blobs to the kernel via an ioctl
interface. Updates are distributed by the sysutils/devcpu-data port.
- Microcode updates may introduce new CPU features, but since we load
microcode from userland, updates are performed well after the kernel
has done CPU feature detection.
- Updates need to be reapplied after an ACPI suspend/resume, and there's
currently no mechanism to automatically reapply the update after a
resume.
- Updates aren't applied until userspace starts running, so there exists
a window in which the kernel is running without vulnerability
mitigations provided by microcode updates.
The aim of this work is to instead use the boot loader to load microcode
updates into kernel memory, and modify the kernel to apply the updates
as the first step in BSP and AP initialization, as well as after an ACPI
resume. To configure an update, one would then just need to add the
cpu_ucode_load="YES"
cpu_ucode_name="/boot/firmware/microcode.bin"
cpu_ucode_type="cpu_ucode"
The kernel would then automatically load the update during processor
initialization in the subsequent boot-up.
A given Intel microcode update applies only to CPUs of a specific
<family, model, stepping> tuple, while AMD releases a single update per
processor family. My plan is to extend cpucontrol(8) to determine the
correct microcode update for the running system, and have the devcpu-data
port install the corresponding file to /boot/firmware. The port could
devcpu_data_load="YES"
devcpu_data_name="/boot/firmware/<update file>"
devcpu_data_type="cpu_ucode"

I'm curious about what would happen if I moved the drives to a new
system and booted off of them, perhaps forgetting to comment out the
above lines in loader.conf beforehand.

Additionally, how would I instruct the system in such a case to
re-probe which firmware file I need?

I recognize this could be construed as an edge case, but I've done
this multiple times (and, thanks to ZFS, really easily).

Thanks,

[Mark Johnston]

Post by Shawn Webb
Hey Mark,
Thank you very much for working on this and opening the discussion.
What you've drafted sounds reasonable to me, but perhaps with one
edge case.

Post by Mark Johnston
I've been working on support for early loading of microcode updates and
wanted to solicit feedback on the approach before starting to get any
code changes reviewed.
Currently we support microcode updates via cpuctl(4), where
cpucontrol(8) passes microcode blobs to the kernel via an ioctl
interface. Updates are distributed by the sysutils/devcpu-data port.
- Microcode updates may introduce new CPU features, but since we load
microcode from userland, updates are performed well after the kernel
has done CPU feature detection.
- Updates need to be reapplied after an ACPI suspend/resume, and there's
currently no mechanism to automatically reapply the update after a
resume.
- Updates aren't applied until userspace starts running, so there exists
a window in which the kernel is running without vulnerability
mitigations provided by microcode updates.
The aim of this work is to instead use the boot loader to load microcode
updates into kernel memory, and modify the kernel to apply the updates
as the first step in BSP and AP initialization, as well as after an ACPI
resume. To configure an update, one would then just need to add the
cpu_ucode_load="YES"
cpu_ucode_name="/boot/firmware/microcode.bin"
cpu_ucode_type="cpu_ucode"
The kernel would then automatically load the update during processor
initialization in the subsequent boot-up.
A given Intel microcode update applies only to CPUs of a specific
<family, model, stepping> tuple, while AMD releases a single update per
processor family. My plan is to extend cpucontrol(8) to determine the
correct microcode update for the running system, and have the devcpu-data
port install the corresponding file to /boot/firmware. The port could
devcpu_data_load="YES"
devcpu_data_name="/boot/firmware/<update file>"
devcpu_data_type="cpu_ucode"

I'm curious about what would happen if I moved the drives to a new
system and booted off of them, perhaps forgetting to comment out the
above lines in loader.conf beforehand.
Additionally, how would I instruct the system in such a case to
re-probe which firmware file I need?
I recognize this could be construed as an edge case, but I've done
this multiple times (and, thanks to ZFS, really easily).

This is being discussed in another subthread. Right now, an update
simply wouldn't be applied during the first boot. My notion is that an
rc script provided by the port would automatically reconfigure the
update, so it'd be applied upon a subsequent reboot.

[Poul-Henning Kamp]

--------

Post by Mark Johnston
My plan is to extend cpucontrol(8) to determine the
correct microcode update for the running system, and have the devcpu-data
port install the corresponding file to /boot/firmware.

This is problematic when a diskimage is migrated to a different CPU,
only on the second reboot on the new hardware are you certain to
have the correct microcode.

For images which are resurrected on demand on whatever hardware is
available this really problematic.

NB: before anybody misunderstands: This is not a problem for
guest systems under virtualization - they don't need microcode updates.

[Mark Johnston]

Post by Poul-Henning Kamp
--------

Post by Mark Johnston
My plan is to extend cpucontrol(8) to determine the
correct microcode update for the running system, and have the devcpu-data
port install the corresponding file to /boot/firmware.

This is problematic when a diskimage is migrated to a different CPU,
only on the second reboot on the new hardware are you certain to
have the correct microcode.
For images which are resurrected on demand on whatever hardware is
available this really problematic.

I can think of three ways to address this case:

1a) Always load all of the updates as a single file, and select the
correct update during boot. As I pointed out, this wastes some
memory (a couple of megabytes currently). On at least amd64 it
doesn't look very practical to release the pages backing the
update file back to the VM. That is, I don't think we can easily
"shed" the preloaded file data once the correct update has been
selected and saved.

1b) Have the devcpu-data port operate in one of two modes: either the
port selects the update for the current machine, as I outlined in my
original mail, or it concatenates all of the updates as in 1a) and
the kernel selects the correct update. This way we'd only
waste memory if the disk image is to be shared among multiple
machines. I'm not sure what the mechanism should be for selecting
the mode.

2) Install all updates to a directory under /boot and add code to the
loader to perform the selection, and pass only the required microcode
file to the kernel. This seems straightforward to me, though I'm not
yet sure exactly where in the loader this logic should go.

[Konstantin Belousov]

Post by Mark Johnston

Post by Poul-Henning Kamp
--------

Post by Mark Johnston
My plan is to extend cpucontrol(8) to determine the
correct microcode update for the running system, and have the devcpu-data
port install the corresponding file to /boot/firmware.

This is problematic when a diskimage is migrated to a different CPU,
only on the second reboot on the new hardware are you certain to
have the correct microcode.
For images which are resurrected on demand on whatever hardware is
available this really problematic.

1a) Always load all of the updates as a single file, and select the
correct update during boot. As I pointed out, this wastes some
memory (a couple of megabytes currently). On at least amd64 it
doesn't look very practical to release the pages backing the
update file back to the VM. That is, I don't think we can easily
"shed" the preloaded file data once the correct update has been
selected and saved.
1b) Have the devcpu-data port operate in one of two modes: either the
port selects the update for the current machine, as I outlined in my
original mail, or it concatenates all of the updates as in 1a) and
the kernel selects the correct update. This way we'd only
waste memory if the disk image is to be shared among multiple
machines. I'm not sure what the mechanism should be for selecting
the mode.
2) Install all updates to a directory under /boot and add code to the
loader to perform the selection, and pass only the required microcode
file to the kernel. This seems straightforward to me, though I'm not
yet sure exactly where in the loader this logic should go.

What is the problem with having the microcode blob unmatched ? The
result would be only lack of the update for the CPU. If user cares about
having the updated microcode, he would run the required command anew.
Or you might add an automatic run of such command on shutdown.

[Mark Johnston]

Post by Konstantin Belousov

Post by Mark Johnston

Post by Poul-Henning Kamp
--------

Post by Mark Johnston
My plan is to extend cpucontrol(8) to determine the
correct microcode update for the running system, and have the devcpu-data
port install the corresponding file to /boot/firmware.

This is problematic when a diskimage is migrated to a different CPU,
only on the second reboot on the new hardware are you certain to
have the correct microcode.
For images which are resurrected on demand on whatever hardware is
available this really problematic.

(To be clear, this case can be handled with my proposal: one would
concatenate all of the updates together and load the result, and the
kernel would select the correct update and apply it during boot. The
issue is with the default behaviour of the devcpu-data port.)

Post by Konstantin Belousov

Post by Mark Johnston
1a) Always load all of the updates as a single file, and select the
correct update during boot. As I pointed out, this wastes some
memory (a couple of megabytes currently). On at least amd64 it
doesn't look very practical to release the pages backing the
update file back to the VM. That is, I don't think we can easily
"shed" the preloaded file data once the correct update has been
selected and saved.
1b) Have the devcpu-data port operate in one of two modes: either the
port selects the update for the current machine, as I outlined in my
original mail, or it concatenates all of the updates as in 1a) and
the kernel selects the correct update. This way we'd only
waste memory if the disk image is to be shared among multiple
machines. I'm not sure what the mechanism should be for selecting
the mode.
2) Install all updates to a directory under /boot and add code to the
loader to perform the selection, and pass only the required microcode
file to the kernel. This seems straightforward to me, though I'm not
yet sure exactly where in the loader this logic should go.

What is the problem with having the microcode blob unmatched ? The
result would be only lack of the update for the CPU. If user cares about
having the updated microcode, he would run the required command anew.
Or you might add an automatic run of such command on shutdown.

Given that the trend seems to be for new CPU vulnerabilities to be
mitigated by microcode updates, I think we'd want a mechanism that makes
a reasonable effort to work reliably once it is configured by the
administrator. From this perspective, special cases which require an
extra reboot or an extra command invocation at shutdown (what if the
system panics?) are undesirable. Perhaps we should indeed declare these
special cases as unsupported by devcpu-data, but I would prefer not to
do so if possible.

[Poul-Henning Kamp]

--------

Post by Mark Johnston
1a) Always load all of the updates as a single file, and select the
correct update during boot. As I pointed out, this wastes some
memory (a couple of megabytes currently). On at least amd64 it
doesn't look very practical to release the pages backing the
update file back to the VM. That is, I don't think we can easily
"shed" the preloaded file data once the correct update has been
selected and saved.

Then we should fix that problem, rather than build an elaborate
workaround for in each and every subsystem which runs into this.

Isn't this the same issue with the splash-screen for instance ?

[Mark Johnston]

Post by Poul-Henning Kamp
--------

Post by Mark Johnston
1a) Always load all of the updates as a single file, and select the
correct update during boot. As I pointed out, this wastes some
memory (a couple of megabytes currently). On at least amd64 it
doesn't look very practical to release the pages backing the
update file back to the VM. That is, I don't think we can easily
"shed" the preloaded file data once the correct update has been
selected and saved.

Then we should fix that problem, rather than build an elaborate
workaround for in each and every subsystem which runs into this.

I implemented this in r336505 and plan to use it for the microcode
update file.

Post by Poul-Henning Kamp
Isn't this the same issue with the splash-screen for instance ?

I didn't look into the splash screen code, but it can be made to
give memory back to the system by using preload_delete_name().