BIOS and Firmware management with EVE

BIOS (or more generally firmware) are the only external pieces of software that EVE has to depend on and sometimes manage. EVE itself depends on firmware during the boot phase and can be currently bootstrapped on:

  1. UEFI compliant firmware on both amd64 and arm64 architectures
  2. Legacy PC BIOS on amd64
  3. Legacy u-boot firmware (only version 2021.01 and higher) on arm64

Last option is presented mostly for completeness' sake, since the minimum version of u-boot EVE depends on comes with a rather complete (although not fully compliant) UEFI implementation which allows EVE to simply rely on it as an UEFI firmware (option #1).

UEFI compliant firmware configuration on amd64

EVE expects

  • VT-x
  • VT-d (if available)
  • hardware watchdog (if available)
  • TPM 2.0 (if available)

options to be enabled by your BIOS/firmware. Most of the time this is a manual step that an operator has to perform sitting in from of the console. On a few pieces of hardware (most notable Dell IoT Gateway 300x family) there are ways of manipulating firmware settings by means of a command line utility (e.g. cctk).

EVE currently doesn't support firmware updates, although preliminary work to support UEFI capsule updates has been started.

EVE relies on UEFI implementation to correctly fill out SMBIOS and ACPI tables. Unlike arm64 (see below), there's currently no way to patch those tables if they are setup incorrectly.

UEFI compliant firmware on arm64

From EVE's standpoint, ARM world is really bifurcated when it comes to firmware. On rackable servers, EVE typically expects to find a self-contained UEFI implementation which makes the rest indistinguishable from what was described in the previous section for amd64 (even device trees are transparently passed to EVE from UEFI implementation without any extra effort).

On smaller ARM boards, the situation becomes much more complex. ARM boards typically have different layers of firmware. Some that can be managed and easily and some that can be not. For example, on a popular Raspberry Pi 4 ARM board there are at least 3 levels of firmware:

  1. Raspberry Pi 4 boot EEPROM (formerly known as bootcode.bin)
  2. VideoCore boot firmware
  3. u-boot firmware or UEFI firmware

EVE helps managing layers #2 and #3 and leaves #1 to the operator. In general, as long as firmware can be derived from u-boot -- EVE can manage it by carrying all the extra firmware blobs in its UEFI compliant exfat partition (UEFI doesn't mind if extra files are present in that partition).

On ARM boards EVE leverages u-boot to provide UEFI environment and fill out SMBIOS tables correctly. Unlike a fully compliant UEFI implementation, u-boot seems to be a bit difficult when it comes to device tree pass-through via EFI_DTB_GUID. Besides, there maybe situations when u-boot's own need for a particular set of settings in the device tree might conflict with the needs of a hypervisor or a kernel that is shipped as part of EVE. That's why on ARM boards EVE typically has two device trees:

  • u-boot specific device tree: typically located somewhere next to u-boot binary/built right into u-boot or supplied by higher level firmware
  • EVE specific device tree: shipped in EVE's rootfs image under /boot/dtb/.

EVE's grub config is in business of selecting an optional, EVE specific device tree based on the vendor/product settings of the board it recognizes. GRUB takes these values from SMBIOS tables and it is very important that they are filled out correctly.

Luckily, starting from u-boot 2021.01 it is now possible to explicitly specify vendor/product settings in the u-boot specific device tree (either directly or via an overlay).

For example, on a stock Raspberry Pi 4, we are using the following overlay and making sure it is explicitly set in config.txt in dtoverlay=raspberrypi-rpi stanza. This has an effect of making GRUB pick up the correct EVE-specific device tree from EVE's root filesystem. For products that bundle Raspberry Pi 4 with various HATs, the idea then is to have

  • a product specific device tree always built as part of EVE
  • an optional device tree overlay provided under overlays folder

This allows for a runtime re-configuration away from a stock Raspberry Pi 4 functionality to a more HAT-specific one by simply changing one line in config.txt.

For example, for an industrial HAT produced by Advantech UNO-220 here's how an EVE-specific device tree (historically in pkg/new-kernel/patches-5.10.x/0021-Add-uno-220-dts.patch, now maintained in lf-edge/eve-kernel) and a u-boot specific overlay would look like.

Using OVMF with EVE

What is OVMF?

OVMF (Open Virtual Machine Firmware) is a project that provides a UEFI firmware implementation for virtual machines. It is part of the TianoCore open-source UEFI development project. OVMF allows virtual machines to leverage UEFI features, offering a modern firmware interface compared to traditional BIOS.

When is OVMF Used?

In the context of EVE, OVMF is used for:

  1. Applications Running in FML Mode.
  2. Applications running on ARM Devices.

OVMF Files

OVMF firmware consists of several files, including:

  1. OVMF_CODE.fd: Contains the firmware code.
  2. OVMF_VARS.fd: Contains the firmware variables.
  3. OVMF.fd: A combined firmware image that includes both the code and variables.

In EVE, we use both approaches, depending on the application's requirements: the combined OVMF.fd file, or separate OVMF_CODE.fd and OVMF_VARS.fd files.

OVMF Settings (_VARS File)

OVMF uses a variable store file, commonly named OVMF_VARS.fd, to persist UEFI variables across reboots. This file contains essential firmware settings, including boot entries, framebuffer resolution, and other UEFI configurations.

In the context of EVE, we use the separate OVMF_VARS.fd and OVMF_CODE.fd files to customize the firmware settings. Having separate files allows us to provide a customized _VARS file when we need to provide some specific predefined settings. At the moment, we provide a custom _VARS file only for setting the framebuffer resolution.

How We Store OVMF Settings Files

In EVE, we store the OVMF settings per application in the persist/vault/ovmf directory. Each application has its own _VARS file, which is a copy of the original OVMF_VARS.fd file.

Generating OVMF Settings Files

At the moment, we customize the OVMF settings manually to provide a way to choose a specific resolution for the framebuffer. For that purpose, we build the original OVMF_VARS.fd file from the edk2 sources, which are compiled as part of the UEFI package in our build system and based on this file vary the settings as needed.

  1. Build an original OVMF_VAR file from the edk2 sources
  2. Start a virtual machine using the freshly built OVMF_CODE.fd and OVMF_VARS.fd files to initialize the UEFI environment.
  3. Access the OVMF UEFI setup menu during the VM's boot sequence.
  4. Navigate through the menu to set the necessary firmware configurations required by EVE.
  5. Clean the unnecessary settings. It is important to remove any auto-detected boot entries to prevent another VM from booting into an unintended path.
  6. Commit the changes to the OVMF_VARS.fd file and save it as the customized _VARS file.

By pre-configuring the _VARS file, we eliminate the need for manual UEFI configuration on each VM instance, streamlining the deployment process.

Alternative Approach for OVMF Settings

In the future, we plan to provide a more automated way to generate the OVMF settings files. We can achieve this by creating a tool that allows EVE to set the necessary firmware configurations programmatically. An example of such a tool can be found in some popular Linux distributions. Technically, it is the same as the manual approach, but it is automated and can be run as a script.

Runtime Boot Order Configuration

While OVMF settings files provide static boot configuration, EVE also supports runtime boot order control for VMs through multiple mechanisms: Controller API, Local Profile Server (LPS), and device configuration properties.

The Challenge

By default, UEFI firmware (OVMF) prioritizes USB devices in the boot order. This can be problematic when bootable USB drives are attached to VMs via USB passthrough - the VM may accidentally boot from USB instead of its configured disk, potentially causing data loss or service disruption.

The Solution

EVE uses a patched OVMF that includes EveBootOrderLib, which reads boot order hints from QEMU's fw_cfg mechanism at runtime. This allows operators to:

  • Disable USB boot (nousb): Deprioritize USB devices, ensuring the VM boots from disk even if a bootable USB is attached
  • Enable USB boot (usb): Prioritize USB devices in boot order
  • Use default (empty): Fall back to standard UEFI boot order behavior

This firmware-level approach is necessary because EVE's USB passthrough uses dynamic hotplug-USB devices are attached to running VMs at runtime via QMP device_add. QEMU's standard bootindex parameter only works for devices configured statically at VM startup and cannot be specified for dynamically-added devices.

How It Works

Boot order can be configured from three sources (highest to lowest priority):

  1. LPS - via POST /api/v1/appbootinfo endpoint (per-VM)
  2. Controller API - via VmConfig.boot_order in application config (per-VM)
  3. Device Property - via app.boot.order device configuration (device-wide)

EVE determines the effective boot order by checking each source in priority order, using the first source that provides a non-default value. The setting is then passed to QEMU via fw_cfg file opt/eve.bootorder, where OVMF's EveBootOrderLib reads it during early boot and adjusts the boot order before the UEFI boot menu is displayed.

Interaction with Custom OVMF.fd

When a custom OVMF.fd is used (e.g., for FML mode with custom framebuffer resolution), the boot order configuration interacts as follows:

Configuration Standard OVMF Custom OVMF.fd
"usb" USB prioritized USB prioritized
"nousb" USB removed from boot order USB removed from boot order
"" (default) USB prioritized (UEFI standard) Uses precooked boot order

The EveBootOrderLib honors the fw_cfg setting regardless of whether standard or custom OVMF is used. The only difference is the fallback behavior when no explicit setting is provided.

For detailed configuration instructions, see VM Boot Order.