Nvidia system extensions not persisting after node reboot.

[Dhruv Dole]

Bug Report

Description

Nvidia system extensions are not persisting after a reboot. The nvidia driver and runtime work perfectly when the node is freshly provisoned. After a reboot they do not show in "talosctl get extensions -n talos-04" and the modules are not loaded either.

Logs

Environment

• 3 control plane nodes and 1 worker. All nodes are schedulable.
• The worker has an Nvidia RTX 3070.
• The nodes are proxmox VMs provisioned and bootstrapped by terraform.
• The worker node has a single machineconfig patch "nvidia-gpu.yaml".
• The install image specified includes the following system extensions:
  • siderolabs/nvidia-container-toolkit (535.129.03-v1.14.6)
  • siderolabs/nonfree-kmod-nvidia (535.129.03-v1.7.0)

nvidia-gpu.yaml:

machine:
  kernel:
    modules:
      - name: nvidia
      - name: nvidia_uvm
      - name: nvidia_drm
      - name: nvidia_modeset
  sysctls:
    net.core.bpf_jit_harden: 1
  install:
    image: factory.talos.dev/installer/0412a9a6369c0fb55e913cdfcbf4ad6ca3fab6e56ab71198ec4b58ad7e7a4ddd:v1.7.0

• Talos version: [talosctl version --nodes <problematic nodes>]

Client:
        Tag:         v1.7.0
        SHA:         70fb41ff
        Built:       
        Go version:  go1.22.2
        OS/Arch:     linux/amd64
Server:
        NODE:        talos-04
        Tag:         v1.7.0
        SHA:         70fb41ff
        Built:       
        Go version:  go1.22.2
        OS/Arch:     linux/amd64
        Enabled:     RBAC

• Kubernetes version: [kubectl version --short]

Client Version: v1.29.4
Kustomize Version: v5.0.4-0.20230601165947-6ce0bf390ce3
Server Version: v1.29.4

• Platform:

[Andrey Smirnov]

Most probably you're booting not from disk, e.g. from an ISO. Or vice versa, booting off a disk image and the install never happened.

See boot assets guide, or use Image Factory wizard

[Dhruv Dole]

Changing the boot order to boot from disk first fixed the issue. Just wanted to point out that on this page of the [docs](https://www.talos.dev/v1.7/talos-guides/install/bare-metal-platforms/iso/), while it does state that the boot order should prefer the disk over ISO, it also says that if it boots from the ISO it will boot into an existing installation. Based on the docs, there is no reason to think that extensions would not load. Thanks for your help