Giters

yevgeny-shnaidman / amd-gpu-operator

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

KMM Issue to create amd-gpu driver image

KakorHolloway opened this issue · comments

commented

Hello,

First of all thank you very much for this operator.

I'm trying to use this solution on my OKD cluster in version 4.14 to install an AMD RADEON RX 7900XT card as GPU to test AI models.

Unfortunately I have a problem creating the image on the internal registry. I have this problem on kmm operator pod :

E0507 13:44:31.120907 1 controller.go:324] kmm "msg"="Reconciler error" "error"="failed to handle build for kernel version 6.5.5-200.fc38.x86_64: could not synchronize the build: could not make Build template: could not get DTK image for kernel 6.5.5-200.fc38.x86_64: could not find kernel 6.5.5-200.fc38.x86_64 in kernel --> OS mapping" "Module"={"name":"dc-harbor","namespace":"openshift-amd-gpu"} "controller"="BuildSignReconciler" "controllerGroup"="kmm.sigs.x-k8s.io" "controllerKind"="Module" "name"="dc-harbor" "namespace"="openshift-amd-gpu" "reconcileID"="ae042701-0f38-4dfe-802b-56795b3f5691"

If I remove DeviceConfig to add the DTK image value of my cluster version on the Module object (I remove the owner reference), and if I change DTK_AUTO in the configuration map to DTK, I can start the build .

Even with this option, I have to add a new variable named "KVERS" and replace it with the kernel version of the DTK image. (And apply this in the config map to replace all KERNEL_VERSION variables with KVERS). (6.5.5-200.fc38.x86_64 to 5.14.0-284.36.1.el9_2.x86_64)

You will find my custom files attached.
cm_and_modude.txt

If all those change are mades, I can build the image and use it by recreating the DeviceConfig file but It will not work as intended because of those logs :

I0507 13:56:47.805280       1 main.go:100] kmm-worker "msg"="Starting worker" "git commit"="e1625b2" "version"="2.0.1"
I0507 13:56:47.805292       1 main.go:101] kmm-worker "msg"="Reading pull secrets" "base dir"="/var/run/kmm/pull-secrets"
I0507 13:56:47.805362       1 pullsecrets.go:39] kmm-worker "msg"="Reading file" "path"="/var/run/kmm/pull-secrets/amd-gpu-operator-kmm-module-loader-dockercfg-wjwn5/..2024_05_07_13_31_23.2974230544/.dockercfg" "type"="kubernetes.io/dockercfg"
I0507 13:56:47.805388       1 pullsecrets.go:35] kmm-worker "msg"="Unhandled file name; ignoring" "path"="/var/run/kmm/pull-secrets/amd-gpu-operator-kmm-module-loader-dockercfg-wjwn5/..data"
I0507 13:56:47.805395       1 pullsecrets.go:39] kmm-worker "msg"="Reading file" "path"="/var/run/kmm/pull-secrets/amd-gpu-operator-kmm-module-loader-dockercfg-wjwn5/.dockercfg" "type"="kubernetes.io/dockercfg"
I0507 13:56:47.807301       1 mirror.go:43] kmm-worker "msg"="No configuration found for registry" "image name"="<myregistry>/public/amddriver:v1.0"
I0507 13:56:47.807314       1 worker.go:190] kmm-worker "msg"="Pulling image" "image name"="<myregistry>/public/amddriver:v1.0"
time="2024-05-07T13:57:12Z" level=warning msg="ignored xattrs in archive: underlying filesystem doesn't support them" errors="[operation not permitted]"
time="2024-05-07T13:57:12Z" level=warning msg="ignored xattrs in archive: underlying filesystem doesn't support them" errors="[operation not permitted]"
I0507 13:57:16.304671       1 worker.go:198] kmm-worker "msg"="Image pulled successfully" "dir"="/var/run/kmm/images/<myregistry>/public/amddriver:v1.0/fs" "image name"="<myregistry>/public/amddriver:v1.0"
I0507 13:57:16.304743       1 worker.go:49] kmm-worker "msg"="Unloading in-tree module" "name"="amdgpu"
I0507 13:57:16.304831       1 modprobe.go:33] kmm-worker "msg"="Running modprobe" "command"="/usr/sbin/modprobe -rv amdgpu"
I0507 13:57:16.306316       1 worker.go:59] kmm-worker "msg"="preparing firmware for loading" "host mount directory"="/var/lib/firmware" "image directory"="/var/run/kmm/images/harbor.kakor.ovh/public/amddriver:v1.0/fs/firmwareDir/updates"
I0507 13:57:16.614772       1 modprobe.go:33] kmm-worker "msg"="Running modprobe" "command"="/usr/sbin/modprobe -vd /var/run/kmm/images/<myregistry>/public/amddriver:v1.0/fs/opt amdgpu"
I0507 13:57:16.615904       1 cmdlog.go:71] kmm-worker/modprobe/stderr "msg"="modprobe: FATAL: Module amdgpu not found in directory /var/run/kmm/images/<myregistry>/public/amddriver:v1.0/fs/opt/lib/modules/6.5.5-200.fc38.x86_64"
E0507 13:57:16.616009       1 cmdutils.go:11] kmm-worker "msg"="Fatal error" "error"="error while waiting on the command: exit status 1"

Thank you in advance :)

Regards

commented