PetaLinux 2023.1 / Yocto Langdale build - no .ko files are installed, thus no package is created
DavidAntliff opened this issue · comments
I'm having a lot of trouble getting this to compile/install with PetaLinux 2023.1 (Yocto Langdale). I'm using a basic recipe, copied from the example "hello-mod" and modified accordingly:
# kernel-module-u-dma-buf.bb
SUMMARY = "User space mappable DMA Buffer"
DESCRIPTION = "${SUMMARY}"
LICENSE = "BSD-2-Clause"
LIC_FILES_CHKSUM = "file://LICENSE;md5=bebf0492502927bef0741aa04d1f35f5"
inherit module
BRANCH= "master"
SRC_URI = "git://github.com/ikwzm/udmabuf.git;protocol=https;branch=${BRANCH}"
SRCREV = "3a9a8b883a491ffef5748971f80b24384634832c"
# I've also tried, without success:
#SRCREV = "9b943d49abc9c92a464e4c71e83d1c479ebbf80e"
S = "${WORKDIR}/git"
RPROVIDES:${PN} += "kernel-module-u-dma-buf"
It does compile correctly in PetaLinux 2021.2 (Yocto Gatesgarth), generating a file that contains the correct .ko file:
$ rpm -qlpv ./build/tmp/deploy/rpm/zynqmp_generic/kernel-module-u-dma-buf-5.10.0-xilinx-v2021.2-git-r0.0.zynqmp_generic.rpm
drwxr-xr-x 2 root root 0 Sep 3 2022 /lib
drwxr-xr-x 2 root root 0 Sep 3 2022 /lib/modules
drwxr-xr-x 2 root root 0 Sep 3 2022 /lib/modules/5.10.0-xilinx-v2021.2
drwxr-xr-x 2 root root 0 Sep 3 2022 /lib/modules/5.10.0-xilinx-v2021.2/extra
-rw-r--r-- 1 root root 42640 Sep 3 2022 /lib/modules/5.10.0-xilinx-v2021.2/extra/u-dma-buf.ko
(Ignore the unusual timestamps, bitbake sets them all to the same date to keep the build reproducible).
However, with PetaLinux 2023.1 / Yocto Langdale, such an .rpm file is not generated at all, because the do_install task does not find any modules to package. From what I've been able to determine, this is because - for reasons I don't understand - the build artifact modules.order is removed by the compilation step by this line in the Kernel Makefile:
single_modules: $(single-no-ko) modules_prepare
...
$(Q)rm -f $(MODORDER) # <---!! right here !!
Note that with earlier PetaLinux (2021.2), the modules.order file is not removed by the Kernel Makefile. I haven't figured out why yet.
Anyway, the removal of this file causes the modules_install target to find no modules at this line in Makefile.modinst, and this leads to bitbake not packaging up any files.
I tried a simpler replacement Makefile for this project, and it did build properly:
obj-m := u-dma-buf.o
SRC := $(shell pwd)
all:
$(MAKE) -C $(KERNEL_SRC) M=$(SRC)
modules_install:
$(MAKE) -C $(KERNEL_SRC) M=$(SRC) modules_install
clean:
rm -f *.o *~ core .depend .*.cmd *.ko *.mod.c
rm -f Module.markers Module.symvers modules.order
rm -rf .tmp_versions Modules.symvers
This installs the .ko file during bitbake's do_install, and results in the expected package file:
$ rpm -qlpv ./build/tmp/deploy/rpm/xilinx_zcu208/kernel-module-u-dma-buf-6.1.5-xilinx-v2023.1-1.0-r0.1.xilinx_zcu208.rpm
drwxr-xr-x 2 root root 0 Apr 6 2011 /lib
drwxr-xr-x 2 root root 0 Apr 6 2011 /lib/modules
drwxr-xr-x 2 root root 0 Apr 6 2011 /lib/modules/6.1.5-xilinx-v2023.1
drwxr-xr-x 2 root root 0 Apr 6 2011 /lib/modules/6.1.5-xilinx-v2023.1/extra
-rw-r--r-- 1 root root 52160 Apr 6 2011 /lib/modules/6.1.5-xilinx-v2023.1/extra/u-dma-buf.ko
This leads me to the conclusion that there's something about this project's Makefile that is altering the behaviour of the Kernel compile/install process, resulting in a failed installation and thus no .rpm package is generated.
Note: PetaLinux 2023.1 builds with Linux Kernel 6.1.5-xilinx-v2023.1.
Do you have any insight into what might be causing this?
After inserting SHELL='sh -x' into the oe_runmake_call() function in bitbake's run.do_compile script, I can see the difference between PetaLinux 2021.2 and 2023.1.
oe_runmake_call() {
make -j 16 SHELL='sh -x' KERNEL_SRC=... # snipped
}
First, 2021.2 (working):
# from devshell
$ ../temp/run.do_compile
...
+ make -C .../build/tmp/work-shared/zynqmp-generic/kernel-build-artifacts -f .../build/tmp/work-shared/zynqmp-generic/kernel-source/Makefile u-dma-buf.ko
...
+ make -f .../build/tmp/work-shared/zynqmp-generic/kernel-source/scripts/Makefile.modfinal
+ cat .../build/tmp/work/zynqmp_generic-xilinx-linux/kernel-module-u-dma-buf/git-r0/git/modules.order
...
# modules.order is NOT removed
Then run.do_install:
$ ../temp/run.do_install
...
+ cat .../build/tmp/work/zynqmp_generic-xilinx-linux/kernel-module-u-dma-buf/git-r0/git/modules.order
+ echo ' INSTALL .../build/tmp/work/zynqmp_generic-xilinx-linux/kernel-module-u-dma-buf/git-r0/git/u-dma-buf.ko'
+ mkdir -p .../build/tmp/work/zynqmp_generic-xilinx-linux/kernel-module-u-dma-buf/git-r0/image/lib/modules/5.10.0-xilinx-v2021.2/extra
+ cp .../build/tmp/work/zynqmp_generic-xilinx-linux/kernel-module-u-dma-buf/git-r0/git/u-dma-buf.ko .../build/tmp/work/zynqmp_generic-xilinx-linux/kernel-module-u-dma-buf/git-r0/image/lib/modules/5.10.0-xilinx-v2021.2/extra
...
DEPMOD 5.10.0-xilinx-v2021.2
Now, for 2023.1, (not working):
$ ../temp/run.do_compile
...
make -C .../build/tmp/work-shared/xilinx-zcu208/kernel-source ARCH=arm64 CROSS_COMPILE=aarch64-xilinx-linux- M=.../build/tmp/work/xilinx_zcu208-xilinx-linux/kernel-module-u-dma-buf/1.0-r0 obj-m=u-dma-buf.o u-dma-buf.ko
...
+ make -f .../build/tmp/work-shared/xilinx-zcu208/kernel-source/scripts/Makefile.modfinal
+ cat .../build/tmp/work/xilinx_zcu208-xilinx-linux/u-dma-buf-test/1.0-r0/modules.order
+ :
+ :
+ :
+ rm -f .../build/tmp/work/xilinx_zcu208-xilinx-linux/u-dma-buf-test/1.0-r0/modules.order
...
Now, run.do_install will fail to install anything:
$ ../temp/run.do_install
...
make -C .../build/tmp/work-shared/xilinx-zcu208/kernel-source ARCH=arm64 CROSS_COMPILE=aarch64-xilinx-linux- M=.../build/tmp/work/xilinx_zcu208-xilinx-linux/kernel-module-u-dma-buf/1.0-r0 obj-m=u-dma-buf.o modules_install
...
cat: .../build/tmp/work/xilinx_zcu208-xilinx-linux/kernel-module-u-dma-buf/1.0-r0/modules.order: No such file or directory
DEPMOD .../build/tmp/work/xilinx_zcu208-xilinx-linux/kernel-modules-u-dma-buf/1.0-r0/image/lib/modules/6.1.5-xilinx-v2023.1
...
Note: I've modified the paths above to .../build/... to preserve privacy.
If it's a clue, PetaLinux 2021.2 builds Linux 5.10, and the relevant part of the Makefile is here - note that it does not rm the modules.order (MODORDER) file afterwards, like the 6.1.5 Makefile does:
single_modpost: $(single-no-ko) modules_prepare
$(Q){ $(foreach m, $(single-ko), echo $(extmod-prefix)$m;) } > $(MODORDER)
$(Q)$(MAKE) -f $(srctree)/scripts/Makefile.modpost
As a temporary workaround, I am currently using this bitbake recipe successfully:
SUMMARY = "User space mappable DMA Buffer"
DESCRIPTION = "${SUMMARY}"
LICENSE = "BSD-2-Clause"
LIC_FILES_CHKSUM = "file://LICENSE;md5=bebf0492502927bef0741aa04d1f35f5"
inherit module
SRC_URI = "file://Makefile \
file://LICENSE \
file://u-dma-buf.c \
file://u-dma-buf-mgr.c \
"
S = "${WORKDIR}"
# The inherit of module.bbclass will automatically name module packages with
# "kernel-module-" prefix as required by the oe-core build environment.
RPROVIDES:${PN} += "kernel-module-u-dma-buf kernel-module-u-dma-buf-mgr"
I have copied the LICENSE, u-dma-buf.c and u-dma-buf-mgr.c files into the recipe's files/ directory, and I'm using this Makefile:
obj-m := u-dma-buf.o u-dma-buf-mgr.o
SRC := $(shell pwd)
all:
$(MAKE) -C $(KERNEL_SRC) M=$(SRC)
modules_install:
$(MAKE) -C $(KERNEL_SRC) M=$(SRC) modules_install
clean:
rm -f *.o *~ core .depend .*.cmd *.ko *.mod.c
rm -f Module.markers Module.symvers modules.order
rm -rf .tmp_versions Modules.symvers
This successfully generates:
$ rpm -qlpv ./build/tmp/deploy/rpm/xilinx_zcu208/kernel-module-u-dma-buf-6.1.5-xilinx-v2023.1-1.0-r0.0.xilinx_zcu208.rpm
drwxr-xr-x 2 root root 0 Apr 6 2011 /lib
drwxr-xr-x 2 root root 0 Apr 6 2011 /lib/modules
drwxr-xr-x 2 root root 0 Apr 6 2011 /lib/modules/6.1.5-xilinx-v2023.1
drwxr-xr-x 2 root root 0 Apr 6 2011 /lib/modules/6.1.5-xilinx-v2023.1/extra
-rw-r--r-- 1 root root 52160 Apr 6 2011 /lib/modules/6.1.5-xilinx-v2023.1/extra/u-dma-buf.ko
$ rpm -qlpv ./build/tmp/deploy/rpm/xilinx_zcu208/kernel-module-u-dma-buf-mgr-6.1.5-xilinx-v2023.1-1.0-r0.0.xilinx_zcu208.rpm
drwxr-xr-x 2 root root 0 Apr 6 2011 /etc
drwxr-xr-x 2 root root 0 Apr 6 2011 /etc/modules-load.d
-rw-r--r-- 1 root root 14 Apr 6 2011 /etc/modules-load.d/u-dma-buf-mgr.conf
drwxr-xr-x 2 root root 0 Apr 6 2011 /lib
drwxr-xr-x 2 root root 0 Apr 6 2011 /lib/modules
drwxr-xr-x 2 root root 0 Apr 6 2011 /lib/modules/6.1.5-xilinx-v2023.1
drwxr-xr-x 2 root root 0 Apr 6 2011 /lib/modules/6.1.5-xilinx-v2023.1/extra
-rw-r--r-- 1 root root 11352 Apr 6 2011 /lib/modules/6.1.5-xilinx-v2023.1/extra/u-dma-buf-mgr.ko
Thanks for your valuable information.
I don't use PetaLinux, so I can't solve the problem.
As you pointed out, for PetaLinux 2023.1 it seems that the Makefile needs to be replaced.
Thanks for responding - do you know if this builds & packages correctly in vanilla Yocto Langdale / Linux 6.1.5? I don’t think there’s anything specific to PetaLinux causing this - it’s a kernel makefile interaction issue AFAICT.
Does the following Makefile work?
I don't use PetaLinux, so I can't try it.
# SPDX-License-Identifier: GPL-2.0 OR MIT
# Copyright (C) 2015-2023 Ichiro Kawazome
#
# For in kernel tree variables
#
obj-$(CONFIG_U_DMA_BUF) += u-dma-buf.o
#
# For out of kernel tree variables
#
HOST_ARCH ?= $(shell uname -m | sed -e s/arm.*/arm/ -e s/aarch64.*/arm64/)
ARCH ?= $(shell uname -m | sed -e s/arm.*/arm/ -e s/aarch64.*/arm64/)
ifeq ($(ARCH), arm)
ifneq ($(HOST_ARCH), arm)
CROSS_COMPILE ?= arm-linux-gnueabihf-
endif
endif
ifeq ($(ARCH), arm64)
ifneq ($(HOST_ARCH), arm64)
CROSS_COMPILE ?= aarch64-linux-gnu-
endif
endif
ifdef KERNEL_SRC
KERNEL_SRC_DIR := $(KERNEL_SRC)
else
KERNEL_SRC_DIR ?= /lib/modules/$(shell uname -r)/build
endif
ifndef BUILD_TARGET
ifdef KERNELVERSION
KERNEL_VERSION_LT_5 ?= $(shell echo $(KERNELVERSION) | awk -F. '{print $$1 < 5}')
else
KERNEL_VERSION_LT_5 ?= $(shell awk '/^VERSION/{print int($$3) < 5}' $(KERNEL_SRC_DIR)/Makefile)
endif
ifeq ($(KERNEL_VERSION_LT_5), 1)
BUILD_TARGET ?= modules
endif
endif
ifndef CONFIG_MODULES
CONFIG_MODULES := CONFIG_U_DMA_BUF=m
endif
#
# For out of kernel tree rules
#
all:
$(MAKE) -C $(KERNEL_SRC_DIR) ARCH=$(ARCH) CROSS_COMPILE=$(CROSS_COMPILE) M=$(PWD) $(CONFIG_MODULES) $(BUILD_TARGET)
modules_install:
$(MAKE) -C $(KERNEL_SRC_DIR) ARCH=$(ARCH) CROSS_COMPILE=$(CROSS_COMPILE) M=$(PWD) $(CONFIG_MODULES) modules_install
clean:
$(MAKE) -C $(KERNEL_SRC_DIR) ARCH=$(ARCH) CROSS_COMPILE=$(CROSS_COMPILE) M=$(PWD) clean
Yes, that Makefile does work:
$ temp/run.do_compile
...
+ make -C .../build/tmp/work/xilinx_zcu208-xilinx-linux/u-dma-buf/1.0-r0 CONFIG_U_DMA_BUF=m
...
+ make -f .../build/tmp/work-shared/xilinx-zcu208/kernel-source/scripts/Makefile.modfinal
+ cat .../build/tmp/work/xilinx_zcu208-xilinx-linux/u-dma-buf/1.0-r0/modules.order
+ set -e
+ echo ' CC [M] .../build/tmp/work/xilinx_zcu208-xilinx-linux/u-dma-buf/1.0-r0/u-dma-buf.mod.o'
CC [M] .../build/tmp/work/xilinx_zcu208-xilinx-linux/u-dma-buf/1.0-r0/u-dma-buf.mod.o
...
This leaves the generated file modules.order in place:
$ cat modules.order
.../build/tmp/work/xilinx_zcu208-xilinx-linux/u-dma-buf/1.0-r0/u-dma-buf.ko
Which run.do_install then uses successfully:
$ temp/run.do_install
...
+ cat .../build/tmp/work/xilinx_zcu208-xilinx-linux/u-dma-buf/1.0-r0/modules.order
+ set -e
+ echo ' INSTALL .../build/tmp/work/xilinx_zcu208-xilinx-linux/u-dma-buf/1.0-r0/image/lib/modules/6.1.5-xilinx-v2023.1/extra/u-dma-buf.ko'
...
Resulting in:
$ rpm -qlpv ./build/tmp/deploy/rpm/xilinx_zcu208/kernel-module-*
drwxr-xr-x 2 root root 0 Apr 6 2011 /lib
drwxr-xr-x 2 root root 0 Apr 6 2011 /lib/modules
drwxr-xr-x 2 root root 0 Apr 6 2011 /lib/modules/6.1.5-xilinx-v2023.1
drwxr-xr-x 2 root root 0 Apr 6 2011 /lib/modules/6.1.5-xilinx-v2023.1/extra
-rw-r--r-- 1 root root 52160 Apr 6 2011 /lib/modules/6.1.5-xilinx-v2023.1/extra/u-dma-buf.ko
Do you know why this new one works, and the one on the master branch does not?
Thank you for trying it. I am relieved to hear that it worked properly.
The Makefile at the time of the failed build specified u-dma-buf.ko directly as the target when making the module.
When *.ko is directly specified as the target of make, modules.order seems to be deleted at the end of make.
Thank you for providing the issue.
I will now close this issue.
Thank you.
BTW I anticipate a similar problem with the u-dma-buf-mgr repository too, but I haven't moved onto testing with that yet.
I can confirm that u-dma-buf-mgr works too - thank you!