vclog doesn't seem to work

Question

vclog doesn't seem to work

vfazio opened this issue 10 months ago · comments

I can't get meaningful logs displayed. This is in direct contrast to the old vcdbg.

I have the following deployment:

root@rpi-87fa00:~# vcgencmd version
Apr 25 2023 18:33:04
Copyright (c) 2012 Broadcom
version d7f9c2b4ef7e4a8c0b04374a879ce89d7a948453 (clean) (release) (start_x)

root@rpi-87fa00:~# uname -a
Linux rpi-87fa00 6.1.42-v8 #1 SMP PREEMPT Wed Aug  2 20:49:45 UTC 2023 aarch64 GNU/Linux


root@rpi-87fa00:~# /lib/aarch64-linux-gnu/libc.so.6
GNU C Library (Debian GLIBC 2.36-9+deb12u1) stable release version 2.36.
Copyright (C) 2022 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.
There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A
PARTICULAR PURPOSE.
Compiled by GNU CC version 12.2.0.
libc ABIs: UNIQUE ABSOLUTE
Minimum supported kernel: 3.7.0
For bug reporting instructions, please see:
<http://www.debian.org/Bugs/>.

I've built vclog off of 8fc892d

root@rpi-87fa00:/var/tmp/tmp.MoicJspBpb/utils/build/# git rev-parse HEAD
8fc892d82f7c58e2adc58354cf62a73d1b6c6850

root@rpi-87fa00:/var/tmp/tmp.MoicJspBpb/utils/build# cmake ..
-- The C compiler identification is GNU 12.2.0
-- The CXX compiler identification is GNU 12.2.0
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Check for working C compiler: /usr/bin/cc - skipped
-- Detecting C compile features
-- Detecting C compile features - done
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Check for working CXX compiler: /usr/bin/c++ - skipped
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- Configuring done
-- Generating done
-- Build files have been written to: /var/tmp/tmp.MoicJspBpb/utils/build
root@rpi-87fa00:/var/tmp/tmp.MoicJspBpb/utils/build# make vclog
[ 50%] Building C object vclog/CMakeFiles/vclog.dir/vcdbg.c.o
[100%] Linking C executable vclog
[100%] Built target vclog

but upon execution, i see nothing

root@rpi-87fa00:/var/tmp/tmp.MoicJspBpb/utils/build# ./vclog/vclog -m
root@rpi-87fa00:/var/tmp/tmp.MoicJspBpb/utils/build# ./vclog/vclog -a

when i run the current 64bit statically compiled vcdbg i do get output:

root@rpi-87fa00:/var/tmp/tmp.MoicJspBpb/utils/build# ~/vcdbg log msg
001303.275: brfs: File read: /mfs/sd/config.txt
001303.735: brfs: File read: 229 bytes
001308.978: brfs: File read: /mfs/sd/dt-blob.bin
001337.037: HDMI0:EDID error reading EDID block 0 attempt 0
001338.305: HDMI0:EDID error reading EDID block 0 attempt 1
001339.569: HDMI0:EDID error reading EDID block 0 attempt 2
001340.835: HDMI0:EDID error reading EDID block 0 attempt 3
001342.100: HDMI0:EDID error reading EDID block 0 attempt 4
001343.366: HDMI0:EDID error reading EDID block 0 attempt 5
001344.630: HDMI0:EDID error reading EDID block 0 attempt 6
001345.896: HDMI0:EDID error reading EDID block 0 attempt 7
001347.161: HDMI0:EDID error reading EDID block 0 attempt 8
001348.427: HDMI0:EDID error reading EDID block 0 attempt 9
001349.449: HDMI0:EDID giving up on reading EDID block 0
001349.593: brfs: File read: 68470 bytes
001352.216: brfs: File read: /mfs/sd/config.txt
001352.724: gpioman: gpioman_get_pin_num: pin DISPLAY_SDA not defined
001352.753: gpioman: gpioman_get_pin_num: pin LEDS_PWR_OK not defined
001365.656: gpioman: gpioman_get_pin_num: pin BT_ON not defined
001365.676: gpioman: gpioman_get_pin_num: pin WL_ON not defined
001392.400: gpioman: gpioman_get_pin_num: pin LEDS_PWR_OK not defined

Am i misunderstanding what i should be expecting to see in the output? raspberrypi/firmware#1118 (comment) seems to imply that vclog is the replacement for vcdbg.

Phil Elwell · Answer 1 · Fri Aug 04 2023 23:09:27 GMT+0800 (China Standard Time)

Strange - it's working for me in both 32-bit and 64-bit builds on a 32-bit and 64-bit OSs.

Phil Elwell · Answer 2 · Fri Aug 04 2023 23:10:29 GMT+0800 (China Standard Time)

What does od -Ax -tx1 /proc/device-tree/chosen/log report?

Phil Elwell · Answer 3 · Fri Aug 04 2023 23:13:51 GMT+0800 (China Standard Time)

And if you group together the first four hexadecimal bytes into a big-endian word and interpret that as an address, what is there in memory?

$ od -Ax -tx1 /proc/device-tree/chosen/log
000000 3f f8 00 00 00 07 ff e0
$ sudo busybox devmem 0x3ff80000
0x564C4F47

Vincent Fazio · Answer 4 · Sat Aug 05 2023 00:24:29 GMT+0800 (China Standard Time)

od -Ax -tx1 /proc/device-tree/chosen/log

root@rpi-87fa00:~# od -Ax -tx1 /proc/device-tree/chosen/log
000000 3f f8 00 00 00 07 ff e0
000008

root@rpi-87fa00:~# busybox devmem 0x3ff80000
0x00000000

Phil Elwell · Answer 5 · Sat Aug 05 2023 18:58:01 GMT+0800 (China Standard Time)

Is this running on a Pi 3 of some sort? vclog is lacking a trick that vcdbg has, namely the ability to use DMA to access the top 16MB of RAM that lives where the I/O space is mapped into the ARM address space. This space starts at 0x3f000000, and in this case the logging space starts at 0x3ff8000, which is well inside the 16MB blind spot of the ARM. Only Pi 3 (BCM2710) has this problem - older Pi 1, Pi 2, Zero and Zero 2 all have less RAM, and Pi 4 (BCM2711) puts the peripherals just below the 4GB boundary in the ARM address space, removing the blind spot.

Adding DMA support to vclog is a conceptually relatively small change, and fortunately the existing vclog implementation starts by making a copy of the buffer into some allocated ARM memory, so it shouldn't be to onerous a task.

Vincent Fazio · Answer 6 · Sat Aug 05 2023 19:02:38 GMT+0800 (China Standard Time)

Yes, I meant to add that yesterday, this is an RPi CM 3

Phil Elwell · Answer 7 · Wed Aug 09 2023 00:45:43 GMT+0800 (China Standard Time)

shouldn't be to onerous a task.

Ha - famous last words. It turned out not to be so straightforward, to the extent that it's been virtually reimplemented. I'm not quite ready to check in the changes, but I have a test build for you to try:

https://drive.google.com/file/d/1D3Bv3kL83-t3o_idDOEilxs5C9uHxFfK/view?usp=sharing

Vincent Fazio · Answer 8 · Wed Aug 09 2023 01:34:57 GMT+0800 (China Standard Time)

I'm out of the office for a bit but I'll pull it down and test it on Thursday

Phil Elwell · Answer 9 · Wed Aug 09 2023 02:33:13 GMT+0800 (China Standard Time)

No problem. I forgot to say that it's a 32-bit static build so it should run on any Pi.

Phil Elwell · Answer 10 · Wed Aug 09 2023 18:34:49 GMT+0800 (China Standard Time)

FYI I've just uploaded a build of the release candidate - please use this version for your testing (the link is unchanged): https://drive.google.com/file/d/1D3Bv3kL83-t3o_idDOEilxs5C9uHxFfK/view?usp=sharing

Jordan Speicher · Answer 11 · Thu Aug 10 2023 01:07:52 GMT+0800 (China Standard Time)

Hi, I tried the RC on an RPi CM3+ and found that it doesn't seem to work:

root@rpi-da84b0:~# ./vclog -m
Could not map VC memory: Inappropriate ioctl for device
Fatal error: Cannot access logs
root@rpi-da84b0:~# ./vclog -a
Could not map VC memory: Inappropriate ioctl for device
Fatal error: Cannot access logs

root@rpi-da84b0:~# od -Ax -tx1 /proc/device-tree/chosen/log
000000 3f f8 00 00 00 07 ff e0
000008
root@rpi-da84b0:~# busybox devmem 0x3ff80000
0x00000000

please let me know if there's more info I can provide.

Phil Elwell · Answer 12 · Thu Aug 10 2023 02:41:17 GMT+0800 (China Standard Time)

Unfortunately it relies on having a kernel DMA driver with an ioctl interface, which in this case is the old accelerated framebuffer driver. You won't get that with if the vc4 DRM drivers are enabled.

Does the old vcdbg work?

Vincent Fazio · Answer 13 · Thu Aug 10 2023 05:43:42 GMT+0800 (China Standard Time)

Unfortunately it relies on having a kernel DMA driver with an ioctl interface, which in this case is the old accelerated framebuffer driver. You won't get that with if the vc4 DRM drivers are enabled.

Does the old vcdbg work?

It should. @uSpike should be running a kernel and rootfs with modules built from the in tree bcmrpi3_defconfig from upstream rpi 5.15.y like I was. We build our own rootfs and have been using the static build but want to switch to open sources if possible.

If there are additional modules necessary, can you call out the kconfig flags? This wasn't required for the old vcdbg method.

Phil Elwell · Answer 14 · Thu Aug 10 2023 17:25:02 GMT+0800 (China Standard Time)

It should.

It would be really helpful if you could confirm that - from what I can see they should behave the same, at least at that basic level.

Either way, run the following commands and post the output:

$ sudo strace vcdbg log msg |& head -150
$ sudo strace vclog -m |& head -25
$ grep -s . /sys/class/graphics/fb0/*

Vincent Fazio · Answer 15 · Thu Aug 10 2023 20:03:13 GMT+0800 (China Standard Time)

Well, you were right but now i'm even more confused. Here's what i see on a failing board that's running 5.15:

root@rpi-2828da:~# uname -a
Linux rpi-2828da 5.15.84-v8 #1 SMP PREEMPT Mon Apr 17 22:57:24 UTC 2023 aarch64 GNU/Linux

root@rpi-2828da:~# vcgencmd version
Jan  5 2023 10:49:28 
Copyright (c) 2012 Broadcom
version 8ba17717fbcedd4c3b6d4bce7e50c7af4155cba9 (clean) (release) (start_x)

root@rpi-2828da:~# vcdbg log msg
debug_sym: AccessVideoCoreMemory: Memory address 0x3900001f + numBytes 0x99afffe2 is > memory end 0x7ebfffff
Unable to read logging from 0xb900001f

root@rpi-2828da:~# grep -s . /sys/class/graphics/fb0/*
/sys/class/graphics/fb0/bits_per_pixel:32
/sys/class/graphics/fb0/dev:29:0
/sys/class/graphics/fb0/modes:U:656x416p-0
/sys/class/graphics/fb0/name:simple
/sys/class/graphics/fb0/pan:0,0
/sys/class/graphics/fb0/rotate:0
/sys/class/graphics/fb0/state:0
/sys/class/graphics/fb0/stride:2624
/sys/class/graphics/fb0/uevent:MAJOR=29
/sys/class/graphics/fb0/uevent:MINOR=0
/sys/class/graphics/fb0/uevent:DEVNAME=fb0
/sys/class/graphics/fb0/virtual_size:656,416

root@rpi-2828da:~# cat /proc/device-tree/model 
Raspberry Pi Compute Module 3 Plus Rev 1.0

And on a successful board:

root@rpi-87fa00:/mnt/persistent# uname -a
Linux rpi-87fa00 6.1.42-v8 #1 SMP PREEMPT Wed Aug  2 20:49:45 UTC 2023 aarch64 GNU/Linux

root@rpi-87fa00:/mnt/persistent# vcgencmd version
Apr 25 2023 18:33:04
Copyright (c) 2012 Broadcom
version d7f9c2b4ef7e4a8c0b04374a879ce89d7a948453 (clean) (release) (start_x)

root@rpi-87fa00:/mnt/persistent# grep -s . /sys/class/graphics/fb0/*
/sys/class/graphics/fb0/bits_per_pixel:32
/sys/class/graphics/fb0/dev:29:0
/sys/class/graphics/fb0/modes:U:656x416p-0
/sys/class/graphics/fb0/name:BCM2708 FB
/sys/class/graphics/fb0/pan:0,0
/sys/class/graphics/fb0/rotate:0
/sys/class/graphics/fb0/state:0
/sys/class/graphics/fb0/stride:2624
/sys/class/graphics/fb0/uevent:MAJOR=29
/sys/class/graphics/fb0/uevent:MINOR=0
/sys/class/graphics/fb0/uevent:DEVNAME=fb0
/sys/class/graphics/fb0/virtual_size:656,416

root@rpi-87fa00:/mnt/persistent# ./vcdbg log msg
001239.890: brfs: File read: /mfs/sd/config.txt
001240.350: brfs: File read: 229 bytes
001245.617: brfs: File read: /mfs/sd/dt-blob.bin
001273.716: HDMI0:EDID error reading EDID block 0 attempt 0
001274.983: HDMI0:EDID error reading EDID block 0 attempt 1
001276.247: HDMI0:EDID error reading EDID block 0 attempt 2
001277.513: HDMI0:EDID error reading EDID block 0 attempt 3
001278.778: HDMI0:EDID error reading EDID block 0 attempt 4
001280.044: HDMI0:EDID error reading EDID block 0 attempt 5
001281.309: HDMI0:EDID error reading EDID block 0 attempt 6
001282.575: HDMI0:EDID error reading EDID block 0 attempt 7
001283.840: HDMI0:EDID error reading EDID block 0 attempt 8
001285.106: HDMI0:EDID error reading EDID block 0 attempt 9
001286.128: HDMI0:EDID giving up on reading EDID block 0

root@rpi-87fa00:/mnt/persistent# ./vclog-static -m | head -n 15
001239.890: brfs: File read: /mfs/sd/config.txt
001240.350: brfs: File read: 229 bytes
001245.617: brfs: File read: /mfs/sd/dt-blob.bin
001273.716: HDMI0:EDID error reading EDID block 0 attempt 0
001274.983: HDMI0:EDID error reading EDID block 0 attempt 1
001276.247: HDMI0:EDID error reading EDID block 0 attempt 2
001277.513: HDMI0:EDID error reading EDID block 0 attempt 3
001278.778: HDMI0:EDID error reading EDID block 0 attempt 4
001280.044: HDMI0:EDID error reading EDID block 0 attempt 5
001281.309: HDMI0:EDID error reading EDID block 0 attempt 6
001282.575: HDMI0:EDID error reading EDID block 0 attempt 7
001283.840: HDMI0:EDID error reading EDID block 0 attempt 8
001285.106: HDMI0:EDID error reading EDID block 0 attempt 9
001286.128: HDMI0:EDID giving up on reading EDID block 0
001286.272: brfs: File read: 68470 bytes

root@rpi-87fa00:/mnt/persistent# cat /proc/device-tree/model
Raspberry Pi Compute Module 3 Plus Rev 1.0

Retesting that "working" board on 5.15 shows it doesn't work:

root@rpi-87fa00:~# uname -a
Linux rpi-87fa00 5.15.84-v8 #1 SMP PREEMPT Mon Apr 17 22:57:24 UTC 2023 aarch64 GNU/Linux

root@rpi-87fa00:~# vcgencmd version
Jan  5 2023 10:49:28
Copyright (c) 2012 Broadcom
version 8ba17717fbcedd4c3b6d4bce7e50c7af4155cba9 (clean) (release) (start_x)

root@rpi-87fa00:~# strace -f vcdbg log msg
execve("/usr/bin/vcdbg", ["vcdbg", "log", "msg"], 0x7fc7c8b248 /* 16 vars */ <unfinished ...>
[ Process PID=1358 runs in 32 bit mode. ]
strace: WARNING: Proper structure decoding for this personality is not supported, please consider building strace with mpers support enabled.
<... execve resumed>)                   = 0
uname({sysname="Linux", nodename="rpi-87fa00", ...}) = 0
brk(NULL)                               = 0x4f5000
brk(0x4f5d10)                           = 0x4f5d10
set_tls(0x4f54c0)                       = 0
set_tid_address(0x4f5068)               = 1358
set_robust_list(0x4f5070, 12)           = 0
rt_sigaction(SIGRTMIN, {sa_handler=0x6b198, sa_mask=[], sa_flags=SA_RESTORER|SA_SIGINFO, sa_restorer=0x79b70}, NULL, 8) = 0
rt_sigaction(SIGRT_1, {sa_handler=0x6b05c, sa_mask=[], sa_flags=SA_RESTORER|SA_RESTART|SA_SIGINFO, sa_restorer=0x79b70}, NULL, 8) = 0
rt_sigprocmask(SIG_UNBLOCK, [RTMIN RT_1], NULL, 8) = 0
ugetrlimit(RLIMIT_STACK, {rlim_cur=8192*1024, rlim_max=RLIM_INFINITY}) = 0
readlink("/proc/self/exe", "/usr/bin/vcdbg", 4096) = 14
brk(0x516d10)                           = 0x516d10
brk(0x517000)                           = 0x517000
futex(0x195b30, FUTEX_WAKE_PRIVATE, 2147483647) = 0
futex(0x1962dc, FUTEX_WAKE_PRIVATE, 2147483647) = 0
open("/dev/vc-mem", O_RDWR|O_SYNC|O_LARGEFILE) = 3
ioctl(3, FS_IOC32_GETVERSION or FS_IOC_GETVERSION, 0x4f5fec) = 0
ioctl(3, _IOC(_IOC_READ, 0x76, 0x2, 0x4), 0x4f5fe0) = 0
ioctl(3, _IOC(_IOC_READ, 0x76, 0x3, 0x4), 0x4f5fe4) = 0
ioctl(3, _IOC(_IOC_READ, 0x76, 0, 0x4), 0x4f5ff0) = 0
open("/dev/fb0", O_RDWR|O_SYNC|O_LARGEFILE) = 4
ioctl(4, _IOC(_IOC_WRITE, 0x7a, 0x22, 0xc), 0xfff4d5e4) = -1 ENOTTY (Inappropriate ioctl for device)
close(4)                                = 0
mmap2(NULL, 4096, PROT_READ, MAP_SHARED, 3, 0x3ec02000) = 0xf7a7a000
munmap(0xf7a7a000, 4096)                = 0

I could have sworn that I had vcdbg working at one point on these boards, otherwise i wouldn't have deployed the binary, but i can't find any logs in my old commits to prove that except for output from vcdbg version which wouldn't have exercised this interface...

Regardless, this is an odd quirk of behavior that it works on a board for one kernel but not another. the loaded kernel module list looks largely the same.

Phil Elwell · Answer 16 · Thu Aug 10 2023 20:20:45 GMT+0800 (China Standard Time)

At least that makes sense - the difference is in the framebuffer driver. bcm2708_fb has the magic ioctl(), simplefb doesn't. This has always been the case, it's just that bcm2708_fb is being used less now we have working DRM drivers.

It may be that we have to graft the DMA support into another driver - vc_mem, perhaps - or find a way of disabling the accelerated framebuffer without sacrificing the DMA support.

Vincent Fazio · Answer 17 · Thu Aug 10 2023 20:42:20 GMT+0800 (China Standard Time)

Well, it's probably pretty apparent how often we use this utility if it's just now coming up. Maybe I don't worry about it for now as we're pending a migration to 6.1 at some point but that has a rather large testing surface area so it won't be soon.

I do have FB_BCM2708 built into this kernel (it's in the default config). is there other magic necessary?

root@rpi-87fa00:~# zcat /proc/config.gz | grep BCM2708
CONFIG_BCM2708_VCMEM=y
CONFIG_I2C_BCM2708=m
CONFIG_I2C_BCM2708_BAUDRATE=100000
CONFIG_FB_BCM2708=y
CONFIG_SND_BCM2708_SOC_CHIPDIP_DAC=m
CONFIG_SND_BCM2708_SOC_GOOGLEVOICEHAT_SOUNDCARD=m
CONFIG_SND_BCM2708_SOC_HIFIBERRY_DAC=m
CONFIG_SND_BCM2708_SOC_HIFIBERRY_DACPLUS=m
CONFIG_SND_BCM2708_SOC_HIFIBERRY_DACPLUSHD=m
CONFIG_SND_BCM2708_SOC_HIFIBERRY_DACPLUSADC=m
CONFIG_SND_BCM2708_SOC_HIFIBERRY_DACPLUSADCPRO=m
CONFIG_SND_BCM2708_SOC_HIFIBERRY_DACPLUSDSP=m
CONFIG_SND_BCM2708_SOC_HIFIBERRY_DIGI=m
CONFIG_SND_BCM2708_SOC_HIFIBERRY_AMP=m
CONFIG_SND_BCM2708_SOC_PIFI_40=m
CONFIG_SND_BCM2708_SOC_RPI_CIRRUS=m
CONFIG_SND_BCM2708_SOC_RPI_DAC=m
CONFIG_SND_BCM2708_SOC_RPI_PROTO=m
CONFIG_SND_BCM2708_SOC_JUSTBOOM_BOTH=m
CONFIG_SND_BCM2708_SOC_JUSTBOOM_DAC=m
CONFIG_SND_BCM2708_SOC_JUSTBOOM_DIGI=m
CONFIG_SND_BCM2708_SOC_IQAUDIO_CODEC=m
CONFIG_SND_BCM2708_SOC_IQAUDIO_DAC=m
CONFIG_SND_BCM2708_SOC_IQAUDIO_DIGI=m
CONFIG_SND_BCM2708_SOC_I_SABRE_Q2M=m
CONFIG_SND_BCM2708_SOC_ADAU1977_ADC=m
CONFIG_SND_BCM2708_SOC_DIONAUDIO_LOCO=m
CONFIG_SND_BCM2708_SOC_DIONAUDIO_LOCO_V2=m
CONFIG_SND_BCM2708_SOC_ALLO_PIANO_DAC=m
CONFIG_SND_BCM2708_SOC_ALLO_PIANO_DAC_PLUS=m
CONFIG_SND_BCM2708_SOC_ALLO_BOSS_DAC=m
CONFIG_SND_BCM2708_SOC_ALLO_BOSS2_DAC=m
CONFIG_SND_BCM2708_SOC_ALLO_DIGIONE=m
CONFIG_SND_BCM2708_SOC_ALLO_KATANA_DAC=m
CONFIG_SND_BCM2708_SOC_FE_PI_AUDIO=m
CONFIG_DMA_BCM2708=y

Vincent Fazio · Answer 18 · Thu Aug 10 2023 20:45:25 GMT+0800 (China Standard Time)

Also, i think it's fair to say that your changes work, which was the original scope of this issue. I don't want to derail this too much but i think the additional conversation may be useful for other users.

Phil Elwell · Answer 19 · Thu Aug 10 2023 20:59:11 GMT+0800 (China Standard Time)

The module is included in all our kernel builds, but using either dtoverlay=vc4-kms-v3d or dtoverlay=vc4-fkms-v3d prevents the device from being instantiated. The firmware detects that, and enables the simplefb driver in its place. I suspect we would end up with:

an extension to the bcm2708_fb driver that just presents the DMA interface (perhaps triggered by a different compatible string),
modified overlays that create an instance of the new DMA-only framebuffer as well as disabling the old one, and
a slight tweak to vclog that looks for a different /dev/ name.

Given that list, extending vc_mem sounds easier.

Phil Elwell · Answer 20 · Thu Aug 10 2023 21:01:27 GMT+0800 (China Standard Time)

By the way, if you aren't particularly memory constrained in your application, you can bypass all this nastiness (at least while you're debugging) by putting total_mem=1008 in config.txt...

Vincent Fazio · Answer 21 · Thu Aug 10 2023 21:19:38 GMT+0800 (China Standard Time)

By the way, if you aren't particularly memory constrained in your application, you can bypass all this nastiness (at least while you're debugging) by putting total_mem=1008 in config.txt...

I can confirm this works but i'm not overly sure of the implications of doing this for a production deployment.

These are 1GB CM3 Lites and we do often run into memory pressure which requires us to page to disk so this may not be something we can do in the wild but my guess is that this is just chopping off an additional 16MB of RAM in our case?

Vincent Fazio · Answer 22 · Thu Aug 10 2023 21:21:33 GMT+0800 (China Standard Time)

Also, for posterity, our config.txt. we do not enable those overlays:

root@rpi-87fa00:/mnt/persistent# cat /boot/config.txt
dtparam=audio=on
dtparam=spi=on
dtparam=i2c_arm=on
dtparam=i2c1=on
dtoverlay=disable-bt
dtoverlay=disable-wifi
kernel=u-boot.bin
enable_uart=1
arm_64bit=1
start_x=1

[pi3+]
# Model 3B
[board-type=0x8]
cmdline=cmdline-noserial.txt

Phil Elwell · Answer 23 · Thu Aug 10 2023 21:46:13 GMT+0800 (China Standard Time)

this is just chopping off an additional 16MB of RAM in our case?

Yes it is.

Vincent Fazio · Answer 24 · Fri Aug 11 2023 04:20:50 GMT+0800 (China Standard Time)

The module is included in all our kernel builds, but using either dtoverlay=vc4-kms-v3d or dtoverlay=vc4-fkms-v3d prevents the device from being instantiated. The firmware detects that, and enables the simplefb driver in its place. I suspect we would end up with:
1. an extension to the bcm2708_fb driver that just presents the DMA interface (perhaps triggered by a different compatible string),

2. modified overlays that create an instance of the new DMA-only framebuffer as well as disabling the old one, and

3. a slight tweak to vclog that looks for a different /dev/ name.
Given that list, extending vc_mem sounds easier.

@pelwell are there additional tasks you want to drive from this issue? Or has everything been implemented to your satisfaction in 94b89a6 ?

Also, as i mentioned, we do not load the overlays you mentioned, but i assume we're still falling victim to the simplefb driver being loaded and not being able to service this ioctl.

Phil Elwell · Answer 25 · Fri Aug 11 2023 05:24:17 GMT+0800 (China Standard Time)

I think we can close the issue, but I've made a note of the vcdbg/vclog vs kms driver issue.

I haven't figured out why you don't have the frame buffer instantiated, but I'm suspicious about u-boot's role in dtb handling.

Vincent Fazio · Answer 26 · Tue Aug 22 2023 00:14:05 GMT+0800 (China Standard Time)

I think we can close the issue, but I've made a note of the vcdbg/vclog vs kms driver issue.

I haven't figured out why you don't have the frame buffer instantiated, but I'm suspicious about u-boot's role in dtb handling.

I haven't chased it down yet, but i can confirm that if i modify my config.txt:

root@rpi-87fa00:~# uname -a
Linux rpi-87fa00 5.15.84-v8 #1 SMP PREEMPT Fri Aug 18 20:37:52 UTC 2023 aarch64 GNU/Linux
root@rpi-87fa00:~# cat /boot/config.txt
dtparam=audio=on
dtparam=spi=on
dtparam=i2c_arm=on
dtparam=i2c1=on
dtoverlay=disable-bt
dtoverlay=disable-wifi
kernel=Image
enable_uart=1
arm_64bit=1
start_x=1
cmdline=cmdline.txt

[pi3+]
# Model 3B
[board-type=0x8]
cmdline=cmdline-noserial.txt
root@rpi-87fa00:~# cat /boot/cmdline.txt
console=ttyAMA0,115200 kgdboc=ttyAMA0,115200 console=tty1 rootfstype=ext4 rootwait root=/dev/mmcblk0p5 cma=64M

That things seem to work...

root@rpi-87fa00:~# vclog -m | head -n20
001262.506: brfs: File read: /mfs/sd/config.txt
001262.986: brfs: File read: 245 bytes
001268.448: brfs: File read: /mfs/sd/dt-blob.bin
001296.651: HDMI0:EDID error reading EDID block 0 attempt 0
001297.919: HDMI0:EDID error reading EDID block 0 attempt 1
001299.183: HDMI0:EDID error reading EDID block 0 attempt 2
001300.449: HDMI0:EDID error reading EDID block 0 attempt 3
001301.714: HDMI0:EDID error reading EDID block 0 attempt 4
001302.980: HDMI0:EDID error reading EDID block 0 attempt 5
001304.244: HDMI0:EDID error reading EDID block 0 attempt 6
001305.510: HDMI0:EDID error reading EDID block 0 attempt 7
001306.775: HDMI0:EDID error reading EDID block 0 attempt 8
001308.041: HDMI0:EDID error reading EDID block 0 attempt 9
001309.063: HDMI0:EDID giving up on reading EDID block 0
001309.211: brfs: File read: 68470 bytes
001311.975: brfs: File read: /mfs/sd/config.txt
001312.504: gpioman: gpioman_get_pin_num: pin DISPLAY_SDA not defined
001312.534: gpioman: gpioman_get_pin_num: pin LEDS_PWR_OK not defined
001325.337: gpioman: gpioman_get_pin_num: pin BT_ON not defined
001325.357: gpioman: gpioman_get_pin_num: pin WL_ON not defined

So there's something about the handoff from firmare to uboot to linux causing the problem.

I dumped /proc/device-tree between the two:

root@rpi-87fa00:/mnt/persistent# diff linux_dev_tree.dts uboot_dev_tree.dts
1116c1116,1117
<               bootargs = "coherent_pool=1M snd_bcm2835.enable_compat_alsa=0 snd_bcm2835.enable_hdmi=1 bcm2708_fb.fbwidth=656 bcm2708_fb.fbheight=416 bcm2708_fb.fbswap=1 smsc95xx.macaddr=B8:27:EB:87:FA:00 vc_mem.mem_base=0x3ec00000 vc_mem.mem_size=0x40000000  console=ttyAMA0,115200 kgdboc=ttyAMA0,115200 console=tty1 rootfstype=ext4 rootwait root=/dev/mmcblk0p5 cma=64M";
---
>               u-boot,version = "2023.01";
>               bootargs = "coherent_pool=1M snd_bcm2835.enable_compat_alsa=0 snd_bcm2835.enable_hdmi=1 bcm2708_fb.fbwidth=656 bcm2708_fb.fbheight=416 bcm2708_fb.fbswap=1 smsc95xx.macaddr=B8:27:EB:87:FA:00 vc_mem.mem_base=0x3ec00000 vc_mem.mem_size=0x40000000  console=ttyAMA0,115200 kgdboc=ttyAMA0,115200 console=tty1 rootfstype=ext4 rootwait root=/dev/mmcblk0p5 cma=64M root=/dev/mmcblk0p5 cma=64M";
1134a1136,1145
>       };
>
>       framebuffer@3eadf000 {
>               format = "a8r8g8b8";
>               height = <0x1a0>;
>               width = <0x290>;
>               stride = <0xa40>;
>               compatible = "simple-framebuffer";
>               status = "okay";
>               reg = <0x3eadf000 0x10a800>;

I'm guessing it's that framebuffer node causing the grief?

Vincent Fazio · Answer 27 · Tue Aug 22 2023 00:29:40 GMT+0800 (China Standard Time)

For posterity, the framebuffer node gets added here: https://source.denx.de/u-boot/u-boot/-/blob/master/board/raspberrypi/rpi/rpi.c#L563

This "fixup" can be avoided if CONFIG_OF_BOARD_SETUP is not set.

It's still a little curious to me that everything continues to work fine in RPi-Linux 6.1 despite this node being present