ChirpStack Gateway OS Images for RPi4 and CM4 do not boot (v3.5.1 and v3.5.0)

Question

ChirpStack Gateway OS Images for RPi4 and CM4 do not boot (v3.5.1 and v3.5.0)

mars96 opened this issue 3 years ago · comments

The issue is present in the latest release (v3.5.1 / 2021-09-20).
I have searched the issues of this repository and believe that this is not a duplicate.
The issue has also been reported in the ChirpStack Forum but it seems nobody really cares: Chirpstack OS – Raspberry pi (4) - Does not boot

What happened?

The latest Raspberry Pi ChirpStack OS (base) image has been flashed onto SDCard (for Raspberry Pi 4) or onto the integrated eMMC (in my case 32 GB) on the Raspberry Pi Compute Module 4 (CM4).

In case the CM4 is not known: The Raspberry Pi CM4 is a Raspberry Pi 4 without an IO-Board for external connections. The CM4 Module can be placed on different IO-Boards available so it is acually an Raspberry Pi 4B (depending on the version with max 8 GB RAM and max 32 GB of eMMC flash memory, which is faster and more reliable compared to an SDCard).

The system is not starting with the flashed image.

What did you expect?

The system should boot, but it does not. The colored box is flashing on the monitor and that's it.
On the older image (v3.4.0 / 2021-05-10), which has also been tested by me with the CM4 module, the boot process starts but ends with a "Kernel panic". This is most likely the case because that image is missing the *.dtb file for the CM4 module.

Steps to reproduce this issue

Steps:

Download image
Extract *.gz image file
Start Win32DiskImager (or another Flash-Tool) on Windows or Linux
Write image to SDCard or eMMC flash memory (CM4 is booted via USB using RPiBoot, so the eMMC drive is showing up on the system)
Start system with the just flashed image
Boot process should start (output on the monitor should display a raspberry image and log output of the starting system - which is not happening)

Could you share your log output?

There is no output as the boot process is not even started.
For v3.4.0 (2021-05-10) there is a Kernel panic for the CM4 boot process where I could share a screenshot showing the last few lines of the crash, but as the *.dtb file for the CM4 is not in the /boot directory the error might be related to the fact that the image is not supporting the Raspi CM4 module in the first place, which is different for the v3.5.x releases where the *.dtb file for the CM4 module is listed in the /boot directory.

Component	Version
ChirpStack Gateway OS	v3.5.1
ChirpStack Gateway OS	v3.5.0

Orne Brocaar · Answer 1 · Tue Jan 04 2022 05:42:33 GMT+0800 (China Standard Time)

Currently the ChirpStack Gateway OS does not support the CM4.

Juergen · Answer 2 · Tue Jan 04 2022 08:15:47 GMT+0800 (China Standard Time)

Hi Orne,
thank you for the reply.

The problem that the image is not working for the standard Raspberry Pi 4 model has been reported by 2 people (akshay and paredlund) in the ChirpStack forum (see link above) and not just only on the Raspberry Pi CM4 I'm using.
I just ordered a standard RPi 4B with 4 GB RAM (with case and memory card) to be able to verify this myself...basically only for this bug report.
It should arrive on Thursday or Friday this week. As I don't have time anyway until then I'll get back to this ticket once the board arrives. Maybe in the meantime one with the standard RPi 4 adds a comment to this as well.

The CM4 and also the CM400 (the Keyboard with integrated RPi4) Overlays (*.dtd files) have been added to the yocto project "hardknott" (in Feb. 2021) on which the last two ChirpStack Gateway OS images are based on as far as I can tell.
https://git.yoctoproject.org/meta-raspberrypi/commit/?id=0c85f0150629e1f5eaf86289f2542744e38b5413

There have been some additional commits for example for the WiFi/BT module in mid November 2021 but I see no reason why it should not work on the CM4, as it now does on other Linux images created for the standard RPi 4.
https://git.yoctoproject.org/meta-raspberrypi/log/

Kind regards,
Juergen

Juergen · Answer 3 · Sat Jan 08 2022 02:22:32 GMT+0800 (China Standard Time)

Ok, I've received the "standard" Raspberry Pi 4 with 4 GB RAM today and tested the image (ChirpStack Gateway OS Base) using a microSD card and it shows the same behaviour as my CM4 based system.
As I said before the current ChirpStack Gateway OS (v3.5.1 - the version vom end of September 2021) is not working.
In the forum now 4 people (now including me) have reported this issue with the standard Raspberry Pi 4 + me with also the CM4-version.

All show the same issue:
The colored square is shown on the connected screen and that's it.

I'm going to look if I can get some information from the logs which might be helpful when I'm home.

Kind regards,
Juergen

Marvin Johanning · Answer 4 · Tue Jan 18 2022 00:45:13 GMT+0800 (China Standard Time)

I have a standard RPi 4 as well (several, actually) and I can confirm that GatewayOS does not boot, at least on the 3.5.1 version of it. The images for the RPi 3 work like a charm on my RPi 3 B+, however. I will try some of the older images tomorrow and see if they work.

tonysmith55 · Answer 5 · Tue Jan 18 2022 05:39:27 GMT+0800 (China Standard Time)

FYI, the Raspi OS was updated when the Pi4 was released. As a result anything that's built before a Pi4 was released will not work on a Pi4. An OS for a Pi4 is backward compatible and will work on a Pi3 or anything earlier. In fact it will work all the way back to Pi1 which is the same chip as PiZero. Hope this helps.

Marvin Johanning · Answer 6 · Tue Jan 18 2022 05:44:37 GMT+0800 (China Standard Time)

@tonysmith55, I am not entirely sure what you are referring to. There are specific versions of GatewayOS for the RPi 4 and I expected those to actually work on it. When installing GatewayOS on my Raspberry Pi 4, I made sure to use the images built for that version of the Pi and not the one for the Raspberry Pi 3 or earlier.

tonysmith55 · Answer 7 · Tue Jan 18 2022 05:50:08 GMT+0800 (China Standard Time)

@mjohanning99, Understand, but the original symptom is exactly the same as seen when using a Pi3 OS on a Pi4. I must admit I also expect the GatewayOS for a RPi 4 should work but I simply shared some information just in case it helps.

Orne Brocaar · Answer 8 · Mon Jan 31 2022 21:51:04 GMT+0800 (China Standard Time)

I have tried to reproduce the issue on my Raspberry Pi 4 (model B), but it runs fine for me.

Images tested:

Steps I have tried (and which are working for me):

Using Balena Etcher, I flash one of the above images on a SD Card (without extracting the .gz, you can use the file as-is using Etcher).
- I download one of the above images
- Within Balena Etcher I click Flash from file and select the .wic.gz file
I have nothing connected to my Pi, except a RAK concentrator shield. Thus no keyboard, HDMI, ...
After inserting the SD Card and connecting the Pi to a power-source it boots fine
On my laptop, I can see the ChirpStackAP in the list of Wi-Fi networks and I can connect to it
I can ssh into the Pi using admin@192.168.0.1 and I can configure it using the sudo gateway-config command

Note: I'm aware that the CM4 is very similar in hardware but this is not officially supported by ChirpStack. It might work without any changes, but please note that the Yocto builds are optimized for a single machine only (where the Raspberry Pi OS work on multiple targets). To request CM4 support, please create a separate GitHub issue.

Orne Brocaar · Answer 9 · Mon Jan 31 2022 21:59:33 GMT+0800 (China Standard Time)

As an indication if the Pi boots, you can look at the green LED which is next to the red LED on the Pi. After connecting the Pi to a power-source it shortly blinks rapidly which is when the bootloader is loaded from the SD card. Then it is off for a few seconds (fixed time in u-boot, during which you normally could interrupt the bootloader from continuing the boot process), after which it continues to blink rapidly for a couple of seconds which is when the kernel is loaded and the OS is started.

Of course this doesn't expose any details in case of an issue, but it gives some idea

swap-file · Answer 10 · Tue Feb 08 2022 00:01:53 GMT+0800 (China Standard Time)

Pi4 that are HW rev 1.4 and newer removed the onboard VL805 EEPROM. This changed the boot process.

All 8GB Pi4s are HW rev 1.4, but rev 1.4 is slowly replacing the smaller memory size versions too.

It might be useful to make sure everyone is testing on the same HW rev.

Orne Brocaar · Answer 11 · Tue Feb 08 2022 00:13:36 GMT+0800 (China Standard Time)

Thanks @Swap-File, that is really helpful information and it probably explains why I was not able to reproduce this.

Marvin Johanning · Answer 12 · Wed Feb 09 2022 20:23:22 GMT+0800 (China Standard Time)

Thank you @Swap-File for the recommendation. I just looked at the output of cat /proc/device-tree/model which told me that I have a Raspberry Pi 4 Model B Rev 1.4. So I am guessing that I, too, have HW Rev. 1.4 and the supplied image is not working for me.

Juergen · Answer 13 · Tue Apr 05 2022 18:07:56 GMT+0800 (China Standard Time)

I can also point the newer stepping "C0" of the Broadcom chipset which replaces the "B0" stepping and which causes the issue.
I've updated the post on the ChirpStack forum with details.
https://forum.chirpstack.io/t/chirpstack-os-raspberry-pi-does-not-boot/12822/19

My device is reporting this (“C0” stepping)…the actual Raspberry Pi 4 4GB I have here and also my Raspberry Pi CM4 module:

/home/pi# od -An -tx1 /proc/device-tree/emmc2bus/dma-ranges
 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
 fc 00 00 00

The older Broadcom SOC stepping “B0” reports:

 00 00 00 00 c0 00 00 00 00 00 00 00 00 00 00 00
 40 00 00 00

One of the first reports regarding this issue 1 year ago point to "u-boot" can be found here:
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=255080

So just rebuilding the image with an updated yocto base should fix it.

Orne Brocaar · Answer 14 · Tue Apr 05 2022 22:48:20 GMT+0800 (China Standard Time)

Thanks @mars96, updating the Yocto version is on my backlog. Also I have ordered a Pi 4 8GB a few months ago to reproduce the issue, but with the shortages it hasn't been shipped yet...

Orne Brocaar · Answer 15 · Mon Apr 11 2022 19:27:16 GMT+0800 (China Standard Time)

Just to let you know that I have started working on integrating the (current) latest Yocto version: https://github.com/brocaar/chirpstack-gateway-os/tree/honister.

Orne Brocaar · Answer 16 · Mon Apr 25 2022 16:33:00 GMT+0800 (China Standard Time)

Unfortunately I have not yet received my Pi 4 8GB so I have not yet been able to test this, but thanks to the initial work of @mars96 I have upgraded ChirpStack Gateway OS to the latest Yocto version.

Could you please test? https://artifacts.chirpstack.io/downloads/chirpstack-gateway-os/raspberrypi/raspberrypi4/4.0.0-test.1/

Note: this will also give you the opportunity to test the upcoming ChirpStack v4 version :-)

Orne Brocaar · Answer 17 · Tue May 03 2022 16:55:33 GMT+0800 (China Standard Time)

@mars96 would you be able to verify if the above image boots on the latest Pi 4?

Neoroxxor · Answer 18 · Tue May 03 2022 21:02:30 GMT+0800 (China Standard Time)

@brocaar i was able to boot with your https://artifacts.chirpstack.io/downloads/chirpstack-gateway-os/raspberrypi/raspberrypi4/4.0.0-test.1/ OS

Im now on my way to configure everything and give it a try.
I will give an update later

Juergen · Answer 19 · Thu May 05 2022 19:43:44 GMT+0800 (China Standard Time)

@mars96 would you be able to verify if the above image boots on the latest Pi 4?

@brocaar Thank you for the update.
I can check my Raspi CM4 and standard Raspi 4 tomorrow evening (currently swamped with work).

Neoroxxor · Answer 20 · Thu May 05 2022 20:46:21 GMT+0800 (China Standard Time)

Update 001
I was able to connect a device to the gateway. On this level everything works fine.
Messages send and receive on both devices.
Some changes will not be saved and after reboot they are set do default.

Not able to test now: internal PI: WIFI, GPS, DHCP,
external: DB, VPN, connection to another AP-Server

but im working on it ;)

Juergen · Answer 21 · Fri May 06 2022 06:35:02 GMT+0800 (China Standard Time)

@mars96 would you be able to verify if the above image boots on the latest Pi 4?

@brocaar Thank you for the update. I can check my Raspi CM4 and standard Raspi 4 tomorrow evening (currently swamped with work).

Ok, here already the first short reply on the image:

Raspberry Pi CM4 module (8 GB RAM, 32 GB eMMC) - System is booting normally from eMMC
Raspberry Pi 4 (4 GB RAM, 32 GB microSD card) - System is booting normally from microSD card

I've not yet done any configuration. This will follow in about 20+ hours, but that the system is booting normally from eMMC on the Raspberry Pi CM4 module as well as the Raspberry PI 4 both using the newer Chip revision is good,

Orne Brocaar · Answer 22 · Thu Nov 24 2022 21:25:15 GMT+0800 (China Standard Time)

I'm closing this issue as I believe it has been solved by upgrading Yocto + the Raspberry Pi BSP layer to the latest version (included in the v4 release, starting with v4.0.0-test.1 image as shared in earlier comment).