After updating to Protect 3.0.22 on my UDM Pro the lets encrypt certificate hinders the mediaserver to start
fribse opened this issue · comments
Describe the bug
I've been running this for quite a while, and I've been very happy with it, it seems to work perfectly.
After I've updated to Protect 3.0.22, the protect media server wouldn't start.
I've had unifi look on it, and as soon as they removed the letsencrypt certificate, the media server started as it should.
To Reproduce
Steps to reproduce the behavior:
- Update Protect to 3.0.22
- Go to view the cameras
- They will show the three dots for 'updating'
- At somepoint a still will be updated, and then it crashes again and restarts.
Expected behavior
It should just show the camera image streams
Version Information (please complete the following information):
- UniFi OS: v3.2.12
- Hardware Type: UDM Pro (non-SE)
- Unifi Protect: 3.0.22
Can you scrounge up some logs of what might be preventing it from starting? Something out of /var/log? I'll try to replicate this on my unvr when I get a few, but it would be helpful to see what you're seeing.
So I don't run Protect on my UDM, I have a UNVR and I just installed udm-le from scratch on there (I didn't actually have it installed) and it installed correctly and Protect 3.0.22 seems to be running just fine after a reboot. I'm going to need some kind of logs to figure this out since I can't replicate it.
I'm quite confused. I found some old letsencrypt in /etc/letsencrypt, I've deleted that.
I then went through the uninstall process to remove udm-le completely.
I then uploaded it once again, entering my email and the cloudflare token.
After doing the 'initial' the unifi-core keeps restarting?
Uninstalling it again, makes it work again (with the locally generated certificate)
Some logs:
root@Fribert:/var/log# tail -f ulcmd.log
2024-03-25 17:28:01 Info Core: Internet: not connected
2024-03-25 17:28:01 Info Core: Internet: connected
2024-03-25 17:28:02 Warning Process: UCoreInfo Is taking longer to respond
2024-03-25 17:28:05 Error Core: ConsoleStatus stream failed
2024-03-25 17:28:17 Warning Process: UCoreInfo Is taking longer to respond
2024-03-25 17:28:17 Info Core: ConsoleStatus stream ready
2024-03-25 17:28:21 Error Core: ConsoleStatus stream failed
2024-03-25 17:28:33 Warning Process: UCoreInfo Is taking longer to respond
2024-03-25 17:28:33 Info Core: ConsoleStatus stream ready
2024-03-25 17:28:36 Error Core: ConsoleStatus stream failed
2024-03-25 17:28:47 Info Core: ConsoleStatus stream ready
2024-03-25 17:28:47 Info Core: Internet: not connected
2024-03-25 17:28:48 Info Core: Internet: connected
2024-03-25 17:28:49 Warning Process: UCoreInfo Is taking longer to respond
2024-03-25 17:28:52 Error Core: ConsoleStatus stream failed
q2024-03-25 17:29:03 Info Core: ConsoleStatus stream ready
2024-03-25 17:29:03 Info Core: Internet: not connected
2024-03-25 17:29:03 Info Core: Internet: connected
2024-03-25 17:29:04 Warning Process: UCoreInfo Is taking longer to respond
More logs:
2024-03-25T17:30:34+01:00 Fribert systemd[1]: freeradius.service: Control process exited, code=exited, status=1/FAILURE
2024-03-25T17:30:34+01:00 Fribert systemd[1]: freeradius.service: Failed with result 'exit-code'.
2024-03-25T17:30:34+01:00 Fribert systemd[1]: Failed to start FreeRADIUS multi-protocol policy server.
2024-03-25T17:30:34+01:00 Fribert systemd[1]: Started UniFi Core.
2024-03-25T17:30:38+01:00 Fribert systemd-timedated[1567169]: Set NTP to enabled (systemd-timesyncd.service).
2024-03-25T17:30:39+01:00 Fribert systemd[1]: freeradius.service: Scheduled restart job, restart counter is at 443.
2024-03-25T17:30:39+01:00 Fribert systemd[1]: Stopped FreeRADIUS multi-protocol policy server.
2024-03-25T17:30:39+01:00 Fribert systemd[1]: Starting FreeRADIUS multi-protocol policy server...
2024-03-25T17:30:39+01:00 Fribert freeradius[1581750]: FreeRADIUS Version 3.2.1
2024-03-25T17:30:39+01:00 Fribert freeradius[1581750]: Copyright (C) 1999-2022 The FreeRADIUS server project and contributors
2024-03-25T17:30:39+01:00 Fribert freeradius[1581750]: There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A
2024-03-25T17:30:39+01:00 Fribert freeradius[1581750]: PARTICULAR PURPOSE
2024-03-25T17:30:39+01:00 Fribert freeradius[1581750]: You may redistribute copies of FreeRADIUS under the terms of the
2024-03-25T17:30:39+01:00 Fribert freeradius[1581750]: GNU General Public License
2024-03-25T17:30:39+01:00 Fribert freeradius[1581750]: For more information about these matters, see the file named COPYRIGHT
2024-03-25T17:30:39+01:00 Fribert freeradius[1581750]: Starting - reading configuration files ...
2024-03-25T17:30:39+01:00 Fribert freeradius[1581750]: Debug state unknown (cap_sys_ptrace capability not set)
2024-03-25T17:30:39+01:00 Fribert freeradius[1581750]: Creating attribute Unix-Group
2024-03-25T17:30:39+01:00 Fribert freeradius[1581750]: rlm_detail (auth_log): 'User-Password' suppressed, will not appear in detail output
2024-03-25T17:30:39+01:00 Fribert freeradius[1581750]: rlm_eap (EAP): Ignoring EAP method 'leap', because it is no longer supported
2024-03-25T17:30:39+01:00 Fribert freeradius[1581750]: tls: (TLS) Failed reading certificate file "/etc//freeradius/3.0/certs/server.pem"
2024-03-25T17:30:39+01:00 Fribert freeradius[1581750]: tls: (TLS) error:0200100D:system library:fopen:Permission denied
2024-03-25T17:30:39+01:00 Fribert freeradius[1581750]: tls: (TLS) error:20074002:BIO routines:file_ctrl:system lib
2024-03-25T17:30:39+01:00 Fribert freeradius[1581750]: tls: (TLS) error:140DC002:SSL routines:use_certificate_chain_file:system lib
2024-03-25T17:30:39+01:00 Fribert freeradius[1581750]: rlm_eap_tls: Failed initializing SSL context
2024-03-25T17:30:39+01:00 Fribert freeradius[1581750]: rlm_eap (EAP): Failed to initialise rlm_eap_tls
2024-03-25T17:30:39+01:00 Fribert freeradius[1581750]: /etc/freeradius/3.0/mods-enabled/eap[2]: Instantiation failed for module "eap"
2024-03-25T17:30:39+01:00 Fribert mcad[2851517]: mcad[2851517]: ace_reporter.reporter_handle_response_json(): cfgversion: 2b24bc5272839595 -> abf6fd073358d2d5
2024-03-25T17:30:39+01:00 Fribert mcad[2851517]: mcad[2851517]: ace_reporter.reporter_save_config(): Saving stun_url as stun://127.0.0.1/
2024-03-25T17:30:39+01:00 Fribert mcad[2851517]: mcad[2851517]: ace_reporter.reporter_save_config(): Saving mgmt_url as https://127.0.0.1:8443/manage/site/default
2024-03-25T17:30:39+01:00 Fribert mcad[2851517]: mcad[2851517]: ace_reporter.reporter_handle_response_json(): [setparam] applying new system.cfg
2024-03-25T17:30:39+01:00 Fribert systemd[1]: freeradius.service: Control process exited, code=exited, status=1/FAILURE
2024-03-25T17:30:39+01:00 Fribert systemd[1]: freeradius.service: Failed with result 'exit-code'.
2024-03-25T17:30:39+01:00 Fribert systemd[1]: Failed to start FreeRADIUS multi-protocol policy server.
2024-03-25T17:30:41+01:00 Fribert mcad[2851517]: mcad[2851517]: ace_reporter.reporter_save_config(): Saving stun_url as stun://127.0.0.1/
2024-03-25T17:30:41+01:00 Fribert mcad[2851517]: mcad[2851517]: ace_reporter.reporter_save_config(): Saving mgmt_url as https://127.0.0.1:8443/manage/site/default
2024-03-25T17:30:42+01:00 Fribert ulcmd[1581574]: curl: (52) Empty reply from server
2024-03-25T17:30:42+01:00 Fribert systemd[1]: unifi-core.service: Main process exited, code=exited, status=1/FAILURE
2024-03-25T17:30:42+01:00 Fribert systemd[1]: unifi-core.service: Failed with result 'exit-code'.
2024-03-25T17:30:42+01:00 Fribert systemd[1]: unifi-core.service: Consumed 14.178s CPU time.
2024-03-25T17:30:42+01:00 Fribert ulcmd[1582065]: curl: (7) Failed to connect to localhost port 11081: Connection refused
2024-03-25T17:30:42+01:00 Fribert systemd[1]: unifi-core.service: Scheduled restart job, restart counter is at 17.
2024-03-25T17:30:42+01:00 Fribert systemd[1]: Stopped UniFi Core.
2024-03-25T17:30:42+01:00 Fribert systemd[1]: unifi-core.service: Consumed 14.178s CPU time.
2024-03-25T17:30:42+01:00 Fribert systemd[1]: Starting UniFi Core...
2024-03-25T17:30:43+01:00 Fribert ulcmd[1582367]: curl: (7) Failed to connect to localhost port 11081: Connection refused
If you want to look at my UDM pro through SSH I can arrange it via tmate ssh, right now unifi is looking at it that way
Ah, I bet you have ENABLE_FREERADIUS=yes in your udm-le.env? Try changing line 89 of udm-le.sh from chmod 600 to chmod 644 and give it another go. It seems like freeradius is running as the freerad user and doesn't have permissions to read those certificates. I don't think many people use this option (I don't) so I'm not surprised it's a little buggy.
Ah red herring, I thought that might be interfering but that's not actually where those certs live and my UDW spits out the same thing even without RADIUS support enabled.
Odd, so I don't run Protect on my UDW (which is running 3.2.12 as well) and I installed Protect just to see if it ran into the same thing, but it seems fine:
2024-03-25 16:12:27 Info UcoreControllers: protect installing
2024-03-25 16:12:27 Info UcoreControllers: protect version: 3.0.22
2024-03-25 16:12:54 Info UcoreControllers: Add protect app
2024-03-25 16:13:03 Warning Process: UCoreInfo Is taking longer to respond
2024-03-25 16:13:12 Info UcoreControllers: Remove protect app
2024-03-25 16:13:15 Info UcoreControllers: protect installing
2024-03-25 16:13:15 Info UcoreControllers: protect version: 3.0.22
2024-03-25 16:13:45 Info UcoreControllers: protect installed
2024-03-25 16:13:45 Info UcoreControllers: Add protect app
2024-03-25 16:13:49 Warning Process: UCoreInfo Is taking longer to respond
I'm not sure why it installed it, removed it, and reinstalled it but once it had done so it's up and running fine and I can access it both via unifi.ui.com as well as locally via https://mydomain.net/protect/dashboard with a valid LE cert.
That said, I did not adopt a camera, which may have something to do with it, but I also installed this on my UNVR and it's working fine over there also.
@kchristensen I think if you don't have any cameras, you may not see any problem in Protect. The issue only occurs when trying to playback video streams. Otherwise the Protect app appears functional.
BTW there is a thread about the problem in the community forum here: https://community.ui.com/questions/Not-seeing-recordings-or-streaming-video-in-app/b62df6bf-0372-453e-ae84-50cccc7c7675
Ah good find, I thought the comment about 4096 bit certs was maybe a clue, but we’re generating 2048 bit certs by default.
I have the standard env file, so
ENABLE_RADIUS="no"
I think it's a permissions problem. 🤔
udm-le puts the certs and keys under /etc/letsencrypt/live/<domain> and creates symlinks in /data/unifi-core/config (at least, that's what my install was doing?) but the /etc/letsencrypt/live directory has mode 0700 so can only be accessed by the root user.
I just tried copying the LE cert and key directly into the /data/unifi-core/config directory, eliminating the symlink. Now I have a signed certificate and Protect video playback works normally.
I don't see the script doing that any more (my old script did this, but the newly downloaded copies them to the correct place).
True. I've just updated udm-le.sh to the latest version and re-run the initial setup. It's now creating the cert and key files directly in /data/unifi-core/config, which appears to be working perfectly. In fact I was able to delete the /etc/letsencrypt directory entirely.
So I guess this issue is just affecting old versions of udm-le?
No, I can't get it working stable here. If I enable the udm-le, it creates the certs, but then unifi-core keeps restarting.
As soon as I go back to the unsigned certs it start unifi-core as it should.
But I did see a lot of errors with freeradius, so I just tried rebooting the entire udm pro, and now I don't see that error at least.
I'll try the udm-le again...
Occurs to me that the /etc/letsencrypt directory and symlinks I had could have been a relic from earlier attempts to set up LE on my UDM and not something that udm-le itself had done at all.
Sorry to hijack your thread @fribse, maybe your issue is different from mine
No problem for me @sgarner, might be a bit confusing for @kchristensen though :-)
I had the same left over folders from a previous attempt, and removed all of those as well, so it's not far off at all.
After the reboot, I uploaded the zip file to /root, and unzipped it on the box, just to make sure that all attributes were correct.
I did the uninstall first off course, and then an initial, and now it has the right certificates AND no errors.
I did have to restart the master switch as well, but the system is running, and I don't see any weird errors in the daemon.log!
I don’t think we ever used /etc/letsencrypt even back in the old docker days of this project, as far as I can remember, but it HAS been a while.
It could be the 'old' way, before your project there was different ways of doing this.
Well, today I had to revert back to selfgenerated again. The live view works fine with the le certs, but if I want to 'go back in time' to earlier recordings, it just hangs. Now that I've gone back to self generated, it works perfectly.
Damned, what is going on?
Looks like UI released a fix:
https://community.ui.com/releases/UniFi-Protect-Application-3-0-24/9b874a04-dc69-4e5c-83e1-a402cae52fdb