systemd: excessive Requires/After can result in dependency failure
paulnivin opened this issue · comments
Bug
Operating System Version
Fedora CoreOS 35.20220103.3.0
Afterburn Version
5.1.0
Environment
AWS
Expected Behavior
The default afterburn systemd service config should not fail if lots of dependent services on a host follow the afterburn documentation for systemd and set Requires=afterburn.service with After=afterburn.service.
Actual Behavior
Since afterburn.service is a oneshot service and RemainAfterExit is not set, afterburn is repeatedly invoked for every service with an afterburn dependency.
If a sufficient number of systemd units invoke afterburn, systemd will fail afterburn as well as the service that caused afterburn.service to be invoked.
failure logs:
Jan 25 01:02:10 systemd[1]: afterburn.service: Start request repeated too quickly.
Jan 25 01:02:10 systemd[1]: afterburn.service: Failed with result 'start-limit-hit'.
Jan 25 01:02:10 systemd[1]: Failed to start Afterburn (Metadata).
Other Information
In our environment, we've added a systemd override to set RemainAfterExit=yes, as a single successful run of afterburn is sufficient on AWS to populate attributes.
If adding RemainAfterExit=yes to the default afterburn.service is not a good fix, documentation should be written for the best practice for ensuring that dependencies do not fail when using the recommended/default configuration.
Thanks for the report. Yes I think the service unit is indeed missing a RemainAfterExit=yes, right I'm not seeing any good reasons why it shouldn't have that.
Ahh yes, that's a classic. Turns out we had multiple services which didn't do this. Opened #695 to address them all.