Twisted: attachBuilder(): "assert not isinstance(result, Deferred)" failed on reconfig
vstinner opened this issue · comments
I applied 94b4769
When I deployed the config using buildbot reconfig command, I got this error:
...
2019-09-19 14:31:49+0000 [Broker,6121,8.43.85.235] Worker cstratak-RHEL8-x86_64 attached to AMD64 RHEL8 LTO 3.x
2019-09-19 14:31:49+0000 [Broker,6121,8.43.85.235] Worker cstratak-RHEL8-x86_64 attached to AMD64 RHEL8 custom
2019-09-19 14:31:49+0000 [Broker,4173,8.43.85.216] during reconfig:
Traceback (most recent call last):
File "/data/buildbot/venv/lib/python3.6/site-packages/twisted/internet/defer.py", line 654, in _runCallbacks
current.result = callback(current.result, *args, **kw)
File "/data/buildbot/venv/lib/python3.6/site-packages/twisted/internet/defer.py", line 1475, in gotResult
_inlineCallbacks(r, g, status)
File "/data/buildbot/venv/lib/python3.6/site-packages/twisted/internet/defer.py", line 1416, in _inlineCallbacks
result = result.throwExceptionIntoGenerator(g)
File "/data/buildbot/venv/lib/python3.6/site-packages/twisted/python/failure.py", line 512, in throwExceptionIntoGenerator
return g.throw(self.type, self.value, self.tb)
--- <exception caught here> ---
File "/data/buildbot/venv/lib/python3.6/site-packages/buildbot/master.py", line 397, in doReconfig
yield self.reconfigServiceWithBuildbotConfig(new_config)
File "/data/buildbot/venv/lib/python3.6/site-packages/twisted/internet/defer.py", line 1416, in _inlineCallbacks
result = result.throwExceptionIntoGenerator(g)
File "/data/buildbot/venv/lib/python3.6/site-packages/twisted/python/failure.py", line 512, in throwExceptionIntoGenerator
return g.throw(self.type, self.value, self.tb)
File "/data/buildbot/venv/lib/python3.6/site-packages/buildbot/util/service.py", line 49, in reconfigServiceWithBuildbotConfig
yield svc.reconfigServiceWithBuildbotConfig(new_config)
File "/data/buildbot/venv/lib/python3.6/site-packages/twisted/internet/defer.py", line 1416, in _inlineCallbacks
result = result.throwExceptionIntoGenerator(g)
File "/data/buildbot/venv/lib/python3.6/site-packages/twisted/python/failure.py", line 512, in throwExceptionIntoGenerator
return g.throw(self.type, self.value, self.tb)
File "/data/buildbot/venv/lib/python3.6/site-packages/buildbot/process/measured_service.py", line 31, in reconfigServiceWithBuildbotConfig
yield super(MeasuredBuildbotServiceManager, self).reconfigServiceWithBuildbotConfig(new_config)
File "/data/buildbot/venv/lib/python3.6/site-packages/twisted/internet/defer.py", line 1416, in _inlineCallbacks
result = result.throwExceptionIntoGenerator(g)
File "/data/buildbot/venv/lib/python3.6/site-packages/twisted/python/failure.py", line 512, in throwExceptionIntoGenerator
return g.throw(self.type, self.value, self.tb)
File "/data/buildbot/venv/lib/python3.6/site-packages/buildbot/util/service.py", line 520, in reconfigServiceWithBuildbotConfig
yield svc.reconfigServiceWithSibling(config_sibling)
File "/data/buildbot/venv/lib/python3.6/site-packages/twisted/internet/defer.py", line 1416, in _inlineCallbacks
result = result.throwExceptionIntoGenerator(g)
File "/data/buildbot/venv/lib/python3.6/site-packages/twisted/python/failure.py", line 512, in throwExceptionIntoGenerator
return g.throw(self.type, self.value, self.tb)
File "/data/buildbot/venv/lib/python3.6/site-packages/buildbot/worker/base.py", line 331, in reconfigServiceWithSibling
yield self.updateWorker()
File "/data/buildbot/venv/lib/python3.6/site-packages/twisted/internet/defer.py", line 1418, in _inlineCallbacks
result = g.send(result)
File "/data/buildbot/venv/lib/python3.6/site-packages/buildbot/worker/base.py", line 538, in sendBuilderList
d1 = self.attachBuilder(b)
File "/data/buildbot/venv/lib/python3.6/site-packages/buildbot/worker/base.py", line 543, in attachBuilder
return builder.attached(self, self.worker_commands)
File "/data/buildbot/venv/lib/python3.6/site-packages/twisted/internet/defer.py", line 1613, in unwindGenerator
return _cancellableInlineCallbacks(gen)
File "/data/buildbot/venv/lib/python3.6/site-packages/twisted/internet/defer.py", line 1529, in _cancellableInlineCallbacks
_inlineCallbacks(None, g, status)
File "/data/buildbot/venv/lib/python3.6/site-packages/twisted/internet/defer.py", line 1421, in _inlineCallbacks
status.deferred.callback(getattr(e, "value", None))
File "/data/buildbot/venv/lib/python3.6/site-packages/twisted/internet/defer.py", line 459, in callback
assert not isinstance(result, Deferred)
builtins.AssertionError:
2019-09-19 14:31:49+0000 [Broker,4173,8.43.85.216] WARNING: reconfig partially applied; master may malfunction
Reconfiguration failed. Please inspect the master.cfg file for errors, correct
them, then try 'buildbot reconfig' again.
I ran stop followed by start buildbot commands: it worked around the issue.
I'm not sure if we should do anything with this error which only occured once?
cc @pablogsal @zware
I think this is a bug in buildbot, but I'm not certain of that. I'm not sure how our configuration can be at fault, but it's also only been minimally updated to work with buildbot >= 0.9.
I think this is a bug in buildbot, but I'm not certain of that. I'm not sure how our configuration can be at fault, but it's also only been minimally updated to work with buildbot >= 0.9.
I upgraded (apt-get update && apt-get dist-upgrade -y) and rebooted (to get the new Linux kernel) the system.
I upgraded Buildbot (recreated the venv): 8047cf5
I close the issue. As I wrote, stop+start worked again the issue. I will only investigate further if the bug strikes back.