Website publishing is borked
daviddias opened this issue · comments
https://ci.ipfs.team/blue/organizations/jenkins/IPFS%2Fblog/detail/master/31/pipeline
It just took 1 hour to fail at the last mile 🙁
Folks, it has been more than 24 hours. What is the best way to tackle this? Is there any manual crank we can use?
Restarted the CI and now it is almost an hour running refs -r
Failing takes 1 hour while being successful takes only 4 mins
#protip if it goes over 5 mins, just restart right away.
@daviddias that sounds like an awful experience but unfortunately I'm not sure there's much I personally can do to help without some assistance from @victorb
I feel there's a much larger issue here in that a critical piece of our dev infra (Jenkins) appears to be unmanned at the moment. In order for infra to take some ownership of it, I would need a lot more insight/documentation into how to debug and fix issues like these. The couple of times that I've gotten involved with debugging Jenkins issues there was a lot of wasted time getting to understand how to deploy changes to the libraries and I still don't know how to make changes to the worker or master instance(s) nor do I have a login to our hosted macOS instances or account. Additionally, the Jenkins deployment as it stands is a very large expense (money wise) that I'm not sure we are realizing true benefit from. Can we please have a larger conversation about website deployments, the requirements, and how we can get to a place where this isn't such a pain for everyone?
cc @mburns
The new saga is https://ci.ipfs.team/blue/organizations/jenkins/IPFS%2Fjs.ipfs.io/activity/. It has been going for almost 2 days.
Having a manual crank would be extremely useful for this scenarios. I'm almost tempted at changing the DNS records manually. Would you advise (or advise against) it? @scout
This was fixed by using circleci and https://github.com/ipfs-shipyard/ipfs-dns-deploy