Certificates fail to renew
cafca opened this issue · comments
Hello beaker people,
Unfortunately homebase has stopped renewing letsencrypt certificates for me. I am seeing several errors repeatedly in the homebase logs and I am not sure which of them is pointing at the original cause of the issue.
The first thing I did was updating my .homebase.yml config by removing dats.*.name entries and updating homebase itself through npm.
I am now running homebase 2.0.8 through pm2 on Ubuntu 16.04.5 LTS.
One kind of error I am seeing is
[Error] approveDomains rejected tls sni 'www.<a domain I am proxying>'
[Error] (see https://git.coolaj86.com/coolaj86/greenlock.js/issues/11)
[Error] This is the rejection message:
Invalid domain
the second kind is
[acme-v2.js] authorizations were not fetched:
{ type: 'urn:ietf:params:acme:error:badNonce',
detail: 'JWS has an invalid anti-replay nonce: "<alphanumeric value>"',
status: 400 }
[acme-v2] handled(?) rejection as errback:
Error: authorizations were not fetched
at /home/pv/.npm-global/lib/node_modules/@beaker/homebase/node_modules/acme-v2/node.js:588:31
at <anonymous>
at process._tickCallback (internal/process/next_tick.js:182:7)
{ Error: authorizations were not fetched
at /home/pv/.npm-global/lib/node_modules/@beaker/homebase/node_modules/acme-v2/node.js:588:31
at <anonymous>
at process._tickCallback (internal/process/next_tick.js:182:7)
cause: Error: authorizations were not fetched
at /home/pv/.npm-global/lib/node_modules/@beaker/homebase/node_modules/acme-v2/node.js:588:31
at <anonymous>
at process._tickCallback (internal/process/next_tick.js:182:7),
isOperational: true } Promise {
_bitField: 152305664,
_fulfillmentHandler0:
{ Error: authorizations were not fetched
at /home/pv/.npm-global/lib/node_modules/@beaker/homebase/node_modules/acme-v2/node.js:588:31
at <anonymous>
at process._tickCallback (internal/process/next_tick.js:182:7)
cause: Error: authorizations were not fetched
at /home/pv/.npm-global/lib/node_modules/@beaker/homebase/node_modules/acme-v2/node.js:588:31
at <anonymous>
at process._tickCallback (internal/process/next_tick.js:182:7),
isOperational: true },
_rejectionHandler0: undefined,
_promise0: undefined,
_receiver0: undefined }
the third kind is
FATAL ERROR: Committing semi space failed. Allocation failed - process out of memory
1: node::Abort() [node /home/pv/.npm-global/bin/homebase]
2: 0x8d05bc [node /home/pv/.npm-global/bin/homebase]
3: v8::Utils::ReportOOMFailure(char const*, bool) [node /home/pv/.npm-global/bin/homebase]
4: v8::internal::V8::FatalProcessOutOfMemory(char const*, bool) [node /home/pv/.npm-global/bin/homebase]
5: v8::internal::Heap::PerformGarbageCollection(v8::internal::GarbageCollector, v8::GCCallbackFlags) [node /home/pv/.np
m-global/bin/homebase]
6: v8::internal::Heap::CollectGarbage(v8::internal::AllocationSpace, v8::internal::GarbageCollectionReason, v8::GCCallbackFlags) [node /home/pv/.npm-global/bin/homebase]
7: v8::internal::Factory::NewFillerObject(int, bool, v8::internal::AllocationSpace) [node /home/pv/.npm-global/bin/homebase]
8: v8::internal::Runtime_AllocateInNewSpace(int, v8::internal::Object**, v8::internal::Isolate*) [node /home/pv/.npm-global/bin/homebase]
9: 0x235a48c842fd
last of all I am seeing this error
{ Error: [ERROR] Certificate issued at '2018-10-30T18:51:42.000Z' and expires at '2019-01-28T18:51:42.000Z'. Ignoring renewal attempt until '2019-01-14T18:51:42.000Z'. Set { duplicate: true } to force.
at Object.renewAsync (/home/pv/.npm-global/lib/node_modules/@beaker/homebase/node_modules/greenlock/lib/core.js:341:17)
at /home/pv/.npm-global/lib/node_modules/@beaker/homebase/node_modules/greenlock/index.js:451:43
at Object.approveDomains (/home/pv/.npm-global/lib/node_modules/@beaker/homebase/lib/lets-encrypt.js:13:14)
at Object.gl.getCertificates (/home/pv/.npm-global/lib/node_modules/@beaker/homebase/node_modules/greenlock/index.js:431:14)
at Object.tryCatcher (/home/pv/.npm-global/lib/node_modules/@beaker/homebase/node_modules/bluebird/js/release/util.js:16:23)
at Object.ret [as getCertificatesAsync] (eval at makeNodePromisifiedEval (/home/pv/.npm-global/lib/node_modules/@beaker/homebase/node_modules/bluebird/js/release/promisify.js:184:12), <anonymous>:13:39)
at Object.sniCallback (/home/pv/.npm-global/lib/node_modules/@beaker/homebase/node_modules/le-sni-auto/index.js:141:19)
at TLSSocket.gl.tlsOptions.SNICallback [as _SNICallback] (/home/pv/.npm-global/lib/node_modules/@beaker/homebase/node_modules/greenlock/index.js:507:16)
at TLSWrap.loadSNI [as oncertcb] (_tls_wrap.js:132:9)
cause: { Error: [ERROR] Certificate issued at '2018-10-30T18:51:42.000Z' and expires at '2019-01-28T18:51:42.000Z'. Ignoring renewal attempt until '2019-01-14T18:51:42.000Z'. Set { duplicate: true } to force.
at Object.renewAsync (/home/pv/.npm-global/lib/node_modules/@beaker/homebase/node_modules/greenlock/lib/core.js:341:17)
at /home/pv/.npm-global/lib/node_modules/@beaker/homebase/node_modules/greenlock/index.js:451:43
at Object.approveDomains (/home/pv/.npm-global/lib/node_modules/@beaker/homebase/lib/lets-encrypt.js:13:14)
at Object.gl.getCertificates (/home/pv/.npm-global/lib/node_modules/@beaker/homebase/node_modules/greenlock/index.js:431:14)
at Object.tryCatcher (/home/pv/.npm-global/lib/node_modules/@beaker/homebase/node_modules/bluebird/js/release/util.js:16:23)
at Object.ret [as getCertificatesAsync] (eval at makeNodePromisifiedEval (/home/pv/.npm-global/lib/node_modules/@beaker/homebase/node_modules/bluebird/js/release/promisify.js:184:12), <anonymous>:13:39)
at Object.sniCallback (/home/pv/.npm-global/lib/node_modules/@beaker/homebase/node_modules/le-sni-auto/index.js:141:19)
at TLSSocket.gl.tlsOptions.SNICallback [as _SNICallback] (/home/pv/.npm-global/lib/node_modules/@beaker/homebase/node_modules/greenlock/index.js:507:16)
at TLSWrap.loadSNI [as oncertcb] (_tls_wrap.js:132:9) code: 'E_NOT_RENEWABLE' },
isOperational: true,
code: 'E_NOT_RENEWABLE' },
_rejectionHandler0: undefined,
_promise0: undefined,
_receiver0: undefined }
Any help would be very appreciated. Please tell me if additional info would be helpful or if you can think of any steps I could take to find the error cause.
The last error indicates that renewals are blocked right now anyway. How can I get homebase to attempt renewal again?
Also, in another issue Paul said that deleting the letsencrypt cache at .homebase/letsencrypt might help. I am not sure where the cache is inside that folder (I am seeing etc/{accounts,archive,live,removal} and var/lib/acme-challenge subfolders). Notably, there is no etc/renewal subfolder, which I had expected because I saw it being used in another letsencrypt installation.
Thank you!
@ciex whew, that's a big collection of bugs emerging there. One thing you might try is making sure that your homebase's dependencies are up to their latest version -- the letsencrypt module we use (greenlock) is frequently updated, in no small part because letsencrypt itself is updating frequently.
I tried updating greenlock but now I am not sure how to even get homebase to attempt updating certificates outside of its own schedule to see whether that helps. Everything is so automated! Restarting homebase doesn't seem to have that effect.
Are there letsencrypt caches of some sort I can safely delete?
The next best thing I would try is deleting ~/.homebase and reinstalling everything, keeping my config file. I am not knowledgeable enough about dat to know whether homebase could automatically fetch my dats again from my local beaker browser though.
I am not sure what happened but the certificates have been renewed again in the meantime. ¯\_(ツ)_/¯