Caddy SSL certificate acquisition fails on subsequent deploys

[Manuel Thomsen]

Hi there!

First: PullPreview is great and we love using it.

We're seeing an issue when we push multiple (3+) commits to the same PR within a short-ish (2 hour) timeframe.
It looks like Caddy tries to get a new SSL certificate but is ratelimited by Lets Encrypt.

We're using the Caddy/Proxy service as defined here:
https://github.com/pullpreview/action/wiki/SSL-HTTPS-Configuration

  proxy:
    image: caddy:2
    restart: unless-stopped
    command: "caddy reverse-proxy --from '${PULLPREVIEW_PUBLIC_DNS}' --to app:3000"
    depends_on:
      - app
    ports:
      - 80:80
      - 443:443

This is the error we're seeing:

app-proxy-1  | {"level":"warn","ts":1717422234.096732,"logger":"admin","msg":"admin endpoint disabled"}
app-proxy-1  | {"level":"info","ts":1717422234.096971,"logger":"tls.cache.maintenance","msg":"started background certificate maintenance","cache":"0xc0004ac100"}
app-proxy-1  | {"level":"info","ts":1717422234.0970614,"logger":"http.auto_https","msg":"server is listening only on the HTTPS port but has no TLS connection policies; adding one to enable TLS","server_name":"proxy","https_port":443}
app-proxy-1  | {"level":"info","ts":1717422234.0971072,"logger":"http.auto_https","msg":"enabling automatic HTTP->HTTPS redirects","server_name":"proxy"}
app-proxy-1  | {"level":"info","ts":1717422234.0980291,"logger":"http","msg":"enabling HTTP/3 listener","addr":":443"}
app-proxy-1  | {"level":"info","ts":1717422234.0983136,"msg":"failed to sufficiently increase receive buffer size (was: 208 kiB, wanted: 7168 kiB, got: 416 kiB). See https://github.com/quic-go/quic-go/wiki/UDP-Buffer-Sizes for details."}
app-proxy-1  | {"level":"info","ts":1717422234.0985816,"logger":"http.log","msg":"server running","name":"proxy","protocols":["h1","h2","h3"]}
app-proxy-1  | {"level":"info","ts":1717422234.0986974,"logger":"http.log","msg":"server running","name":"remaining_auto_https_redirects","protocols":["h1","h2","h3"]}
app-proxy-1  | {"level":"info","ts":1717422234.0987363,"logger":"http","msg":"enabling automatic TLS certificate management","domains":["OUR URL HERE"]}
app-proxy-1  | {"level":"info","ts":1717422234.098888,"msg":"caddy proxying","from":"https://OUR URL HERE","to":["app:3000"]}
app-proxy-1  | {"level":"info","ts":1717422234.0993028,"logger":"tls","msg":"cleaning storage unit","storage":"FileStorage:/data/caddy"}
app-proxy-1  | {"level":"info","ts":1717422234.0995574,"logger":"tls","msg":"finished cleaning storage units"}
app-proxy-1  | {"level":"info","ts":1717422234.0998597,"logger":"tls.obtain","msg":"acquiring lock","identifier":"OUR URL HERE"}
app-proxy-1  | {"level":"info","ts":1717422234.1015437,"logger":"tls.obtain","msg":"lock acquired","identifier":"OUR URL HERE"}
app-proxy-1  | {"level":"info","ts":1717422234.1017861,"logger":"tls.obtain","msg":"obtaining certificate","identifier":"OUR URL HERE"}
app-proxy-1  | {"level":"info","ts":1717422234.3759623,"logger":"http","msg":"waiting on internal rate limiter","identifiers":["OUR URL HERE"],"ca":"https://acme-v02.api.letsencrypt.org/directory","account":""}
app-proxy-1  | {"level":"info","ts":1717422234.3760061,"logger":"http","msg":"done waiting on internal rate limiter","identifiers":["OUR URL HERE"],"ca":"https://acme-v02.api.letsencrypt.org/directory","account":""}
app-proxy-1  | {"level":"info","ts":1717422234.376037,"logger":"http","msg":"using ACME account","account_id":"https://acme-v02.api.letsencrypt.org/acme/acct/1761309062","account_contact":[]}
app-proxy-1  | {"level":"error","ts":1717422234.4323957,"logger":"tls.obtain","msg":"could not get certificate from issuer","identifier":"OUR URL HERE","issuer":"acme-v02.api.letsencrypt.org-directory","error":"HTTP 429 urn:ietf:params:acme:error:rateLimited - Error creating new order :: too many certificates (5) already issued for this exact set of domains in the last 168 hours: OUR URL HERE, retry after 2024-06-04T22:04:46Z: see https://letsencrypt.org/docs/duplicate-certificate-limit/"}
app-proxy-1  | {"level":"error","ts":1717422234.432511,"logger":"tls.obtain","msg":"will retry","error":"[OUR URL HERE] Obtain: [OUR URL HERE] creating new order: attempt 1: https://acme-v02.api.letsencrypt.org/acme/new-order: HTTP 429 urn:ietf:params:acme:error:rateLimited - Error creating new order :: too many certificates (5) already issued for this exact set of domains in the last 168 hours: OUR URL HERE, retry after 2024-06-04T22:04:46Z: see https://letsencrypt.org/docs/duplicate-certificate-limit/ (ca=https://acme-v02.api.letsencrypt.org/directory)","attempt":1,"retrying_in":60,"elapsed":0.33085037,"max_duration":2592000}

Do you know how we can avoid this issue?

[Cyril Rohr]

@DerJacques does your compose file include the volume to permanently store the certificates? The documentation is as follows:

services:
  # use Caddy for SSL termination
  proxy:
    image: caddy:2
    restart: unless-stopped
    command: "caddy reverse-proxy --from '${PULLPREVIEW_URL}' --to web:4567"
    depends_on:
      - web
    ports:
      - "80:80"
      - "443:443"
    volumes:
      - "/data"

[Manuel Thomsen]

@crohr Thanks for the quick reply! I can't believe we missed that 🫣
We did not have that volume set. I added it to our docker-compose file now, and I assume that this will do the trick.

Thank you!