fix(tls): absolutise CERT_DIR so cron renewals actually install

acme.sh persists the --install-cert paths and replays them on every cron
renewal, where cwd is $HOME rather than the repo. The relative default
resolved to ~/certs/... so renewals wrote nothing: acme.sh kept renewing
cleanly while nginx served the stale copy until it expired. Apex, auth. and
pack. served expired certs on 2026-09-06.

Absolutise CERT_DIR in issue-letsencrypt.sh and document the failure mode,
the recovery command, and off-host verification in plan/15-letsencrypt.md.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018UJzZrBaZUoU6qHMLpLz64

plan/15-letsencrypt.md

@@ -36,6 +36,49 @@ Certs install to `certs/<name>/{cert.pem,key.pem}` (gitignored). acme.sh adds a
 cron for renewal; `--reloadcmd` reloads nginx automatically on renew. Override
 the reload with `RELOAD_CMD=...` if nginx isn't systemd-managed.
 
+### Gotcha — install paths must be ABSOLUTE (silent expiry, 2026-09-06)
+
+`--install-cert` paths are **persisted in acme.sh's domain conf**
+(`~/.acme.sh/<name>_ecc/<name>.conf` → `Le_RealKeyPath` /
+`Le_RealFullChainPath`) and replayed on every cron renewal. The cron runs with
+`cwd=$HOME`, not the repo — so a **relative** path (`certs/<name>/key.pem`)
+resolves to `~/certs/...` and the install writes nothing.
+
+Failure mode is silent and delayed: `acme.sh --list` shows every cert freshly
+renewed with a future `Renew` date, `~/.acme.sh/<name>_ecc/fullchain.cer` is
+valid — while `certs/<name>/cert.pem` is frozen at the last **manual** run and
+nginx serves that until it expires. There is no error anywhere. Cost: apex,
+`auth.` and `pack.` served expired certs on 2026-09-06.
+
+Fix / recovery — re-run `--install-cert` with absolute paths, which both copies
+the current cert and rewrites the persisted paths:
+
+```bash
+APP=/home/ubuntu/mc/ulicraft-server-v1
+for d in $(ls $APP/certs); do
+  ~/.acme.sh/acme.sh --install-cert -d "$d" --ecc \
+    --key-file       "$APP/certs/$d/key.pem" \
+    --fullchain-file "$APP/certs/$d/cert.pem" \
+    --reloadcmd      "sudo systemctl reload nginx || true"
+done
+```
+
+`issue-letsencrypt.sh` now absolutises `CERT_DIR` so a fresh issue can't
+reintroduce it.
+
+**Verify from off-host, not from the files** — the installed file and what nginx
+serves are the same thing here, but the acme.sh copy is not:
+
+```bash
+for h in ulicraft.net auth.ulicraft.net avatar.ulicraft.net status.ulicraft.net \
+         distribution.ulicraft.net files.ulicraft.net upload.ulicraft.net \
+         monitor.ulicraft.net generators.ulicraft.net; do
+  printf '%-30s ' "$h"
+  echo | openssl s_client -servername $h -connect $h:443 2>/dev/null \
+    | openssl x509 -noout -enddate
+done
+```
+
 ## nginx
 
 1. Bring the stack up (caddy publishes on `127.0.0.1:${CADDY_HTTP_PORT}`):

tooling/issue-letsencrypt.sh

@@ -52,7 +52,12 @@ ACME="${ACME_SH:-$HOME/.acme.sh/acme.sh}"
   curl https://get.acme.sh | sh -s email=${LE_EMAIL}
 then re-run (or set ACME_SH=/path/to/acme.sh)."
 
-CERT_DIR="${CERT_DIR:-certs}"
+# MUST be absolute: acme.sh persists these paths in its domain conf and replays
+# them on every cron renewal, where cwd is $HOME, not the repo. A relative path
+# there makes the renewal install silently write nothing -- acme.sh keeps renewing
+# happily while nginx serves the stale (eventually expired) copy. Bit us 2026-09-06.
+CERT_DIR="${CERT_DIR:-$PWD/certs}"
+case "$CERT_DIR" in /*) ;; *) CERT_DIR="$PWD/$CERT_DIR" ;; esac
 read -r -a SUBDOMAINS <<< "${LE_SUBDOMAINS:-auth pack distribution www avatar status}"
 
 if [ "${LE_STAGING:-0}" = "1" ]; then