deploy(prod): pre-deploy DB backup + paired code/DB rollback skill

Every `just deploy` now takes a pg_dumpall to /opt/colectivo/backups/predeploy-<ts>-<sha>.sql.gz BEFORE rebuilding the app or applying migrations, aborts the deploy if the dump is < 1 KiB, and (on success) appends `<iso-ts> \t <sha> \t <backup-file>` to /opt/colectivo/.deploys.log on ambrosio. Keeps the newest 10 backups (prunes older predeploy-* files). New `infra/scripts/rollback-erosi.sh` reads .deploys.log and pairs a code rollback with a DB restore atomically: just rollback-list # show recent deploys just rollback # roll back to N-1 just rollback-to <sha> # roll back to a specific deploy just rollback-code # roll back code only, keep current DB Rollback safety: - 5-second abort window. - Verifies the target SHA exists locally + the backup is still on prod (warns if it's been pruned past the 10-deploy window). - Uses a temporary git worktree so the user's working tree isn't disturbed. - Stops app/auth/rest/realtime/storage before the gunzip|psql restore. - Rebuilds the app image at the rolled-back SHA with the same GIT_SHA build-arg path the deploy uses (so __APP_COMMIT__ in the bundle matches the running code). - Does NOT write a new .deploys.log entry — rollback is intentionally not a deploy event; the next `just deploy` is from current HEAD. - Storage volume (/var/lib/storage user uploads) is NOT rolled back. Justfile `deploy` recipe repointed from the stale `deploy.sh` (which referenced GHCR pull) to `deploy-erosi.sh` (the active prod path). New project-scoped skill: `.claude/skills/deploy/SKILL.md` documents the flow + safety boundaries for Claude-assisted invocations. `.gitignore` keeps `.claude/settings.local.json` ignored but tracks `.claude/skills/`. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
2026-05-18 14:57:25 +02:00
parent c0e5b5ed7f
commit ae4fd45b99
6 changed files with 363 additions and 4 deletions
--- a/.claude/skills/deploy/SKILL.md
+++ b/.claude/skills/deploy/SKILL.md
@@ -0,0 +1,76 @@
+---
+name: deploy
+description: Ship the colectivo stack to ambrosio (https://erosi.limonia.net) with pre-deploy DB backup, or roll back code + DB to a previous deploy. Trigger when the user asks to deploy, ship, push to prod, roll back, or undo a deploy.
+---
+
+# Deploy + rollback skill
+
+Deploys and rollbacks for the production stack on ambrosio. **Every deploy takes a paired DB backup first**, every successful deploy is logged with `(timestamp, git-sha, backup-filename)`, and rollback uses that log to restore both code and DB atomically.
+
+## Trigger
+
+Use this skill when the user says any of:
+- "deploy", "deploy to ambrosio", "deploy to prod", "ship it"
+- "roll back", "rollback", "undo the deploy", "revert prod"
+- "list deploys", "what was the last deploy"
+
+## What the skill does
+
+### Forward (deploy)
+
+1. **`just deploy`** runs `infra/scripts/deploy-erosi.sh`:
+   - Captures the local short SHA (`git rev-parse --short HEAD`).
+   - rsyncs the working tree to `ambrosio:/opt/colectivo/` (excludes `.git`, `node_modules`, `**/.env`, etc.).
+   - On a fresh `.env`, generates secrets and bails — operator fills `PUBLIC_KEYCLOAK_URL` + `KEYCLOAK_CLIENT_SECRET` and re-runs.
+   - **Pre-deploy DB backup**: `pg_dumpall` from the prod `db` container → `backups/predeploy-<ts>-<sha>.sql.gz` on ambrosio. Aborts the deploy if the dump is < 1 KiB. Keeps the newest 10 backups.
+   - Rebuilds the `app` image with `GIT_SHA` build-arg (so `__APP_COMMIT__` lands in the bundle, Fase 14.3).
+   - `docker compose up -d` brings the stack to the new image.
+   - Applies any pending DB migrations.
+   - On success, appends `<iso-ts>\t<sha>\t<backup-file>` to `/opt/colectivo/.deploys.log`.
+2. Site smoke: `curl -I https://erosi.limonia.net/` should return 200.
+
+### Reverse (rollback)
+
+1. **`just rollback-list`** prints recent rows from `.deploys.log` so the user can pick a target.
+2. **`just rollback`** rolls back code + DB to the previous deploy (last but one line of `.deploys.log`).
+3. **`just rollback-to <sha>`** rolls back to a specific deploy.
+4. **`just rollback-code`** rolls back code only, leaves the DB at current state — useful when the regression is UI-only and the DB schema is compatible.
+
+`infra/scripts/rollback-erosi.sh` enforces:
+- Confirms with a 5-second abort window before doing anything destructive.
+- Verifies the target SHA is reachable locally (`git cat-file -e`) and that the backup file still exists on ambrosio.
+- Materialises the target commit into a temporary `git worktree`, rsyncs from there, never disturbs the user's working tree.
+- Stops `app / auth / rest / realtime / storage` before the DB restore so they don't observe inconsistent state.
+- `gunzip | psql -v ON_ERROR_STOP=1` restores the dump.
+- Rebuilds the app image at the rolled-back SHA and brings the stack up.
+- **Does not** write a new `.deploys.log` line — rollback is intentionally not a deploy event.
+
+## Safety boundaries
+
+- The stack lives on ambrosio. The external proxy + TLS terminator are off-stack (`docs/deployment.md`). Neither this skill nor the scripts ever touch host TLS or the external proxy.
+- `infra/scripts/deploy-erosi.sh` never runs `sudo`.
+- `--no-verify` / `--amend` are never used. Pre-commit hooks must pass.
+- Always confirm with the user before invoking `just rollback*` — DB restore is destructive and irreversible.
+- Storage volume (`/var/lib/storage`, user uploads) is **not** part of the rollback. Treated as append-only.
+
+## What to tell the user
+
+Before running anything:
+- Show the local short SHA + branch + remote.
+- For deploy: confirm with the user. Mention "this will take a fresh DB backup before deploying."
+- For rollback: show `just rollback-list` output, ask which SHA, confirm destructive nature.
+
+After running:
+- Print the new `.deploys.log` tail (or the rollback target).
+- Curl-probe the site and report.
+
+## Files this skill touches
+
+- `infra/scripts/deploy-erosi.sh` — deploy, with pre-deploy backup + .deploys.log writing.
+- `infra/scripts/rollback-erosi.sh` — rollback, paired code + DB.
+- `Justfile` — `deploy`, `rollback`, `rollback-list`, `rollback-to`, `rollback-code` recipes.
+- `docs/deployment.md` — runbook for both flows.
+
+## Anti-trigger
+
+If the user is just asking *what would happen* on deploy (status, dry-run-style questions), DO NOT run anything. Answer from the script content.
--- a/.gitignore
+++ b/.gitignore
@@ -42,6 +42,9 @@ apps/web/tests/.auth/
 # PWA dev-mode service worker output
 apps/web/dev-dist/

-# Claude Code per-project runtime state (schedulers etc.)
-.claude/
+# Claude Code per-project runtime state (schedulers etc.). Project-scoped
+# skills under .claude/skills/ ARE tracked — they document workflows for
+# anyone working on the repo.
+.claude/*
+!.claude/skills/
 graphify-out/cache/
--- a/23
+++ b/23
@@ -203,9 +203,28 @@ restore service file:

 # ── Production ────────────────────────────────────────────────────────────────

-# Deploy to production via SSH
+# Deploy to production via SSH (ambrosio / erosi.limonia.net).
+# Pre-deploy: dumps the prod DB to /opt/colectivo/backups/predeploy-<ts>-<sha>.sql.gz
+# Post-deploy: appends (timestamp, sha, backup) to /opt/colectivo/.deploys.log
 deploy:
-    infra/scripts/deploy.sh
+    infra/scripts/deploy-erosi.sh
+
+# List recent prod deploys (timestamp / sha / backup-file).
+rollback-list:
+    infra/scripts/rollback-erosi.sh --list
+
+# Roll back code AND database to the previous deploy.
+# For a specific deploy: `just rollback-to <sha>`.
+rollback:
+    infra/scripts/rollback-erosi.sh --previous
+
+# Roll back to a specific deploy by sha (must exist in .deploys.log on prod).
+rollback-to sha:
+    infra/scripts/rollback-erosi.sh --to={{sha}}
+
+# Roll back only code (skip DB restore). Useful if the regression is UI-only.
+rollback-code:
+    infra/scripts/rollback-erosi.sh --previous --code-only

 # Stream production Docker logs
 logs:
--- a/docs/deployment.md
+++ b/docs/deployment.md
@@ -45,6 +45,28 @@ ssh ambrosio 'cd /opt/colectivo && docker compose --env-file .env -f infra/docke
 ```
 `KEYCLOAK_CLIENT_SECRET` is runtime-only (GoTrue reads it) — for that one a `docker compose restart auth` is enough.

+### Rollback (code + DB)
+
+Every `just deploy` pairs a code-rsync with a `pg_dumpall` taken just before the build, and logs the pair to `/opt/colectivo/.deploys.log` on ambrosio. Rollback uses that log to restore both atomically.
+
+```sh
+just rollback-list          # show recent deploys (timestamp / sha / backup)
+just rollback               # roll back code + DB to N-1
+just rollback-to <git-sha>  # roll back to a specific deploy
+just rollback-code          # roll back code only, leave DB at current state
+```
+
+Safety:
+- 5-second abort window before anything destructive runs.
+- Verifies the target SHA exists in the local git repo (fetch first if you're rolling back to an old commit).
+- Verifies the matching backup is still on ambrosio (older than 10 deploys = pruned).
+- Stops `app` / `auth` / `rest` / `realtime` / `storage` before the `gunzip | psql` restore so they don't observe inconsistent state.
+- Materialises the target commit into a temp `git worktree`; your working tree is untouched.
+- Does **not** write to `.deploys.log` — rollback is intentionally not a deploy event. The next `just deploy` will be from your current HEAD.
+- The storage volume (user uploads at `/var/lib/storage`) is **not** rolled back. Treated as append-only.
+
+When to use `--code-only`: regression is UI/JS only and the DB schema is forward-compatible. Skips the destructive restore but still rebuilds + redeploys the older app image.
+
 ### Wipe all app + auth data (keep schema, keep migrations)
 When switching Keycloak realms or starting fresh:
 ```sql
--- a/infra/scripts/deploy-erosi.sh
+++ b/infra/scripts/deploy-erosi.sh
@@ -19,6 +19,14 @@
 # it only manages the Docker stack.
 #
 # Subsequent runs: rsync + rebuild app + rolling restart.
+#
+# Pre-deploy safety (every run, before any code/migration changes):
+#   * pg_dumpall to /opt/colectivo/backups/predeploy-<ts>-<sha>.sql.gz
+#   * Keep newest 10 backups, prune older.
+#   * On success, append a line to /opt/colectivo/.deploys.log:
+#       <iso-timestamp>  <git-sha>  <backup-filename>
+# The rollback script (infra/scripts/rollback-erosi.sh) reads .deploys.log
+# to pair (sha, backup) when rolling back code + DB together.

 set -euo pipefail

@@ -109,6 +117,38 @@ if grep -q '^PUBLIC_KEYCLOAK_URL=FILL_IN_' .env \
 fi
 REMOTE

+# ── 3b. Pre-deploy DB backup ──────────────────────────────────────────────
+# Always taken BEFORE the new code/migrations land, so a paired rollback
+# can restore both. Stored on ambrosio at /opt/colectivo/backups/. Keep
+# newest 10; older ones get pruned.
+ssh "$DEPLOY_HOST" env GIT_SHA="$GIT_SHA" bash -s << 'REMOTE'
+set -euo pipefail
+cd /opt/colectivo
+: "${GIT_SHA:=dev}"
+
+mkdir -p backups
+BACKUP_TS=$(date -u +"%Y%m%dT%H%M%SZ")
+BACKUP_FILE="backups/predeploy-${BACKUP_TS}-${GIT_SHA}.sql.gz"
+
+echo "--- Pre-deploy backup → $BACKUP_FILE"
+docker compose --env-file .env -f infra/docker-compose.erosi.yml exec -T db \
+	pg_dumpall -U postgres --clean --if-exists </dev/null \
+	| gzip > "$BACKUP_FILE"
+size=$(stat -c%s "$BACKUP_FILE")
+if [ "$size" -lt 1024 ]; then
+	echo "ERROR: backup is suspiciously small ($size bytes). Aborting deploy." >&2
+	rm -f "$BACKUP_FILE"
+	exit 1
+fi
+echo "    ok ($(numfmt --to=iec --suffix=B "$size" 2>/dev/null || echo "$size bytes"))"
+
+# Prune: keep newest 10 predeploy backups.
+ls -1t backups/predeploy-*.sql.gz 2>/dev/null | tail -n +11 | xargs -r rm -f
+
+# Stash filename for step 4 to record once everything succeeds.
+echo "$BACKUP_FILE" > /tmp/.colectivo-last-backup
+REMOTE
+
 # ── 4. Build + bring up the stack ─────────────────────────────────────────
 # We forward GIT_SHA over the ssh env so the remote `docker compose build`
 # can substitute it into the Dockerfile build-arg (compose reads it from
@@ -165,6 +205,14 @@ for f in supabase/migrations/*.sql; do
 		echo "FAIL"; exit 1
 	fi
 done
+
+# Record the successful deploy in .deploys.log (read by rollback-erosi.sh).
+backup_file=$(cat /tmp/.colectivo-last-backup 2>/dev/null || echo "")
+deploy_ts=$(date -u +"%Y-%m-%dT%H:%M:%SZ")
+printf '%s\t%s\t%s\n' "$deploy_ts" "$GIT_SHA" "$backup_file" \
+	>> /opt/colectivo/.deploys.log
+rm -f /tmp/.colectivo-last-backup
+echo "--- Recorded deploy in /opt/colectivo/.deploys.log"
 REMOTE

 echo "==> Deploy complete."
--- a/infra/scripts/rollback-erosi.sh
+++ b/infra/scripts/rollback-erosi.sh
@@ -0,0 +1,191 @@
+#!/usr/bin/env bash
+# Roll back the ambrosio production stack to a previously-deployed commit.
+#
+# Usage:
+#   infra/scripts/rollback-erosi.sh --list             # print recent deploys
+#   infra/scripts/rollback-erosi.sh --previous         # roll back to N-1
+#   infra/scripts/rollback-erosi.sh --to=<git-sha>     # roll back to a specific deploy
+#   infra/scripts/rollback-erosi.sh --code-only --to=<sha>   # skip DB restore
+#
+# Env:
+#   DEPLOY_HOST=ambrosio (override if needed)
+#   DEPLOY_PATH=/opt/colectivo
+#
+# How it works:
+#   The deploy script (deploy-erosi.sh) writes one line per successful
+#   deploy to /opt/colectivo/.deploys.log:
+#       <iso-ts>  <git-sha>  <backup-file>
+#   Rollback reads that log, picks the row matching --previous or --to,
+#   then:
+#     1. Verifies the SHA is reachable in the local git repo.
+#     2. Verifies the matching backup file still exists on ambrosio.
+#     3. Asks for explicit confirmation (destructive op).
+#     4. Checks out that SHA into a temp worktree on this machine.
+#     5. rsync (excluding .git, etc.) → ambrosio.
+#     6. Restores the DB from the matched backup.
+#     7. Rebuilds the app image at the rollback SHA and brings the stack
+#        up. NOTE: --code-only skips step 6.
+#     8. Cleans up the worktree. Does NOT write a new line to
+#        .deploys.log — rollback is intentionally not deploy-equivalent.
+#
+# What is NOT rolled back:
+#   * Storage volume (/var/lib/storage) — user uploads. Treated as
+#     append-only; backup taken pre-deploy but never restored on rollback.
+#   * Realtime tenant state, Kong runtime config. They are stateless or
+#     re-bootstrap from the DB.
+
+set -euo pipefail
+
+DEPLOY_HOST="${DEPLOY_HOST:-ambrosio}"
+DEPLOY_PATH="${DEPLOY_PATH:-/opt/colectivo}"
+REPO_ROOT="$(cd "$(dirname "$0")/../.." && pwd)"
+
+MODE=""
+TARGET_SHA=""
+CODE_ONLY=0
+
+while [ $# -gt 0 ]; do
+	case "$1" in
+		--list)        MODE="list" ;;
+		--previous)    MODE="previous" ;;
+		--to=*)        MODE="to"; TARGET_SHA="${1#--to=}" ;;
+		--code-only)   CODE_ONLY=1 ;;
+		-h|--help)
+			grep '^#' "$0" | sed 's/^# \{0,1\}//' | head -40
+			exit 0
+			;;
+		*) echo "Unknown arg: $1" >&2; exit 1 ;;
+	esac
+	shift
+done
+
+[ -z "$MODE" ] && { echo "Specify --list, --previous, or --to=<sha>." >&2; exit 1; }
+
+# ── 1. Read deploy log from ambrosio ──────────────────────────────────────
+log=$(ssh "$DEPLOY_HOST" "cat $DEPLOY_PATH/.deploys.log 2>/dev/null || true")
+if [ -z "$log" ]; then
+	echo "No deploy history found at $DEPLOY_HOST:$DEPLOY_PATH/.deploys.log" >&2
+	exit 1
+fi
+
+if [ "$MODE" = "list" ]; then
+	echo "Recent deploys ($DEPLOY_HOST):"
+	echo "------------------------------------------------------------"
+	printf '%s\n' "$log" | nl -ba
+	exit 0
+fi
+
+# ── 2. Pick the target row ────────────────────────────────────────────────
+if [ "$MODE" = "previous" ]; then
+	# Last line = current deploy. Second-to-last = previous.
+	target_row=$(printf '%s\n' "$log" | tail -n 2 | head -n 1)
+elif [ "$MODE" = "to" ]; then
+	target_row=$(printf '%s\n' "$log" | grep -F "	$TARGET_SHA	" | tail -n 1 || true)
+fi
+
+if [ -z "${target_row:-}" ]; then
+	echo "No matching deploy found." >&2
+	exit 1
+fi
+
+target_ts=$(printf '%s' "$target_row" | cut -f1)
+target_sha=$(printf '%s' "$target_row" | cut -f2)
+target_backup=$(printf '%s' "$target_row" | cut -f3)
+
+echo "==> Rollback target"
+echo "    timestamp: $target_ts"
+echo "    git sha:   $target_sha"
+echo "    backup:    $target_backup"
+
+# ── 3. Verify SHA + backup ───────────────────────────────────────────────
+if ! git -C "$REPO_ROOT" cat-file -e "${target_sha}^{commit}" 2>/dev/null; then
+	echo "ERROR: commit $target_sha not present in local repo. Fetch first." >&2
+	exit 1
+fi
+
+if [ "$CODE_ONLY" -eq 0 ]; then
+	if ! ssh "$DEPLOY_HOST" "test -f $DEPLOY_PATH/$target_backup"; then
+		echo "ERROR: backup $target_backup missing on $DEPLOY_HOST." >&2
+		echo "       Use --code-only to skip the DB restore." >&2
+		exit 1
+	fi
+fi
+
+# ── 4. Confirm ────────────────────────────────────────────────────────────
+echo ""
+echo "WARNING: This is DESTRUCTIVE."
+if [ "$CODE_ONLY" -eq 1 ]; then
+	echo "         Code will be rolled back to $target_sha; DB is NOT touched."
+else
+	echo "         Code AND database will be rolled back."
+	echo "         Current DB state will be overwritten by $target_backup."
+fi
+echo "         5s window to abort (Ctrl-C)..."
+sleep 5
+
+# ── 5. Worktree at the target SHA ────────────────────────────────────────
+worktree_dir=$(mktemp -d -t colectivo-rollback-XXXXXX)
+trap 'git -C "$REPO_ROOT" worktree remove --force "$worktree_dir" 2>/dev/null || rm -rf "$worktree_dir"' EXIT
+
+echo "--- Materialising worktree at $target_sha → $worktree_dir"
+git -C "$REPO_ROOT" worktree add --detach "$worktree_dir" "$target_sha" >/dev/null
+
+# ── 6. rsync worktree → ambrosio ─────────────────────────────────────────
+echo "--- rsync code to $DEPLOY_HOST:$DEPLOY_PATH"
+rsync -az --delete \
+	--exclude '.git' \
+	--exclude 'node_modules' \
+	--exclude '.svelte-kit' \
+	--exclude 'apps/web/build' \
+	--exclude 'apps/web/.vite' \
+	--exclude '**/*.log' \
+	--exclude '**/.env' \
+	--exclude '**/.env.development' \
+	--exclude '**/.env.local' \
+	--exclude 'backups' \
+	--exclude '.deploys.log' \
+	"$worktree_dir/" "$DEPLOY_HOST:$DEPLOY_PATH/"
+
+# ── 7. Restore DB (unless --code-only) ───────────────────────────────────
+if [ "$CODE_ONLY" -eq 0 ]; then
+	echo "--- Stopping app + auth + rest before DB restore"
+	ssh "$DEPLOY_HOST" bash -s << REMOTE
+set -euo pipefail
+cd $DEPLOY_PATH
+docker compose --env-file .env -f infra/docker-compose.erosi.yml stop app auth rest realtime storage
+REMOTE
+
+	echo "--- Restoring DB from $target_backup"
+	ssh "$DEPLOY_HOST" bash -s << REMOTE
+set -euo pipefail
+cd $DEPLOY_PATH
+gunzip -c $target_backup \
+	| docker compose --env-file .env -f infra/docker-compose.erosi.yml exec -T db \
+		psql -U postgres -d postgres -v ON_ERROR_STOP=1
+REMOTE
+fi
+
+# ── 8. Rebuild + bring the stack back up at the rollback SHA ────────────
+echo "--- Rebuild app at $target_sha + bring stack up"
+ssh "$DEPLOY_HOST" env GIT_SHA="$target_sha" bash -s << 'REMOTE'
+set -euo pipefail
+cd /opt/colectivo
+: "${GIT_SHA:=dev}"
+
+GIT_SHA="$GIT_SHA" docker compose --env-file .env -f infra/docker-compose.erosi.yml \
+	build --build-arg GIT_SHA="$GIT_SHA" app
+GIT_SHA="$GIT_SHA" docker compose --env-file .env -f infra/docker-compose.erosi.yml up -d
+REMOTE
+
+echo ""
+echo "==> Rollback complete."
+echo "    code:  $target_sha"
+if [ "$CODE_ONLY" -eq 1 ]; then
+	echo "    db:    UNCHANGED (--code-only)"
+else
+	echo "    db:    restored from $target_backup"
+fi
+echo "    App:   https://erosi.limonia.net"
+echo ""
+echo "NOTE: .deploys.log is intentionally not updated by rollback."
+echo "      The next 'just deploy' will be from your current HEAD."