diff --git a/.claude/skills/deploy/SKILL.md b/.claude/skills/deploy/SKILL.md new file mode 100644 index 0000000..c1edc72 --- /dev/null +++ b/.claude/skills/deploy/SKILL.md @@ -0,0 +1,76 @@ +--- +name: deploy +description: Ship the colectivo stack to ambrosio (https://erosi.limonia.net) with pre-deploy DB backup, or roll back code + DB to a previous deploy. Trigger when the user asks to deploy, ship, push to prod, roll back, or undo a deploy. +--- + +# Deploy + rollback skill + +Deploys and rollbacks for the production stack on ambrosio. **Every deploy takes a paired DB backup first**, every successful deploy is logged with `(timestamp, git-sha, backup-filename)`, and rollback uses that log to restore both code and DB atomically. + +## Trigger + +Use this skill when the user says any of: +- "deploy", "deploy to ambrosio", "deploy to prod", "ship it" +- "roll back", "rollback", "undo the deploy", "revert prod" +- "list deploys", "what was the last deploy" + +## What the skill does + +### Forward (deploy) + +1. **`just deploy`** runs `infra/scripts/deploy-erosi.sh`: + - Captures the local short SHA (`git rev-parse --short HEAD`). + - rsyncs the working tree to `ambrosio:/opt/colectivo/` (excludes `.git`, `node_modules`, `**/.env`, etc.). + - On a fresh `.env`, generates secrets and bails — operator fills `PUBLIC_KEYCLOAK_URL` + `KEYCLOAK_CLIENT_SECRET` and re-runs. + - **Pre-deploy DB backup**: `pg_dumpall` from the prod `db` container → `backups/predeploy--.sql.gz` on ambrosio. Aborts the deploy if the dump is < 1 KiB. Keeps the newest 10 backups. + - Rebuilds the `app` image with `GIT_SHA` build-arg (so `__APP_COMMIT__` lands in the bundle, Fase 14.3). + - `docker compose up -d` brings the stack to the new image. + - Applies any pending DB migrations. + - On success, appends `\t\t` to `/opt/colectivo/.deploys.log`. +2. Site smoke: `curl -I https://erosi.limonia.net/` should return 200. + +### Reverse (rollback) + +1. **`just rollback-list`** prints recent rows from `.deploys.log` so the user can pick a target. +2. **`just rollback`** rolls back code + DB to the previous deploy (last but one line of `.deploys.log`). +3. **`just rollback-to `** rolls back to a specific deploy. +4. **`just rollback-code`** rolls back code only, leaves the DB at current state — useful when the regression is UI-only and the DB schema is compatible. + +`infra/scripts/rollback-erosi.sh` enforces: +- Confirms with a 5-second abort window before doing anything destructive. +- Verifies the target SHA is reachable locally (`git cat-file -e`) and that the backup file still exists on ambrosio. +- Materialises the target commit into a temporary `git worktree`, rsyncs from there, never disturbs the user's working tree. +- Stops `app / auth / rest / realtime / storage` before the DB restore so they don't observe inconsistent state. +- `gunzip | psql -v ON_ERROR_STOP=1` restores the dump. +- Rebuilds the app image at the rolled-back SHA and brings the stack up. +- **Does not** write a new `.deploys.log` line — rollback is intentionally not a deploy event. + +## Safety boundaries + +- The stack lives on ambrosio. The external proxy + TLS terminator are off-stack (`docs/deployment.md`). Neither this skill nor the scripts ever touch host TLS or the external proxy. +- `infra/scripts/deploy-erosi.sh` never runs `sudo`. +- `--no-verify` / `--amend` are never used. Pre-commit hooks must pass. +- Always confirm with the user before invoking `just rollback*` — DB restore is destructive and irreversible. +- Storage volume (`/var/lib/storage`, user uploads) is **not** part of the rollback. Treated as append-only. + +## What to tell the user + +Before running anything: +- Show the local short SHA + branch + remote. +- For deploy: confirm with the user. Mention "this will take a fresh DB backup before deploying." +- For rollback: show `just rollback-list` output, ask which SHA, confirm destructive nature. + +After running: +- Print the new `.deploys.log` tail (or the rollback target). +- Curl-probe the site and report. + +## Files this skill touches + +- `infra/scripts/deploy-erosi.sh` — deploy, with pre-deploy backup + .deploys.log writing. +- `infra/scripts/rollback-erosi.sh` — rollback, paired code + DB. +- `Justfile` — `deploy`, `rollback`, `rollback-list`, `rollback-to`, `rollback-code` recipes. +- `docs/deployment.md` — runbook for both flows. + +## Anti-trigger + +If the user is just asking *what would happen* on deploy (status, dry-run-style questions), DO NOT run anything. Answer from the script content. diff --git a/.gitignore b/.gitignore index 4b9b418..8bf2e39 100644 --- a/.gitignore +++ b/.gitignore @@ -42,6 +42,9 @@ apps/web/tests/.auth/ # PWA dev-mode service worker output apps/web/dev-dist/ -# Claude Code per-project runtime state (schedulers etc.) -.claude/ +# Claude Code per-project runtime state (schedulers etc.). Project-scoped +# skills under .claude/skills/ ARE tracked — they document workflows for +# anyone working on the repo. +.claude/* +!.claude/skills/ graphify-out/cache/ diff --git a/Justfile b/Justfile index bcfa1b4..b0f6eae 100644 --- a/Justfile +++ b/Justfile @@ -203,9 +203,28 @@ restore service file: # ── Production ──────────────────────────────────────────────────────────────── -# Deploy to production via SSH +# Deploy to production via SSH (ambrosio / erosi.limonia.net). +# Pre-deploy: dumps the prod DB to /opt/colectivo/backups/predeploy--.sql.gz +# Post-deploy: appends (timestamp, sha, backup) to /opt/colectivo/.deploys.log deploy: - infra/scripts/deploy.sh + infra/scripts/deploy-erosi.sh + +# List recent prod deploys (timestamp / sha / backup-file). +rollback-list: + infra/scripts/rollback-erosi.sh --list + +# Roll back code AND database to the previous deploy. +# For a specific deploy: `just rollback-to `. +rollback: + infra/scripts/rollback-erosi.sh --previous + +# Roll back to a specific deploy by sha (must exist in .deploys.log on prod). +rollback-to sha: + infra/scripts/rollback-erosi.sh --to={{sha}} + +# Roll back only code (skip DB restore). Useful if the regression is UI-only. +rollback-code: + infra/scripts/rollback-erosi.sh --previous --code-only # Stream production Docker logs logs: diff --git a/docs/deployment.md b/docs/deployment.md index f00ac3b..b227484 100644 --- a/docs/deployment.md +++ b/docs/deployment.md @@ -45,6 +45,28 @@ ssh ambrosio 'cd /opt/colectivo && docker compose --env-file .env -f infra/docke ``` `KEYCLOAK_CLIENT_SECRET` is runtime-only (GoTrue reads it) — for that one a `docker compose restart auth` is enough. +### Rollback (code + DB) + +Every `just deploy` pairs a code-rsync with a `pg_dumpall` taken just before the build, and logs the pair to `/opt/colectivo/.deploys.log` on ambrosio. Rollback uses that log to restore both atomically. + +```sh +just rollback-list # show recent deploys (timestamp / sha / backup) +just rollback # roll back code + DB to N-1 +just rollback-to # roll back to a specific deploy +just rollback-code # roll back code only, leave DB at current state +``` + +Safety: +- 5-second abort window before anything destructive runs. +- Verifies the target SHA exists in the local git repo (fetch first if you're rolling back to an old commit). +- Verifies the matching backup is still on ambrosio (older than 10 deploys = pruned). +- Stops `app` / `auth` / `rest` / `realtime` / `storage` before the `gunzip | psql` restore so they don't observe inconsistent state. +- Materialises the target commit into a temp `git worktree`; your working tree is untouched. +- Does **not** write to `.deploys.log` — rollback is intentionally not a deploy event. The next `just deploy` will be from your current HEAD. +- The storage volume (user uploads at `/var/lib/storage`) is **not** rolled back. Treated as append-only. + +When to use `--code-only`: regression is UI/JS only and the DB schema is forward-compatible. Skips the destructive restore but still rebuilds + redeploys the older app image. + ### Wipe all app + auth data (keep schema, keep migrations) When switching Keycloak realms or starting fresh: ```sql diff --git a/infra/scripts/deploy-erosi.sh b/infra/scripts/deploy-erosi.sh index ae88da9..3d5ea1c 100755 --- a/infra/scripts/deploy-erosi.sh +++ b/infra/scripts/deploy-erosi.sh @@ -19,6 +19,14 @@ # it only manages the Docker stack. # # Subsequent runs: rsync + rebuild app + rolling restart. +# +# Pre-deploy safety (every run, before any code/migration changes): +# * pg_dumpall to /opt/colectivo/backups/predeploy--.sql.gz +# * Keep newest 10 backups, prune older. +# * On success, append a line to /opt/colectivo/.deploys.log: +# +# The rollback script (infra/scripts/rollback-erosi.sh) reads .deploys.log +# to pair (sha, backup) when rolling back code + DB together. set -euo pipefail @@ -109,6 +117,38 @@ if grep -q '^PUBLIC_KEYCLOAK_URL=FILL_IN_' .env \ fi REMOTE +# ── 3b. Pre-deploy DB backup ────────────────────────────────────────────── +# Always taken BEFORE the new code/migrations land, so a paired rollback +# can restore both. Stored on ambrosio at /opt/colectivo/backups/. Keep +# newest 10; older ones get pruned. +ssh "$DEPLOY_HOST" env GIT_SHA="$GIT_SHA" bash -s << 'REMOTE' +set -euo pipefail +cd /opt/colectivo +: "${GIT_SHA:=dev}" + +mkdir -p backups +BACKUP_TS=$(date -u +"%Y%m%dT%H%M%SZ") +BACKUP_FILE="backups/predeploy-${BACKUP_TS}-${GIT_SHA}.sql.gz" + +echo "--- Pre-deploy backup → $BACKUP_FILE" +docker compose --env-file .env -f infra/docker-compose.erosi.yml exec -T db \ + pg_dumpall -U postgres --clean --if-exists "$BACKUP_FILE" +size=$(stat -c%s "$BACKUP_FILE") +if [ "$size" -lt 1024 ]; then + echo "ERROR: backup is suspiciously small ($size bytes). Aborting deploy." >&2 + rm -f "$BACKUP_FILE" + exit 1 +fi +echo " ok ($(numfmt --to=iec --suffix=B "$size" 2>/dev/null || echo "$size bytes"))" + +# Prune: keep newest 10 predeploy backups. +ls -1t backups/predeploy-*.sql.gz 2>/dev/null | tail -n +11 | xargs -r rm -f + +# Stash filename for step 4 to record once everything succeeds. +echo "$BACKUP_FILE" > /tmp/.colectivo-last-backup +REMOTE + # ── 4. Build + bring up the stack ───────────────────────────────────────── # We forward GIT_SHA over the ssh env so the remote `docker compose build` # can substitute it into the Dockerfile build-arg (compose reads it from @@ -165,6 +205,14 @@ for f in supabase/migrations/*.sql; do echo "FAIL"; exit 1 fi done + +# Record the successful deploy in .deploys.log (read by rollback-erosi.sh). +backup_file=$(cat /tmp/.colectivo-last-backup 2>/dev/null || echo "") +deploy_ts=$(date -u +"%Y-%m-%dT%H:%M:%SZ") +printf '%s\t%s\t%s\n' "$deploy_ts" "$GIT_SHA" "$backup_file" \ + >> /opt/colectivo/.deploys.log +rm -f /tmp/.colectivo-last-backup +echo "--- Recorded deploy in /opt/colectivo/.deploys.log" REMOTE echo "==> Deploy complete." diff --git a/infra/scripts/rollback-erosi.sh b/infra/scripts/rollback-erosi.sh new file mode 100755 index 0000000..c9ffdcc --- /dev/null +++ b/infra/scripts/rollback-erosi.sh @@ -0,0 +1,191 @@ +#!/usr/bin/env bash +# Roll back the ambrosio production stack to a previously-deployed commit. +# +# Usage: +# infra/scripts/rollback-erosi.sh --list # print recent deploys +# infra/scripts/rollback-erosi.sh --previous # roll back to N-1 +# infra/scripts/rollback-erosi.sh --to= # roll back to a specific deploy +# infra/scripts/rollback-erosi.sh --code-only --to= # skip DB restore +# +# Env: +# DEPLOY_HOST=ambrosio (override if needed) +# DEPLOY_PATH=/opt/colectivo +# +# How it works: +# The deploy script (deploy-erosi.sh) writes one line per successful +# deploy to /opt/colectivo/.deploys.log: +# +# Rollback reads that log, picks the row matching --previous or --to, +# then: +# 1. Verifies the SHA is reachable in the local git repo. +# 2. Verifies the matching backup file still exists on ambrosio. +# 3. Asks for explicit confirmation (destructive op). +# 4. Checks out that SHA into a temp worktree on this machine. +# 5. rsync (excluding .git, etc.) → ambrosio. +# 6. Restores the DB from the matched backup. +# 7. Rebuilds the app image at the rollback SHA and brings the stack +# up. NOTE: --code-only skips step 6. +# 8. Cleans up the worktree. Does NOT write a new line to +# .deploys.log — rollback is intentionally not deploy-equivalent. +# +# What is NOT rolled back: +# * Storage volume (/var/lib/storage) — user uploads. Treated as +# append-only; backup taken pre-deploy but never restored on rollback. +# * Realtime tenant state, Kong runtime config. They are stateless or +# re-bootstrap from the DB. + +set -euo pipefail + +DEPLOY_HOST="${DEPLOY_HOST:-ambrosio}" +DEPLOY_PATH="${DEPLOY_PATH:-/opt/colectivo}" +REPO_ROOT="$(cd "$(dirname "$0")/../.." && pwd)" + +MODE="" +TARGET_SHA="" +CODE_ONLY=0 + +while [ $# -gt 0 ]; do + case "$1" in + --list) MODE="list" ;; + --previous) MODE="previous" ;; + --to=*) MODE="to"; TARGET_SHA="${1#--to=}" ;; + --code-only) CODE_ONLY=1 ;; + -h|--help) + grep '^#' "$0" | sed 's/^# \{0,1\}//' | head -40 + exit 0 + ;; + *) echo "Unknown arg: $1" >&2; exit 1 ;; + esac + shift +done + +[ -z "$MODE" ] && { echo "Specify --list, --previous, or --to=." >&2; exit 1; } + +# ── 1. Read deploy log from ambrosio ────────────────────────────────────── +log=$(ssh "$DEPLOY_HOST" "cat $DEPLOY_PATH/.deploys.log 2>/dev/null || true") +if [ -z "$log" ]; then + echo "No deploy history found at $DEPLOY_HOST:$DEPLOY_PATH/.deploys.log" >&2 + exit 1 +fi + +if [ "$MODE" = "list" ]; then + echo "Recent deploys ($DEPLOY_HOST):" + echo "------------------------------------------------------------" + printf '%s\n' "$log" | nl -ba + exit 0 +fi + +# ── 2. Pick the target row ──────────────────────────────────────────────── +if [ "$MODE" = "previous" ]; then + # Last line = current deploy. Second-to-last = previous. + target_row=$(printf '%s\n' "$log" | tail -n 2 | head -n 1) +elif [ "$MODE" = "to" ]; then + target_row=$(printf '%s\n' "$log" | grep -F " $TARGET_SHA " | tail -n 1 || true) +fi + +if [ -z "${target_row:-}" ]; then + echo "No matching deploy found." >&2 + exit 1 +fi + +target_ts=$(printf '%s' "$target_row" | cut -f1) +target_sha=$(printf '%s' "$target_row" | cut -f2) +target_backup=$(printf '%s' "$target_row" | cut -f3) + +echo "==> Rollback target" +echo " timestamp: $target_ts" +echo " git sha: $target_sha" +echo " backup: $target_backup" + +# ── 3. Verify SHA + backup ─────────────────────────────────────────────── +if ! git -C "$REPO_ROOT" cat-file -e "${target_sha}^{commit}" 2>/dev/null; then + echo "ERROR: commit $target_sha not present in local repo. Fetch first." >&2 + exit 1 +fi + +if [ "$CODE_ONLY" -eq 0 ]; then + if ! ssh "$DEPLOY_HOST" "test -f $DEPLOY_PATH/$target_backup"; then + echo "ERROR: backup $target_backup missing on $DEPLOY_HOST." >&2 + echo " Use --code-only to skip the DB restore." >&2 + exit 1 + fi +fi + +# ── 4. Confirm ──────────────────────────────────────────────────────────── +echo "" +echo "WARNING: This is DESTRUCTIVE." +if [ "$CODE_ONLY" -eq 1 ]; then + echo " Code will be rolled back to $target_sha; DB is NOT touched." +else + echo " Code AND database will be rolled back." + echo " Current DB state will be overwritten by $target_backup." +fi +echo " 5s window to abort (Ctrl-C)..." +sleep 5 + +# ── 5. Worktree at the target SHA ──────────────────────────────────────── +worktree_dir=$(mktemp -d -t colectivo-rollback-XXXXXX) +trap 'git -C "$REPO_ROOT" worktree remove --force "$worktree_dir" 2>/dev/null || rm -rf "$worktree_dir"' EXIT + +echo "--- Materialising worktree at $target_sha → $worktree_dir" +git -C "$REPO_ROOT" worktree add --detach "$worktree_dir" "$target_sha" >/dev/null + +# ── 6. rsync worktree → ambrosio ───────────────────────────────────────── +echo "--- rsync code to $DEPLOY_HOST:$DEPLOY_PATH" +rsync -az --delete \ + --exclude '.git' \ + --exclude 'node_modules' \ + --exclude '.svelte-kit' \ + --exclude 'apps/web/build' \ + --exclude 'apps/web/.vite' \ + --exclude '**/*.log' \ + --exclude '**/.env' \ + --exclude '**/.env.development' \ + --exclude '**/.env.local' \ + --exclude 'backups' \ + --exclude '.deploys.log' \ + "$worktree_dir/" "$DEPLOY_HOST:$DEPLOY_PATH/" + +# ── 7. Restore DB (unless --code-only) ─────────────────────────────────── +if [ "$CODE_ONLY" -eq 0 ]; then + echo "--- Stopping app + auth + rest before DB restore" + ssh "$DEPLOY_HOST" bash -s << REMOTE +set -euo pipefail +cd $DEPLOY_PATH +docker compose --env-file .env -f infra/docker-compose.erosi.yml stop app auth rest realtime storage +REMOTE + + echo "--- Restoring DB from $target_backup" + ssh "$DEPLOY_HOST" bash -s << REMOTE +set -euo pipefail +cd $DEPLOY_PATH +gunzip -c $target_backup \ + | docker compose --env-file .env -f infra/docker-compose.erosi.yml exec -T db \ + psql -U postgres -d postgres -v ON_ERROR_STOP=1 +REMOTE +fi + +# ── 8. Rebuild + bring the stack back up at the rollback SHA ──────────── +echo "--- Rebuild app at $target_sha + bring stack up" +ssh "$DEPLOY_HOST" env GIT_SHA="$target_sha" bash -s << 'REMOTE' +set -euo pipefail +cd /opt/colectivo +: "${GIT_SHA:=dev}" + +GIT_SHA="$GIT_SHA" docker compose --env-file .env -f infra/docker-compose.erosi.yml \ + build --build-arg GIT_SHA="$GIT_SHA" app +GIT_SHA="$GIT_SHA" docker compose --env-file .env -f infra/docker-compose.erosi.yml up -d +REMOTE + +echo "" +echo "==> Rollback complete." +echo " code: $target_sha" +if [ "$CODE_ONLY" -eq 1 ]; then + echo " db: UNCHANGED (--code-only)" +else + echo " db: restored from $target_backup" +fi +echo " App: https://erosi.limonia.net" +echo "" +echo "NOTE: .deploys.log is intentionally not updated by rollback." +echo " The next 'just deploy' will be from your current HEAD."