deploy(prod): pre-deploy DB backup + paired code/DB rollback skill

Every `just deploy` now takes a pg_dumpall to
/opt/colectivo/backups/predeploy-<ts>-<sha>.sql.gz BEFORE rebuilding the
app or applying migrations, aborts the deploy if the dump is < 1 KiB,
and (on success) appends `<iso-ts> \t <sha> \t <backup-file>` to
/opt/colectivo/.deploys.log on ambrosio. Keeps the newest 10 backups
(prunes older predeploy-* files).

New `infra/scripts/rollback-erosi.sh` reads .deploys.log and pairs a
code rollback with a DB restore atomically:

  just rollback-list          # show recent deploys
  just rollback               # roll back to N-1
  just rollback-to <sha>      # roll back to a specific deploy
  just rollback-code          # roll back code only, keep current DB

Rollback safety:
- 5-second abort window.
- Verifies the target SHA exists locally + the backup is still on prod
  (warns if it's been pruned past the 10-deploy window).
- Uses a temporary git worktree so the user's working tree isn't
  disturbed.
- Stops app/auth/rest/realtime/storage before the gunzip|psql restore.
- Rebuilds the app image at the rolled-back SHA with the same GIT_SHA
  build-arg path the deploy uses (so __APP_COMMIT__ in the bundle
  matches the running code).
- Does NOT write a new .deploys.log entry — rollback is intentionally
  not a deploy event; the next `just deploy` is from current HEAD.
- Storage volume (/var/lib/storage user uploads) is NOT rolled back.

Justfile `deploy` recipe repointed from the stale `deploy.sh` (which
referenced GHCR pull) to `deploy-erosi.sh` (the active prod path).

New project-scoped skill: `.claude/skills/deploy/SKILL.md` documents the
flow + safety boundaries for Claude-assisted invocations. `.gitignore`
keeps `.claude/settings.local.json` ignored but tracks `.claude/skills/`.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
This commit is contained in:
2026-05-18 14:57:25 +02:00
parent c0e5b5ed7f
commit ae4fd45b99
6 changed files with 363 additions and 4 deletions

View File

@@ -19,6 +19,14 @@
# it only manages the Docker stack.
#
# Subsequent runs: rsync + rebuild app + rolling restart.
#
# Pre-deploy safety (every run, before any code/migration changes):
# * pg_dumpall to /opt/colectivo/backups/predeploy-<ts>-<sha>.sql.gz
# * Keep newest 10 backups, prune older.
# * On success, append a line to /opt/colectivo/.deploys.log:
# <iso-timestamp> <git-sha> <backup-filename>
# The rollback script (infra/scripts/rollback-erosi.sh) reads .deploys.log
# to pair (sha, backup) when rolling back code + DB together.
set -euo pipefail
@@ -109,6 +117,38 @@ if grep -q '^PUBLIC_KEYCLOAK_URL=FILL_IN_' .env \
fi
REMOTE
# ── 3b. Pre-deploy DB backup ──────────────────────────────────────────────
# Always taken BEFORE the new code/migrations land, so a paired rollback
# can restore both. Stored on ambrosio at /opt/colectivo/backups/. Keep
# newest 10; older ones get pruned.
ssh "$DEPLOY_HOST" env GIT_SHA="$GIT_SHA" bash -s << 'REMOTE'
set -euo pipefail
cd /opt/colectivo
: "${GIT_SHA:=dev}"
mkdir -p backups
BACKUP_TS=$(date -u +"%Y%m%dT%H%M%SZ")
BACKUP_FILE="backups/predeploy-${BACKUP_TS}-${GIT_SHA}.sql.gz"
echo "--- Pre-deploy backup → $BACKUP_FILE"
docker compose --env-file .env -f infra/docker-compose.erosi.yml exec -T db \
pg_dumpall -U postgres --clean --if-exists </dev/null \
| gzip > "$BACKUP_FILE"
size=$(stat -c%s "$BACKUP_FILE")
if [ "$size" -lt 1024 ]; then
echo "ERROR: backup is suspiciously small ($size bytes). Aborting deploy." >&2
rm -f "$BACKUP_FILE"
exit 1
fi
echo " ok ($(numfmt --to=iec --suffix=B "$size" 2>/dev/null || echo "$size bytes"))"
# Prune: keep newest 10 predeploy backups.
ls -1t backups/predeploy-*.sql.gz 2>/dev/null | tail -n +11 | xargs -r rm -f
# Stash filename for step 4 to record once everything succeeds.
echo "$BACKUP_FILE" > /tmp/.colectivo-last-backup
REMOTE
# ── 4. Build + bring up the stack ─────────────────────────────────────────
# We forward GIT_SHA over the ssh env so the remote `docker compose build`
# can substitute it into the Dockerfile build-arg (compose reads it from
@@ -165,6 +205,14 @@ for f in supabase/migrations/*.sql; do
echo "FAIL"; exit 1
fi
done
# Record the successful deploy in .deploys.log (read by rollback-erosi.sh).
backup_file=$(cat /tmp/.colectivo-last-backup 2>/dev/null || echo "")
deploy_ts=$(date -u +"%Y-%m-%dT%H:%M:%SZ")
printf '%s\t%s\t%s\n' "$deploy_ts" "$GIT_SHA" "$backup_file" \
>> /opt/colectivo/.deploys.log
rm -f /tmp/.colectivo-last-backup
echo "--- Recorded deploy in /opt/colectivo/.deploys.log"
REMOTE
echo "==> Deploy complete."