deploy(prod): pre-deploy DB backup + paired code/DB rollback skill
Every `just deploy` now takes a pg_dumpall to /opt/colectivo/backups/predeploy-<ts>-<sha>.sql.gz BEFORE rebuilding the app or applying migrations, aborts the deploy if the dump is < 1 KiB, and (on success) appends `<iso-ts> \t <sha> \t <backup-file>` to /opt/colectivo/.deploys.log on ambrosio. Keeps the newest 10 backups (prunes older predeploy-* files). New `infra/scripts/rollback-erosi.sh` reads .deploys.log and pairs a code rollback with a DB restore atomically: just rollback-list # show recent deploys just rollback # roll back to N-1 just rollback-to <sha> # roll back to a specific deploy just rollback-code # roll back code only, keep current DB Rollback safety: - 5-second abort window. - Verifies the target SHA exists locally + the backup is still on prod (warns if it's been pruned past the 10-deploy window). - Uses a temporary git worktree so the user's working tree isn't disturbed. - Stops app/auth/rest/realtime/storage before the gunzip|psql restore. - Rebuilds the app image at the rolled-back SHA with the same GIT_SHA build-arg path the deploy uses (so __APP_COMMIT__ in the bundle matches the running code). - Does NOT write a new .deploys.log entry — rollback is intentionally not a deploy event; the next `just deploy` is from current HEAD. - Storage volume (/var/lib/storage user uploads) is NOT rolled back. Justfile `deploy` recipe repointed from the stale `deploy.sh` (which referenced GHCR pull) to `deploy-erosi.sh` (the active prod path). New project-scoped skill: `.claude/skills/deploy/SKILL.md` documents the flow + safety boundaries for Claude-assisted invocations. `.gitignore` keeps `.claude/settings.local.json` ignored but tracks `.claude/skills/`. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
This commit is contained in:
@@ -19,6 +19,14 @@
|
||||
# it only manages the Docker stack.
|
||||
#
|
||||
# Subsequent runs: rsync + rebuild app + rolling restart.
|
||||
#
|
||||
# Pre-deploy safety (every run, before any code/migration changes):
|
||||
# * pg_dumpall to /opt/colectivo/backups/predeploy-<ts>-<sha>.sql.gz
|
||||
# * Keep newest 10 backups, prune older.
|
||||
# * On success, append a line to /opt/colectivo/.deploys.log:
|
||||
# <iso-timestamp> <git-sha> <backup-filename>
|
||||
# The rollback script (infra/scripts/rollback-erosi.sh) reads .deploys.log
|
||||
# to pair (sha, backup) when rolling back code + DB together.
|
||||
|
||||
set -euo pipefail
|
||||
|
||||
@@ -109,6 +117,38 @@ if grep -q '^PUBLIC_KEYCLOAK_URL=FILL_IN_' .env \
|
||||
fi
|
||||
REMOTE
|
||||
|
||||
# ── 3b. Pre-deploy DB backup ──────────────────────────────────────────────
|
||||
# Always taken BEFORE the new code/migrations land, so a paired rollback
|
||||
# can restore both. Stored on ambrosio at /opt/colectivo/backups/. Keep
|
||||
# newest 10; older ones get pruned.
|
||||
ssh "$DEPLOY_HOST" env GIT_SHA="$GIT_SHA" bash -s << 'REMOTE'
|
||||
set -euo pipefail
|
||||
cd /opt/colectivo
|
||||
: "${GIT_SHA:=dev}"
|
||||
|
||||
mkdir -p backups
|
||||
BACKUP_TS=$(date -u +"%Y%m%dT%H%M%SZ")
|
||||
BACKUP_FILE="backups/predeploy-${BACKUP_TS}-${GIT_SHA}.sql.gz"
|
||||
|
||||
echo "--- Pre-deploy backup → $BACKUP_FILE"
|
||||
docker compose --env-file .env -f infra/docker-compose.erosi.yml exec -T db \
|
||||
pg_dumpall -U postgres --clean --if-exists </dev/null \
|
||||
| gzip > "$BACKUP_FILE"
|
||||
size=$(stat -c%s "$BACKUP_FILE")
|
||||
if [ "$size" -lt 1024 ]; then
|
||||
echo "ERROR: backup is suspiciously small ($size bytes). Aborting deploy." >&2
|
||||
rm -f "$BACKUP_FILE"
|
||||
exit 1
|
||||
fi
|
||||
echo " ok ($(numfmt --to=iec --suffix=B "$size" 2>/dev/null || echo "$size bytes"))"
|
||||
|
||||
# Prune: keep newest 10 predeploy backups.
|
||||
ls -1t backups/predeploy-*.sql.gz 2>/dev/null | tail -n +11 | xargs -r rm -f
|
||||
|
||||
# Stash filename for step 4 to record once everything succeeds.
|
||||
echo "$BACKUP_FILE" > /tmp/.colectivo-last-backup
|
||||
REMOTE
|
||||
|
||||
# ── 4. Build + bring up the stack ─────────────────────────────────────────
|
||||
# We forward GIT_SHA over the ssh env so the remote `docker compose build`
|
||||
# can substitute it into the Dockerfile build-arg (compose reads it from
|
||||
@@ -165,6 +205,14 @@ for f in supabase/migrations/*.sql; do
|
||||
echo "FAIL"; exit 1
|
||||
fi
|
||||
done
|
||||
|
||||
# Record the successful deploy in .deploys.log (read by rollback-erosi.sh).
|
||||
backup_file=$(cat /tmp/.colectivo-last-backup 2>/dev/null || echo "")
|
||||
deploy_ts=$(date -u +"%Y-%m-%dT%H:%M:%SZ")
|
||||
printf '%s\t%s\t%s\n' "$deploy_ts" "$GIT_SHA" "$backup_file" \
|
||||
>> /opt/colectivo/.deploys.log
|
||||
rm -f /tmp/.colectivo-last-backup
|
||||
echo "--- Recorded deploy in /opt/colectivo/.deploys.log"
|
||||
REMOTE
|
||||
|
||||
echo "==> Deploy complete."
|
||||
|
||||
191
infra/scripts/rollback-erosi.sh
Executable file
191
infra/scripts/rollback-erosi.sh
Executable file
@@ -0,0 +1,191 @@
|
||||
#!/usr/bin/env bash
|
||||
# Roll back the ambrosio production stack to a previously-deployed commit.
|
||||
#
|
||||
# Usage:
|
||||
# infra/scripts/rollback-erosi.sh --list # print recent deploys
|
||||
# infra/scripts/rollback-erosi.sh --previous # roll back to N-1
|
||||
# infra/scripts/rollback-erosi.sh --to=<git-sha> # roll back to a specific deploy
|
||||
# infra/scripts/rollback-erosi.sh --code-only --to=<sha> # skip DB restore
|
||||
#
|
||||
# Env:
|
||||
# DEPLOY_HOST=ambrosio (override if needed)
|
||||
# DEPLOY_PATH=/opt/colectivo
|
||||
#
|
||||
# How it works:
|
||||
# The deploy script (deploy-erosi.sh) writes one line per successful
|
||||
# deploy to /opt/colectivo/.deploys.log:
|
||||
# <iso-ts> <git-sha> <backup-file>
|
||||
# Rollback reads that log, picks the row matching --previous or --to,
|
||||
# then:
|
||||
# 1. Verifies the SHA is reachable in the local git repo.
|
||||
# 2. Verifies the matching backup file still exists on ambrosio.
|
||||
# 3. Asks for explicit confirmation (destructive op).
|
||||
# 4. Checks out that SHA into a temp worktree on this machine.
|
||||
# 5. rsync (excluding .git, etc.) → ambrosio.
|
||||
# 6. Restores the DB from the matched backup.
|
||||
# 7. Rebuilds the app image at the rollback SHA and brings the stack
|
||||
# up. NOTE: --code-only skips step 6.
|
||||
# 8. Cleans up the worktree. Does NOT write a new line to
|
||||
# .deploys.log — rollback is intentionally not deploy-equivalent.
|
||||
#
|
||||
# What is NOT rolled back:
|
||||
# * Storage volume (/var/lib/storage) — user uploads. Treated as
|
||||
# append-only; backup taken pre-deploy but never restored on rollback.
|
||||
# * Realtime tenant state, Kong runtime config. They are stateless or
|
||||
# re-bootstrap from the DB.
|
||||
|
||||
set -euo pipefail
|
||||
|
||||
DEPLOY_HOST="${DEPLOY_HOST:-ambrosio}"
|
||||
DEPLOY_PATH="${DEPLOY_PATH:-/opt/colectivo}"
|
||||
REPO_ROOT="$(cd "$(dirname "$0")/../.." && pwd)"
|
||||
|
||||
MODE=""
|
||||
TARGET_SHA=""
|
||||
CODE_ONLY=0
|
||||
|
||||
while [ $# -gt 0 ]; do
|
||||
case "$1" in
|
||||
--list) MODE="list" ;;
|
||||
--previous) MODE="previous" ;;
|
||||
--to=*) MODE="to"; TARGET_SHA="${1#--to=}" ;;
|
||||
--code-only) CODE_ONLY=1 ;;
|
||||
-h|--help)
|
||||
grep '^#' "$0" | sed 's/^# \{0,1\}//' | head -40
|
||||
exit 0
|
||||
;;
|
||||
*) echo "Unknown arg: $1" >&2; exit 1 ;;
|
||||
esac
|
||||
shift
|
||||
done
|
||||
|
||||
[ -z "$MODE" ] && { echo "Specify --list, --previous, or --to=<sha>." >&2; exit 1; }
|
||||
|
||||
# ── 1. Read deploy log from ambrosio ──────────────────────────────────────
|
||||
log=$(ssh "$DEPLOY_HOST" "cat $DEPLOY_PATH/.deploys.log 2>/dev/null || true")
|
||||
if [ -z "$log" ]; then
|
||||
echo "No deploy history found at $DEPLOY_HOST:$DEPLOY_PATH/.deploys.log" >&2
|
||||
exit 1
|
||||
fi
|
||||
|
||||
if [ "$MODE" = "list" ]; then
|
||||
echo "Recent deploys ($DEPLOY_HOST):"
|
||||
echo "------------------------------------------------------------"
|
||||
printf '%s\n' "$log" | nl -ba
|
||||
exit 0
|
||||
fi
|
||||
|
||||
# ── 2. Pick the target row ────────────────────────────────────────────────
|
||||
if [ "$MODE" = "previous" ]; then
|
||||
# Last line = current deploy. Second-to-last = previous.
|
||||
target_row=$(printf '%s\n' "$log" | tail -n 2 | head -n 1)
|
||||
elif [ "$MODE" = "to" ]; then
|
||||
target_row=$(printf '%s\n' "$log" | grep -F " $TARGET_SHA " | tail -n 1 || true)
|
||||
fi
|
||||
|
||||
if [ -z "${target_row:-}" ]; then
|
||||
echo "No matching deploy found." >&2
|
||||
exit 1
|
||||
fi
|
||||
|
||||
target_ts=$(printf '%s' "$target_row" | cut -f1)
|
||||
target_sha=$(printf '%s' "$target_row" | cut -f2)
|
||||
target_backup=$(printf '%s' "$target_row" | cut -f3)
|
||||
|
||||
echo "==> Rollback target"
|
||||
echo " timestamp: $target_ts"
|
||||
echo " git sha: $target_sha"
|
||||
echo " backup: $target_backup"
|
||||
|
||||
# ── 3. Verify SHA + backup ───────────────────────────────────────────────
|
||||
if ! git -C "$REPO_ROOT" cat-file -e "${target_sha}^{commit}" 2>/dev/null; then
|
||||
echo "ERROR: commit $target_sha not present in local repo. Fetch first." >&2
|
||||
exit 1
|
||||
fi
|
||||
|
||||
if [ "$CODE_ONLY" -eq 0 ]; then
|
||||
if ! ssh "$DEPLOY_HOST" "test -f $DEPLOY_PATH/$target_backup"; then
|
||||
echo "ERROR: backup $target_backup missing on $DEPLOY_HOST." >&2
|
||||
echo " Use --code-only to skip the DB restore." >&2
|
||||
exit 1
|
||||
fi
|
||||
fi
|
||||
|
||||
# ── 4. Confirm ────────────────────────────────────────────────────────────
|
||||
echo ""
|
||||
echo "WARNING: This is DESTRUCTIVE."
|
||||
if [ "$CODE_ONLY" -eq 1 ]; then
|
||||
echo " Code will be rolled back to $target_sha; DB is NOT touched."
|
||||
else
|
||||
echo " Code AND database will be rolled back."
|
||||
echo " Current DB state will be overwritten by $target_backup."
|
||||
fi
|
||||
echo " 5s window to abort (Ctrl-C)..."
|
||||
sleep 5
|
||||
|
||||
# ── 5. Worktree at the target SHA ────────────────────────────────────────
|
||||
worktree_dir=$(mktemp -d -t colectivo-rollback-XXXXXX)
|
||||
trap 'git -C "$REPO_ROOT" worktree remove --force "$worktree_dir" 2>/dev/null || rm -rf "$worktree_dir"' EXIT
|
||||
|
||||
echo "--- Materialising worktree at $target_sha → $worktree_dir"
|
||||
git -C "$REPO_ROOT" worktree add --detach "$worktree_dir" "$target_sha" >/dev/null
|
||||
|
||||
# ── 6. rsync worktree → ambrosio ─────────────────────────────────────────
|
||||
echo "--- rsync code to $DEPLOY_HOST:$DEPLOY_PATH"
|
||||
rsync -az --delete \
|
||||
--exclude '.git' \
|
||||
--exclude 'node_modules' \
|
||||
--exclude '.svelte-kit' \
|
||||
--exclude 'apps/web/build' \
|
||||
--exclude 'apps/web/.vite' \
|
||||
--exclude '**/*.log' \
|
||||
--exclude '**/.env' \
|
||||
--exclude '**/.env.development' \
|
||||
--exclude '**/.env.local' \
|
||||
--exclude 'backups' \
|
||||
--exclude '.deploys.log' \
|
||||
"$worktree_dir/" "$DEPLOY_HOST:$DEPLOY_PATH/"
|
||||
|
||||
# ── 7. Restore DB (unless --code-only) ───────────────────────────────────
|
||||
if [ "$CODE_ONLY" -eq 0 ]; then
|
||||
echo "--- Stopping app + auth + rest before DB restore"
|
||||
ssh "$DEPLOY_HOST" bash -s << REMOTE
|
||||
set -euo pipefail
|
||||
cd $DEPLOY_PATH
|
||||
docker compose --env-file .env -f infra/docker-compose.erosi.yml stop app auth rest realtime storage
|
||||
REMOTE
|
||||
|
||||
echo "--- Restoring DB from $target_backup"
|
||||
ssh "$DEPLOY_HOST" bash -s << REMOTE
|
||||
set -euo pipefail
|
||||
cd $DEPLOY_PATH
|
||||
gunzip -c $target_backup \
|
||||
| docker compose --env-file .env -f infra/docker-compose.erosi.yml exec -T db \
|
||||
psql -U postgres -d postgres -v ON_ERROR_STOP=1
|
||||
REMOTE
|
||||
fi
|
||||
|
||||
# ── 8. Rebuild + bring the stack back up at the rollback SHA ────────────
|
||||
echo "--- Rebuild app at $target_sha + bring stack up"
|
||||
ssh "$DEPLOY_HOST" env GIT_SHA="$target_sha" bash -s << 'REMOTE'
|
||||
set -euo pipefail
|
||||
cd /opt/colectivo
|
||||
: "${GIT_SHA:=dev}"
|
||||
|
||||
GIT_SHA="$GIT_SHA" docker compose --env-file .env -f infra/docker-compose.erosi.yml \
|
||||
build --build-arg GIT_SHA="$GIT_SHA" app
|
||||
GIT_SHA="$GIT_SHA" docker compose --env-file .env -f infra/docker-compose.erosi.yml up -d
|
||||
REMOTE
|
||||
|
||||
echo ""
|
||||
echo "==> Rollback complete."
|
||||
echo " code: $target_sha"
|
||||
if [ "$CODE_ONLY" -eq 1 ]; then
|
||||
echo " db: UNCHANGED (--code-only)"
|
||||
else
|
||||
echo " db: restored from $target_backup"
|
||||
fi
|
||||
echo " App: https://erosi.limonia.net"
|
||||
echo ""
|
||||
echo "NOTE: .deploys.log is intentionally not updated by rollback."
|
||||
echo " The next 'just deploy' will be from your current HEAD."
|
||||
Reference in New Issue
Block a user