deploy(prod): pre-deploy DB backup + paired code/DB rollback skill
Every `just deploy` now takes a pg_dumpall to /opt/colectivo/backups/predeploy-<ts>-<sha>.sql.gz BEFORE rebuilding the app or applying migrations, aborts the deploy if the dump is < 1 KiB, and (on success) appends `<iso-ts> \t <sha> \t <backup-file>` to /opt/colectivo/.deploys.log on ambrosio. Keeps the newest 10 backups (prunes older predeploy-* files). New `infra/scripts/rollback-erosi.sh` reads .deploys.log and pairs a code rollback with a DB restore atomically: just rollback-list # show recent deploys just rollback # roll back to N-1 just rollback-to <sha> # roll back to a specific deploy just rollback-code # roll back code only, keep current DB Rollback safety: - 5-second abort window. - Verifies the target SHA exists locally + the backup is still on prod (warns if it's been pruned past the 10-deploy window). - Uses a temporary git worktree so the user's working tree isn't disturbed. - Stops app/auth/rest/realtime/storage before the gunzip|psql restore. - Rebuilds the app image at the rolled-back SHA with the same GIT_SHA build-arg path the deploy uses (so __APP_COMMIT__ in the bundle matches the running code). - Does NOT write a new .deploys.log entry — rollback is intentionally not a deploy event; the next `just deploy` is from current HEAD. - Storage volume (/var/lib/storage user uploads) is NOT rolled back. Justfile `deploy` recipe repointed from the stale `deploy.sh` (which referenced GHCR pull) to `deploy-erosi.sh` (the active prod path). New project-scoped skill: `.claude/skills/deploy/SKILL.md` documents the flow + safety boundaries for Claude-assisted invocations. `.gitignore` keeps `.claude/settings.local.json` ignored but tracks `.claude/skills/`. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
This commit is contained in:
@@ -19,6 +19,14 @@
|
||||
# it only manages the Docker stack.
|
||||
#
|
||||
# Subsequent runs: rsync + rebuild app + rolling restart.
|
||||
#
|
||||
# Pre-deploy safety (every run, before any code/migration changes):
|
||||
# * pg_dumpall to /opt/colectivo/backups/predeploy-<ts>-<sha>.sql.gz
|
||||
# * Keep newest 10 backups, prune older.
|
||||
# * On success, append a line to /opt/colectivo/.deploys.log:
|
||||
# <iso-timestamp> <git-sha> <backup-filename>
|
||||
# The rollback script (infra/scripts/rollback-erosi.sh) reads .deploys.log
|
||||
# to pair (sha, backup) when rolling back code + DB together.
|
||||
|
||||
set -euo pipefail
|
||||
|
||||
@@ -109,6 +117,38 @@ if grep -q '^PUBLIC_KEYCLOAK_URL=FILL_IN_' .env \
|
||||
fi
|
||||
REMOTE
|
||||
|
||||
# ── 3b. Pre-deploy DB backup ──────────────────────────────────────────────
|
||||
# Always taken BEFORE the new code/migrations land, so a paired rollback
|
||||
# can restore both. Stored on ambrosio at /opt/colectivo/backups/. Keep
|
||||
# newest 10; older ones get pruned.
|
||||
ssh "$DEPLOY_HOST" env GIT_SHA="$GIT_SHA" bash -s << 'REMOTE'
|
||||
set -euo pipefail
|
||||
cd /opt/colectivo
|
||||
: "${GIT_SHA:=dev}"
|
||||
|
||||
mkdir -p backups
|
||||
BACKUP_TS=$(date -u +"%Y%m%dT%H%M%SZ")
|
||||
BACKUP_FILE="backups/predeploy-${BACKUP_TS}-${GIT_SHA}.sql.gz"
|
||||
|
||||
echo "--- Pre-deploy backup → $BACKUP_FILE"
|
||||
docker compose --env-file .env -f infra/docker-compose.erosi.yml exec -T db \
|
||||
pg_dumpall -U postgres --clean --if-exists </dev/null \
|
||||
| gzip > "$BACKUP_FILE"
|
||||
size=$(stat -c%s "$BACKUP_FILE")
|
||||
if [ "$size" -lt 1024 ]; then
|
||||
echo "ERROR: backup is suspiciously small ($size bytes). Aborting deploy." >&2
|
||||
rm -f "$BACKUP_FILE"
|
||||
exit 1
|
||||
fi
|
||||
echo " ok ($(numfmt --to=iec --suffix=B "$size" 2>/dev/null || echo "$size bytes"))"
|
||||
|
||||
# Prune: keep newest 10 predeploy backups.
|
||||
ls -1t backups/predeploy-*.sql.gz 2>/dev/null | tail -n +11 | xargs -r rm -f
|
||||
|
||||
# Stash filename for step 4 to record once everything succeeds.
|
||||
echo "$BACKUP_FILE" > /tmp/.colectivo-last-backup
|
||||
REMOTE
|
||||
|
||||
# ── 4. Build + bring up the stack ─────────────────────────────────────────
|
||||
# We forward GIT_SHA over the ssh env so the remote `docker compose build`
|
||||
# can substitute it into the Dockerfile build-arg (compose reads it from
|
||||
@@ -165,6 +205,14 @@ for f in supabase/migrations/*.sql; do
|
||||
echo "FAIL"; exit 1
|
||||
fi
|
||||
done
|
||||
|
||||
# Record the successful deploy in .deploys.log (read by rollback-erosi.sh).
|
||||
backup_file=$(cat /tmp/.colectivo-last-backup 2>/dev/null || echo "")
|
||||
deploy_ts=$(date -u +"%Y-%m-%dT%H:%M:%SZ")
|
||||
printf '%s\t%s\t%s\n' "$deploy_ts" "$GIT_SHA" "$backup_file" \
|
||||
>> /opt/colectivo/.deploys.log
|
||||
rm -f /tmp/.colectivo-last-backup
|
||||
echo "--- Recorded deploy in /opt/colectivo/.deploys.log"
|
||||
REMOTE
|
||||
|
||||
echo "==> Deploy complete."
|
||||
|
||||
Reference in New Issue
Block a user