# 23 — Monitor (`monitor.${BASE_DOMAIN}`) > Admin-only, read-only monitoring page: **realtime `latest.log` stream** with > autoscroll + filter, plus a small dashboard of live server facts. One new Go > service (`docker/monitor/`), one caddy vhost behind HTTP basic auth, **no > published port**, **no actions of any kind in the UI**. ## Scope, stated as constraints These were settled by design interview; they are the constraints, not preferences. 1. **Admin-only.** Nothing here is player-facing. Player-facing status already has two homes: the landing's ServerCard (`15-mc-status.md`) and Uptime Kuma (`10-uptime-kuma.md`). Because the audience is one person, log lines are never sanitized — they carry IPs, chat and admin commands verbatim. 2. **Read-only. No controls.** No console, no start/stop, no command input, no kick/ban. The UI issues `GET`s and nothing else. This is a design rule, not a v1 limitation — see *RCON* for how it survives contact with TPS. 3. **`latest.log` only.** Not whole-stack container logs. This is what kills the docker-socket requirement outright (see *Rejected*). 4. **Delay is acceptable.** Explicitly accepted at design time, and it is what makes polling viable over SSE. 5. **Unreachable except through the subdomain.** No host port, no LAN access, no direct container access from outside `mcnet`. ## The one fact that drives every decision **The monitor never needs to reach outside `mcnet`, and nothing outside `mcnet` needs to reach it except caddy.** So it publishes no port at all — same as `drasl`, `nmsr`, `filestash` and `mc-status`. Constraint 5 then costs *zero* firewall work: there is no host-level listener to firewall. `nmap` against the host still shows only 25565, 24454/udp, 80 and 443. Everything else follows: no `DOCKER-USER` iptables chain, no bind-address env var, no ufw rule that Docker would silently bypass anyway. ## Architecture ``` browser ── https://monitor.${BASE_DOMAIN} └► host nginx (TLS, own LE cert) └► caddy 80-monitor.caddy ── basic_auth (bcrypt) ──► monitor:8080 │ monitor container: │ mc_data:/data:ro ← tail /data/logs/latest.log ────────┤ ./backups:/backups:ro ← newest archive stat (phase 2) │ RCON → minecraft:25575 ← allowlisted queries only (phase 3) │ HTTP → mc-status:8080 ← status + roster reuse (phase 2) │ //go:embed ui/ ← single index.html, no build step ───┘ ``` | Piece | Where | |---|---| | Service source | `docker/monitor/` (`main.go`, `go.mod`, `Dockerfile`, `ui/index.html`) | | Image | `ulicraft/monitor:local`, two-stage → scratch (same shape as mc-status) | | Compose service | `monitor`, `networks: [mcnet]`, **no `ports:`** | | Ingress | `caddy/conf.d/80-monitor.caddy` | | TLS / vhost | `nginx/ulicraft-caddy.conf.tmpl` + `LE_SUBDOMAINS` | | Per-host toggle | `MONITOR_ENABLED` (mirrors the `FILES_BROWSE` on/off pattern) | ## Log pipeline ### Transport: polling, not SSE `GET /api/log?since=` → `{lines: [...], seq: N, dropped: M}`, client polls every ~2s. **Why not SSE.** The request chain is host-nginx → caddy → app. nginx buffers proxied responses by default, so an `EventSource` stalls until the buffer fills unless the vhost carries `proxy_buffering off;`. That vhost is the per-host untracked file on cochi (see `cochi host-local divergence`), so the failure would appear **only on prod** and not reproduce locally. Constraint 4 means polling costs nothing, and it deletes the whole bug class. WebSocket was rejected for the same reason plus an upgrade-header hop it doesn't need. ### Cursor is a sequence number, NOT a byte offset The server keeps an in-memory ring buffer (5000 lines) and numbers lines monotonically. `since` is that number. - A **byte offset stops meaning anything the moment the file rotates**, and itzg rotates `latest.log` → `YYYY-MM-DD-N.log.gz` on *every server restart* — i.e. on every deploy. Offsets are wrong routinely, not rarely. - `dropped` > 0 tells the client it fell behind the ring and must re-request the full buffer rather than silently skip lines. Without this the UI lies by omission after any burst (mod crash spam, world load). ### Rotation handling The tailer stats the path each tick and reopens on **inode change or size shrink**. Both happen on every deploy. A tailer that only follows an open fd goes permanently silent after the first restart while still returning HTTP 200 — the classic version of "the monitor says everything is fine." ### Cold start On boot, read the tail of `/data/logs/latest.log` to prime the ring. That yields 5000 lines of history immediately with **no database, no persistence, no state volume**. A monitor restart costs nothing that isn't still on disk. Session history (joins/leaves) is derived from the same buffer, not stored. ## Dashboard tiles Everything below is available with **no new privileges**. Uptime Kuma already owns up/down and uptime *history* — do not rebuild that here. | Tile | Source | Phase | |---|---|---| | Server online + MOTD + icon | `mc-status:8080/v2/status/java/…` over mcnet | 2 | | Players online (names + NMSR heads) | same, `players.list[]` | 2 | | Registered players | `mc-status:8080/players` (Drasl-backed) | 2 | | Disk free on the world volume | `statfs` on `/data` | 2 | | Backup freshness | newest file in `./backups` (`:ro` mount) | 2 | | Join/leave session history | parsed from the buffer already tailed | 2 | | TPS / MSPT | RCON, see below | 3 | **Deliberately excluded:** service reachability probes (Kuma's job), world-size walk (expensive, low value, cache-or-nothing), container health (needs the docker socket). ### `/api/summary` — one response, independently-degrading parts Every tile has its own TTL cache and its own failure mode (`collect.go`). A collector that fails keeps serving its last good value and reports the reason in a sibling `*_err` field, so the page shows *stale with a warning* rather than a blank tile or a spinner that never resolves. Verified: with mc-status killed, `server_err` is populated while the disk, backup and log tiles keep working. The five collectors are refreshed **concurrently**. Sequentially, one request during an outage would cost the sum of every timeout (~15s with mc-status hanging and RCON blackholed) — and an outage is precisely when someone is staring at the page. Measured with a dead dependency: 0.24s. TTLs: status 15s, roster 60s, disk 60s, backup 60s, TPS 30s. The UI polls `/api/summary` every 10s and `/api/log` every 2s — tiles move far more slowly than the log does. ### Running as root, on purpose `/data` is `drwxr-x---` owned by uid 1000 (itzg). A non-root container user cannot traverse it, so the tailer would read nothing — silently. The container therefore runs as root, and what actually bounds the damage is that every mount is `:ro` and no port is published. Do not "harden" this by adding a `USER` directive without re-checking those permissions first. ## RCON — the one credential, and why it doesn't break constraint 2 TPS is the number you actually want when someone says "the server is lagging", and **the only source without adding a 165th mod is an RCON query**. So the monitor holds `RCON_PASSWORD` and dials `minecraft:25575`. Constraint 2 survives because the UI never sends a command: the poller runs a **compile-time allowlist** (`neoforge tps`, `list`) on a 30s timer. There is no code path from an HTTP request to an RCON string. Hardcode the allowlist so a future refactor can't quietly turn this into a web console. - **The command is `neoforge tps` on NeoForge 1.21.1**, not `forge tps`. Confirmed on the running server; `forge tps` answers *"Unknown or incomplete command"*. An unknown-command reply is treated as "TPS unavailable" rather than being allowed to kill the poll loop. - **The output format is NOT the old Forge "Mean tick time / Mean TPS" wording** this doc originally assumed. Actual, captured from prod: ``` Overworld: 20.000 TPS (0.298 ms/tick) The Nether: 20.000 TPS (0.014 ms/tick) Overall: 20.000 TPS (0.357 ms/tick) ``` One line per dimension plus an `Overall:` line, with `§`/ANSI colour codes mixed in that have to be stripped. `main_test.go` pins this against the real captured string — if a NeoForge update changes the wording, that test fails instead of the tile silently reading zero. - The RCON client is implemented in-repo (`rcon.go`, ~120 lines of the Source RCON protocol) rather than pulled in as a dependency, specifically so the allowlist is a compile-time constant with no code path from HTTP input to a command string. `main_test.go` asserts `op`, `stop`, `say`, `ban` and near-misses like `LIST` and a trailing-space `"neoforge tps "` are all rejected. - Alternative considered: the `spark` mod. Better data (real profiling), but a 165th jar into a pack already over its stated ceiling, plus its own web endpoint to secure. Rejected for now; still the upgrade path if RCON TPS proves too coarse. ## Auth + ingress Basic auth lives in **caddy**, not host nginx: `caddy/conf.d/*.caddy` is in git, the nginx vhosts are per-host untracked drift on cochi. ```caddy # Monitor — admin-only log stream + dashboard (monitor service). Read-only. http://monitor.{$BASE_DOMAIN} { basic_auth { {$MONITOR_USER} {$MONITOR_PASSWORD_HASH} } reverse_proxy monitor:8080 } ``` - **caddy `basic_auth` needs a bcrypt hash** (`caddy hash-password`). Note this is the *opposite* of the filestash trap in `20-files.md`, where bcrypt is the hash that silently fails and `$6$` is required. The two services take different formats; don't copy a hash between them. - One user, one password (`MONITOR_USER` / `MONITOR_PASSWORD_HASH` in `.env`). - **Uptime Kuma does not need an auth exemption.** Kuma is on `mcnet`, so it probes `http://monitor:8080/healthz` directly and never traverses caddy. The whole vhost stays gated with no public path carve-out. ### Rejected ingress alternatives - **Apex path `/monitor/*`** (the mc-status precedent — no DNS, no cert). Serving a JS app from a subpath is exactly the failure class `20-files.md` documents for filestash. Fixable with relative asset paths, but it's a permanent footgun tax to save one DNS record. - **NetBird / VPN-only, no public listener.** Considered and dropped. Worth recording *why it was attractive*: zero auth surface, logs never touch the internet. And why it was dropped: NetBird gates *reachability*, not *identity* — any peer the ACL admits gets the full log stream with no login, so the security model silently degrades the moment a second device or person joins that network. - **SSH tunnel to `127.0.0.1:9090`.** Most secure option available. Dropped because it defeats checking the server from a phone. ## Rejected: whole-stack logs via the docker socket Following caddy/drasl/filestash/mc-backup logs is genuinely where a lot of the debugging value lives (deploy failures surface in caddy and drasl, not in `latest.log`). It was dropped because constraint 3 limits scope to `latest.log`, and that removes the need for the socket entirely. Recorded so the tradeoff isn't relitigated blind: > Mounting `/var/run/docker.sock` into a container is equivalent to granting > that container root on the host, and the `:ro` flag does **not** prevent this > — it makes the socket *file* read-only while the Docker API behind it still > accepts container-create calls. The mitigation, had we needed it, is > `tecnativa/docker-socket-proxy` restricted to GET containers+logs. As designed, the monitor's only mount is `mc_data:/data:ro` (plus `./backups:ro` in phase 2), and its only secret is `RCON_PASSWORD` in phase 3. ## UI Single `index.html`, `//go:embed`-ed into the binary. Vanilla JS + CSS, **no build step, no pnpm in the deploy path**, image stays a two-stage scratch build. Copy the landing's CSS custom properties so it doesn't look alien. Behaviour: - **Autoscroll sticks to bottom only while already at the bottom.** Scroll up and it stops fighting you; a "jump to live" button returns. - **Filter** is plain substring, case-insensitive, client-side over the loaded buffer. `/re/` syntax opts into regex. - **Level chips** (ERROR / WARN / INFO) as quick toggles — log4j lines are `[HH:mm:ss] [thread/LEVEL]`, so this is a cheap parse. - ERROR red, WARN amber, everything else default. Rejected: an Astro page (reuses landing components but adds a second pnpm build to `update-all.sh`), and caddy-served static assets (UI and API then version-skew independently). ## Deploy surface New/changed files: - `docker/monitor/{main.go,go.mod,Dockerfile,ui/index.html}` - `docker-compose.yml` — `monitor` service (no `ports:`) - `caddy/conf.d/80-monitor.caddy` - `nginx/ulicraft-caddy.conf.tmpl` — one `server` block + the `:80` redirect name - `.env.example` — `MONITOR_USER`, `MONITOR_PASSWORD_HASH`, `MONITOR_ENABLED`, `LE_SUBDOMAINS += monitor` - `plan/00-overview.md`, `plan/19-routes.md` — index + route table Gotchas: - **`update-all.sh` rolling mode cannot see caddy conf changes.** It has no pre-pull baseline for `caddy/conf.d/*.caddy`, so the first deploy needs `--hard` — a full `down`/`up`, i.e. **Minecraft world downtime**. Later monitor-only changes are rolling (`up -d --build monitor`). - **DNS + cert are the long pole and live outside this repo.** `monitor.` needs an A record, an `LE_SUBDOMAINS` entry, and `tooling/issue-letsencrypt.sh`. Nothing works until that lands; do it first. - **Per-host.** cochi and oxps diverge — gate with `MONITOR_ENABLED` rather than assuming both hosts want it. ## Build phases Ship the thin slice first. Every hard part of this project is plumbing — new subdomain, cert, nginx vhost, basic auth through two proxies, `mc_data:ro`, tail-across-rotation. None of it is in the tiles. Once a password-gated page shows raw log lines, everything after is a small additive commit with a rolling one-container restart. ### Phase 1 — thin slice: logs only — **DONE, live on cochi 2026-08-01** - [x] DNS A record for `monitor.${BASE_DOMAIN}`; add to `LE_SUBDOMAINS`; issue cert. (DNS already resolved — the zone has a wildcard, so no record was needed.) - [x] `docker/monitor/` — Go service: ring buffer, tailer (inode/shrink reopen), cold-start backfill, `GET /api/log?since=`, `GET /healthz`, embedded UI. - [x] `ui/index.html` — log pane, sticky autoscroll, filter, level chips. - [x] compose service (no `ports:`, `mc_data:/data:ro`). - [x] `caddy/conf.d/80-monitor.caddy` + `MONITOR_USER`/`MONITOR_PASSWORD_HASH`. - [x] nginx vhost + `MONITOR_ENABLED` gate; render + reload. - [x] Deploy with `--hard` (world downtime, 0 players online). Verified below. ### Phase 2 — tiles — **DONE** - [x] `./backups:/backups:ro` mount; backup-freshness + disk-free tiles. - [x] mc-status reuse: online/MOTD, players online w/ NMSR heads, registered count. (Icon skipped — the MOTD and version carry the same information in a row of small tiles, and the favicon is a base64 blob.) - [x] Join/leave session history parsed from the buffer. - [x] **Fixed a phase-1 bug found while building this:** the UI's level regex assumed itzg's console format (`[12:34:56]`), but the *file* uses `[01Aug2026 22:55:34.358]`. Every line was classified INFO, so the ERROR/WARN chips silently did nothing. Now matches on shape (`[…] […/LEVEL]`) instead of on the timestamp. ### Phase 3 — TPS — **DONE** - [x] RCON client, compile-time allowlist, 30s poll. - [x] Confirmed `neoforge tps` is the right command, and captured its real output format (see *RCON* above — it is not what this doc first assumed). ### Follow-ups - [ ] Uptime Kuma HTTP monitor against `http://monitor:8080/healthz` (internal, unauthenticated by construction). - [ ] Decide whether "search all history" (server-side grep over rotated `.log.gz`) is missed. Deferred from v1 on purpose: `latest.log` on a 164-mod server is mostly boot spam, and the real workflow is watching the last few hundred lines filtered by a player name or `ERROR`. ## Verification A 200 on `/` proves the server is up, not that the page works (see `spa-curl-200-is-not-verification`). After deploy, check all of: ```bash # auth actually gates it (expect 401) curl -s -o /dev/null -w '%{http_code}\n' https://monitor.${BASE_DOMAIN}/ # with creds, expect 200 + a non-empty buffer curl -s -u "$MONITOR_USER:$MONITOR_PASSWORD" \ "https://monitor.${BASE_DOMAIN}/api/log?since=0" | jq '.seq, (.lines|length)' # the port is NOT reachable from outside the compose network nmap -Pn -p 8080 # expect closed/filtered docker compose port monitor 8080 # expect an error: no published port ``` Then the one that actually matters, because it is the failure this design is most likely to hit: **restart the Minecraft server and confirm the stream keeps moving.** That is the rotation path, it fires on every deploy, and a broken tailer stays HTTP-200 while going silent. ### Phase 1 results (cochi, 2026-08-01) The deploy itself exercised the rotation path: monitor came up while the old `latest.log` still existed, Minecraft then created a new one, and the tailer reported `rotations: 1` and kept streaming. Verified in prod: | Check | Result | |---|---| | `GET /` no creds / wrong creds / right creds | 401 / 401 / 200 | | TLS | `CN=monitor.ulicraft.net`, Let's Encrypt | | Buffer after boot | 908 lines, `tail.ok: true` | | Stream advancing | +109 lines in 12s, `dropped: 0`, `reset: false` | | Rotation across the MC restart | `rotations: 1`, stream uninterrupted | | Port 8080 from the internet | connection refused | | `docker compose port monitor 8080` | `:0` (nothing published) | | Host listeners on :8080 | none | Also exercised locally before deploy, against a synthetic log: backfill capped to buffer size, partial (unterminated) lines withheld until their newline lands, `mv`-style rotation, in-place truncation, delete-and-recreate, ring overflow (`dropped: 15, reset: true`), a cursor from the future, malformed `since` (400), unknown path (404). **Known cosmetic gap:** a rotation done as delete-then-recreate (rather than `mv`) does not increment the `rotations` counter — the tailer takes the "file was gone, reopen" branch instead of the rotation branch. Streaming is unaffected; only the displayed count is. ### Phase 2 + 3 results (cochi, 2026-08-01) Deployed **rolling**, not `--hard` — no caddy conf changed, so only the monitor container was recreated and Minecraft was never restarted (zero downtime). Live values on first request: server online, MOTD "Uli LAN party", 1.21.1 · **14** registered accounts · **20.000 TPS / 0.307 ms/tick across 6 dimensions** · 88 GB free, 59% used · newest backup 0h old, 57 archives, 14 GB. Verified against *real* data rather than assumptions: | Check | Result | |---|---| | `neoforge tps` over the in-repo RCON client | 20.000 TPS, 6 dimensions parsed | | RCON allowlist rejects `op`/`stop`/`say`/`ban`/`LIST`/`"neoforge tps "` | all rejected (unit test) | | Wrong RCON password | clean `authentication failed`, other tiles unaffected | | Session parser vs. real rotated-log join lines | `Patxakita joined/left`, timestamps intact | | UI level regex vs. real log lines | WARN and ERROR now distinguished (was: all INFO) | | NMSR avatar URL the heads use | HTTP 200, `image/png` | | mc-status killed | `server_err` set; disk, backup and log stream unaffected | | 25 concurrent `/api/summary` under `-race` | 0 data races | | `/api/summary` with a dead dependency | 0.24s (sequential would stack timeouts) |