ulicraft-server-v1/plan/23-monitor.md

23 — Monitor (monitor.${BASE_DOMAIN})

Admin-only, read-only monitoring page: realtime latest.log stream with autoscroll + filter, plus a small dashboard of live server facts. One new Go service (docker/monitor/), one caddy vhost behind HTTP basic auth, no published port, no actions of any kind in the UI.

Scope, stated as constraints

These were settled by design interview; they are the constraints, not preferences.

1. Admin-only. Nothing here is player-facing. Player-facing status already has two homes: the landing's ServerCard (15-mc-status.md) and Uptime Kuma (10-uptime-kuma.md). Because the audience is one person, log lines are never sanitized — they carry IPs, chat and admin commands verbatim.
2. Read-only. No controls. No console, no start/stop, no command input, no kick/ban. The UI issues GETs and nothing else. This is a design rule, not a v1 limitation — see RCON for how it survives contact with TPS.
3. latest.log only. Not whole-stack container logs. This is what kills the docker-socket requirement outright (see Rejected).
4. Delay is acceptable. Explicitly accepted at design time, and it is what makes polling viable over SSE.
5. Unreachable except through the subdomain. No host port, no LAN access, no direct container access from outside mcnet.

The one fact that drives every decision

The monitor never needs to reach outside mcnet, and nothing outside mcnet needs to reach it except caddy. So it publishes no port at all — same as drasl, nmsr, filestash and mc-status. Constraint 5 then costs zero firewall work: there is no host-level listener to firewall. nmap against the host still shows only 25565, 24454/udp, 80 and 443.

Everything else follows: no DOCKER-USER iptables chain, no bind-address env var, no ufw rule that Docker would silently bypass anyway.

Architecture

browser ── https://monitor.${BASE_DOMAIN}
   └► host nginx (TLS, own LE cert)
        └► caddy  80-monitor.caddy  ── basic_auth (bcrypt) ──► monitor:8080
                                                                 │
   monitor container:                                            │
     mc_data:/data:ro       ← tail /data/logs/latest.log  ────────┤
     ./backups:/backups:ro  ← newest archive stat (phase 2)       │
     RCON → minecraft:25575 ← allowlisted queries only (phase 3)  │
     HTTP → mc-status:8080  ← status + roster reuse (phase 2)     │
     //go:embed ui/         ← single index.html, no build step ───┘

Piece	Where
Service source	docker/monitor/ (main.go, go.mod, Dockerfile, ui/index.html)
Image	ulicraft/monitor:local, two-stage → scratch (same shape as mc-status)
Compose service	monitor, networks: [mcnet], no ports:
Ingress	caddy/conf.d/80-monitor.caddy
TLS / vhost	nginx/ulicraft-caddy.conf.tmpl + LE_SUBDOMAINS
Per-host toggle	MONITOR_ENABLED (mirrors the FILES_BROWSE on/off pattern)

Log pipeline

Transport: polling, not SSE

GET /api/log?since=<seq> → {lines: [...], seq: N, dropped: M}, client polls every ~2s.

Why not SSE. The request chain is host-nginx → caddy → app. nginx buffers proxied responses by default, so an EventSource stalls until the buffer fills unless the vhost carries proxy_buffering off;. That vhost is the per-host untracked file on cochi (see cochi host-local divergence), so the failure would appear only on prod and not reproduce locally. Constraint 4 means polling costs nothing, and it deletes the whole bug class. WebSocket was rejected for the same reason plus an upgrade-header hop it doesn't need.

Cursor is a sequence number, NOT a byte offset

The server keeps an in-memory ring buffer (5000 lines) and numbers lines monotonically. since is that number.

• A byte offset stops meaning anything the moment the file rotates, and itzg rotates latest.log → YYYY-MM-DD-N.log.gz on every server restart — i.e. on every deploy. Offsets are wrong routinely, not rarely.
• dropped > 0 tells the client it fell behind the ring and must re-request the full buffer rather than silently skip lines. Without this the UI lies by omission after any burst (mod crash spam, world load).

Rotation handling

The tailer stats the path each tick and reopens on inode change or size shrink. Both happen on every deploy. A tailer that only follows an open fd goes permanently silent after the first restart while still returning HTTP 200 — the classic version of "the monitor says everything is fine."

Cold start

On boot, read the tail of /data/logs/latest.log to prime the ring. That yields 5000 lines of history immediately with no database, no persistence, no state volume. A monitor restart costs nothing that isn't still on disk. Session history (joins/leaves) is derived from the same buffer, not stored.

Dashboard tiles

Everything below is available with no new privileges. Uptime Kuma already owns up/down and uptime history — do not rebuild that here.

Tile	Source	Phase
Server online + MOTD + icon	mc-status:8080/v2/status/java/… over mcnet	2
Players online (names + NMSR heads)	same, players.list[]	2
Registered players	mc-status:8080/players (Drasl-backed)	2
Disk free on the world volume	statfs on /data	2
Backup freshness	newest file in ./backups (:ro mount)	2
Join/leave session history	parsed from the buffer already tailed	2
TPS / MSPT	RCON, see below	3

Deliberately excluded: service reachability probes (Kuma's job), world-size walk (expensive, low value, cache-or-nothing), container health (needs the docker socket).

/api/summary — one response, independently-degrading parts

Every tile has its own TTL cache and its own failure mode (collect.go). A collector that fails keeps serving its last good value and reports the reason in a sibling *_err field, so the page shows stale with a warning rather than a blank tile or a spinner that never resolves. Verified: with mc-status killed, server_err is populated while the disk, backup and log tiles keep working.

The five collectors are refreshed concurrently. Sequentially, one request during an outage would cost the sum of every timeout (~15s with mc-status hanging and RCON blackholed) — and an outage is precisely when someone is staring at the page. Measured with a dead dependency: 0.24s.

TTLs: status 15s, roster 60s, disk 60s, backup 60s, TPS 30s. The UI polls /api/summary every 10s and /api/log every 2s — tiles move far more slowly than the log does.

Running as root, on purpose

/data is drwxr-x--- owned by uid 1000 (itzg). A non-root container user cannot traverse it, so the tailer would read nothing — silently. The container therefore runs as root, and what actually bounds the damage is that every mount is :ro and no port is published. Do not "harden" this by adding a USER directive without re-checking those permissions first.

RCON — the one credential, and why it doesn't break constraint 2

TPS is the number you actually want when someone says "the server is lagging", and the only source without adding a 165th mod is an RCON query. So the monitor holds RCON_PASSWORD and dials minecraft:25575.

Constraint 2 survives because the UI never sends a command: the poller runs a compile-time allowlist (neoforge tps, list) on a 30s timer. There is no code path from an HTTP request to an RCON string. Hardcode the allowlist so a future refactor can't quietly turn this into a web console.

• The command is neoforge tps on NeoForge 1.21.1, not forge tps. Confirmed on the running server; forge tps answers "Unknown or incomplete command". An unknown-command reply is treated as "TPS unavailable" rather than being allowed to kill the poll loop.

• The output format is NOT the old Forge "Mean tick time / Mean TPS" wording this doc originally assumed. Actual, captured from prod:

  Overworld: 20.000 TPS (0.298 ms/tick)
  The Nether: 20.000 TPS (0.014 ms/tick)
  Overall: 20.000 TPS (0.357 ms/tick)
  

  One line per dimension plus an Overall: line, with §/ANSI colour codes mixed in that have to be stripped. main_test.go pins this against the real captured string — if a NeoForge update changes the wording, that test fails instead of the tile silently reading zero.

• The RCON client is implemented in-repo (rcon.go, ~120 lines of the Source RCON protocol) rather than pulled in as a dependency, specifically so the allowlist is a compile-time constant with no code path from HTTP input to a command string. main_test.go asserts op, stop, say, ban and near-misses like LIST and a trailing-space "neoforge tps " are all rejected.

• Alternative considered: the spark mod. Better data (real profiling), but a 165th jar into a pack already over its stated ceiling, plus its own web endpoint to secure. Rejected for now; still the upgrade path if RCON TPS proves too coarse.

Auth + ingress

Basic auth lives in caddy, not host nginx: caddy/conf.d/*.caddy is in git, the nginx vhosts are per-host untracked drift on cochi.

# Monitor — admin-only log stream + dashboard (monitor service). Read-only.
http://monitor.{$BASE_DOMAIN} {
	basic_auth {
		{$MONITOR_USER} {$MONITOR_PASSWORD_HASH}
	}
	reverse_proxy monitor:8080
}

• caddy basic_auth needs a bcrypt hash (caddy hash-password). Note this is the opposite of the filestash trap in 20-files.md, where bcrypt is the hash that silently fails and $6$ is required. The two services take different formats; don't copy a hash between them.
• One user, one password (MONITOR_USER / MONITOR_PASSWORD_HASH in .env).
• Uptime Kuma does not need an auth exemption. Kuma is on mcnet, so it probes http://monitor:8080/healthz directly and never traverses caddy. The whole vhost stays gated with no public path carve-out.

Rejected ingress alternatives

• Apex path /monitor/* (the mc-status precedent — no DNS, no cert). Serving a JS app from a subpath is exactly the failure class 20-files.md documents for filestash. Fixable with relative asset paths, but it's a permanent footgun tax to save one DNS record.
• NetBird / VPN-only, no public listener. Considered and dropped. Worth recording why it was attractive: zero auth surface, logs never touch the internet. And why it was dropped: NetBird gates reachability, not identity — any peer the ACL admits gets the full log stream with no login, so the security model silently degrades the moment a second device or person joins that network.
• SSH tunnel to 127.0.0.1:9090. Most secure option available. Dropped because it defeats checking the server from a phone.

Rejected: whole-stack logs via the docker socket

Following caddy/drasl/filestash/mc-backup logs is genuinely where a lot of the debugging value lives (deploy failures surface in caddy and drasl, not in latest.log). It was dropped because constraint 3 limits scope to latest.log, and that removes the need for the socket entirely.

Recorded so the tradeoff isn't relitigated blind:

Mounting /var/run/docker.sock into a container is equivalent to granting that container root on the host, and the :ro flag does not prevent this — it makes the socket file read-only while the Docker API behind it still accepts container-create calls. The mitigation, had we needed it, is tecnativa/docker-socket-proxy restricted to GET containers+logs.

As designed, the monitor's only mount is mc_data:/data:ro (plus ./backups:ro in phase 2), and its only secret is RCON_PASSWORD in phase 3.

UI

Single index.html, //go:embed-ed into the binary. Vanilla JS + CSS, no build step, no pnpm in the deploy path, image stays a two-stage scratch build. Copy the landing's CSS custom properties so it doesn't look alien.

Behaviour:

• Autoscroll sticks to bottom only while already at the bottom. Scroll up and it stops fighting you; a "jump to live" button returns.
• Filter is plain substring, case-insensitive, client-side over the loaded buffer. /re/ syntax opts into regex.
• Level chips (ERROR / WARN / INFO) as quick toggles — log4j lines are [HH:mm:ss] [thread/LEVEL], so this is a cheap parse.
• ERROR red, WARN amber, everything else default.

Rejected: an Astro page (reuses landing components but adds a second pnpm build to update-all.sh), and caddy-served static assets (UI and API then version-skew independently).

Deploy surface

New/changed files:

• docker/monitor/{main.go,go.mod,Dockerfile,ui/index.html}
• docker-compose.yml — monitor service (no ports:)
• caddy/conf.d/80-monitor.caddy
• nginx/ulicraft-caddy.conf.tmpl — one server block + the :80 redirect name
• .env.example — MONITOR_USER, MONITOR_PASSWORD_HASH, MONITOR_ENABLED, LE_SUBDOMAINS += monitor
• plan/00-overview.md, plan/19-routes.md — index + route table

Gotchas:

• update-all.sh rolling mode cannot see caddy conf changes. It has no pre-pull baseline for caddy/conf.d/*.caddy, so the first deploy needs --hard — a full down/up, i.e. Minecraft world downtime. Later monitor-only changes are rolling (up -d --build monitor).
• DNS + cert are the long pole and live outside this repo. monitor. needs an A record, an LE_SUBDOMAINS entry, and tooling/issue-letsencrypt.sh. Nothing works until that lands; do it first.
• Per-host. cochi and oxps diverge — gate with MONITOR_ENABLED rather than assuming both hosts want it.

Build phases

Ship the thin slice first. Every hard part of this project is plumbing — new subdomain, cert, nginx vhost, basic auth through two proxies, mc_data:ro, tail-across-rotation. None of it is in the tiles. Once a password-gated page shows raw log lines, everything after is a small additive commit with a rolling one-container restart.

Phase 1 — thin slice: logs only — DONE, live on cochi 2026-08-01

• DNS A record for monitor.${BASE_DOMAIN}; add to LE_SUBDOMAINS; issue cert. (DNS already resolved — the zone has a wildcard, so no record was needed.)
• docker/monitor/ — Go service: ring buffer, tailer (inode/shrink reopen), cold-start backfill, GET /api/log?since=, GET /healthz, embedded UI.
• ui/index.html — log pane, sticky autoscroll, filter, level chips.
• compose service (no ports:, mc_data:/data:ro).
• caddy/conf.d/80-monitor.caddy + MONITOR_USER/MONITOR_PASSWORD_HASH.
• nginx vhost + MONITOR_ENABLED gate; render + reload.
• Deploy with --hard (world downtime, 0 players online). Verified below.

Phase 2 — tiles — DONE

• ./backups:/backups:ro mount; backup-freshness + disk-free tiles.
• mc-status reuse: online/MOTD, players online w/ NMSR heads, registered count. (Icon skipped — the MOTD and version carry the same information in a row of small tiles, and the favicon is a base64 blob.)
• Join/leave session history parsed from the buffer.
• Fixed a phase-1 bug found while building this: the UI's level regex assumed itzg's console format ([12:34:56]), but the file uses [01Aug2026 22:55:34.358]. Every line was classified INFO, so the ERROR/WARN chips silently did nothing. Now matches on shape ([…] […/LEVEL]) instead of on the timestamp.

Phase 3 — TPS — DONE

• RCON client, compile-time allowlist, 30s poll.
• Confirmed neoforge tps is the right command, and captured its real output format (see RCON above — it is not what this doc first assumed).

Follow-ups

• Uptime Kuma HTTP monitor against http://monitor:8080/healthz (internal, unauthenticated by construction).
• Decide whether "search all history" (server-side grep over rotated .log.gz) is missed. Deferred from v1 on purpose: latest.log on a 164-mod server is mostly boot spam, and the real workflow is watching the last few hundred lines filtered by a player name or ERROR.

Verification

A 200 on / proves the server is up, not that the page works (see spa-curl-200-is-not-verification). After deploy, check all of:

# auth actually gates it (expect 401)
curl -s -o /dev/null -w '%{http_code}\n' https://monitor.${BASE_DOMAIN}/

# with creds, expect 200 + a non-empty buffer
curl -s -u "$MONITOR_USER:$MONITOR_PASSWORD" \
  "https://monitor.${BASE_DOMAIN}/api/log?since=0" | jq '.seq, (.lines|length)'

# the port is NOT reachable from outside the compose network
nmap -Pn -p 8080 <host-public-ip>        # expect closed/filtered
docker compose port monitor 8080          # expect an error: no published port

Then the one that actually matters, because it is the failure this design is most likely to hit: restart the Minecraft server and confirm the stream keeps moving. That is the rotation path, it fires on every deploy, and a broken tailer stays HTTP-200 while going silent.

Phase 1 results (cochi, 2026-08-01)

The deploy itself exercised the rotation path: monitor came up while the old latest.log still existed, Minecraft then created a new one, and the tailer reported rotations: 1 and kept streaming. Verified in prod:

Check	Result
GET / no creds / wrong creds / right creds	401 / 401 / 200
TLS	CN=monitor.ulicraft.net, Let's Encrypt
Buffer after boot	908 lines, tail.ok: true
Stream advancing	+109 lines in 12s, dropped: 0, reset: false
Rotation across the MC restart	rotations: 1, stream uninterrupted
Port 8080 from the internet	connection refused
docker compose port monitor 8080	:0 (nothing published)
Host listeners on :8080	none

Also exercised locally before deploy, against a synthetic log: backfill capped to buffer size, partial (unterminated) lines withheld until their newline lands, mv-style rotation, in-place truncation, delete-and-recreate, ring overflow (dropped: 15, reset: true), a cursor from the future, malformed since (400), unknown path (404).

Known cosmetic gap: a rotation done as delete-then-recreate (rather than mv) does not increment the rotations counter — the tailer takes the "file was gone, reopen" branch instead of the rotation branch. Streaming is unaffected; only the displayed count is.

Phase 2 + 3 results (cochi, 2026-08-01)

Deployed rolling, not --hard — no caddy conf changed, so only the monitor container was recreated and Minecraft was never restarted (zero downtime).

Live values on first request: server online, MOTD "Uli LAN party", 1.21.1 · 14 registered accounts · 20.000 TPS / 0.307 ms/tick across 6 dimensions · 88 GB free, 59% used · newest backup 0h old, 57 archives, 14 GB.

Verified against real data rather than assumptions:

Check	Result
neoforge tps over the in-repo RCON client	20.000 TPS, 6 dimensions parsed
RCON allowlist rejects op/stop/say/ban/LIST/"neoforge tps "	all rejected (unit test)
Wrong RCON password	clean authentication failed, other tiles unaffected
Session parser vs. real rotated-log join lines	Patxakita joined/left, timestamps intact
UI level regex vs. real log lines	WARN and ERROR now distinguished (was: all INFO)
NMSR avatar URL the heads use	HTTP 200, image/png
mc-status killed	server_err set; disk, backup and log stream unaffected
25 concurrent /api/summary under -race	0 data races
/api/summary with a dead dependency	0.24s (sequential would stack timeouts)