ulicraft-server-v1/plan/23-monitor.md

# 23 — Monitor (`monitor.${BASE_DOMAIN}`)

> Admin-only, read-only monitoring page: **realtime `latest.log` stream** with
> autoscroll + filter, plus a small dashboard of live server facts. One new Go
> service (`docker/monitor/`), one caddy vhost behind HTTP basic auth, **no
> published port**, **no actions of any kind in the UI**.

## Scope, stated as constraints

These were settled by design interview; they are the constraints, not preferences.

1. **Admin-only.** Nothing here is player-facing. Player-facing status already
   has two homes: the landing's ServerCard (`15-mc-status.md`) and Uptime Kuma
   (`10-uptime-kuma.md`). Because the audience is one person, log lines are
   never sanitized — they carry IPs, chat and admin commands verbatim.
2. **Read-only. No controls.** No console, no start/stop, no command input, no
   kick/ban. The UI issues `GET`s and nothing else. This is a design rule, not a
   v1 limitation — see *RCON* for how it survives contact with TPS.
3. **`latest.log` only.** Not whole-stack container logs. This is what kills the
   docker-socket requirement outright (see *Rejected*).
4. **Delay is acceptable.** Explicitly accepted at design time, and it is what
   makes polling viable over SSE.
5. **Unreachable except through the subdomain.** No host port, no LAN access,
   no direct container access from outside `mcnet`.

## The one fact that drives every decision

**The monitor never needs to reach outside `mcnet`, and nothing outside `mcnet`
needs to reach it except caddy.** So it publishes no port at all — same as
`drasl`, `nmsr`, `filestash` and `mc-status`. Constraint 5 then costs *zero*
firewall work: there is no host-level listener to firewall. `nmap` against the
host still shows only 25565, 24454/udp, 80 and 443.

Everything else follows: no `DOCKER-USER` iptables chain, no bind-address env
var, no ufw rule that Docker would silently bypass anyway.

## Architecture

```
browser ── https://monitor.${BASE_DOMAIN}
   └► host nginx (TLS, own LE cert)
        └► caddy  80-monitor.caddy  ── basic_auth (bcrypt) ──► monitor:8080
                                                                 │
   monitor container:                                            │
     mc_data:/data:ro       ← tail /data/logs/latest.log  ────────┤
     ./backups:/backups:ro  ← newest archive stat (phase 2)       │
     RCON → minecraft:25575 ← allowlisted queries only (phase 3)  │
     HTTP → mc-status:8080  ← status + roster reuse (phase 2)     │
     //go:embed ui/         ← single index.html, no build step ───┘
```

| Piece | Where |
|---|---|
| Service source | `docker/monitor/` (`main.go`, `go.mod`, `Dockerfile`, `ui/index.html`) |
| Image | `ulicraft/monitor:local`, two-stage → scratch (same shape as mc-status) |
| Compose service | `monitor`, `networks: [mcnet]`, **no `ports:`** |
| Ingress | `caddy/conf.d/80-monitor.caddy` |
| TLS / vhost | `nginx/ulicraft-caddy.conf.tmpl` + `LE_SUBDOMAINS` |
| Per-host toggle | `MONITOR_ENABLED` (mirrors the `FILES_BROWSE` on/off pattern) |

## Log pipeline

### Transport: polling, not SSE

`GET /api/log?since=<seq>` → `{lines: [...], seq: N, dropped: M}`, client polls
every ~2s.

**Why not SSE.** The request chain is host-nginx → caddy → app. nginx buffers
proxied responses by default, so an `EventSource` stalls until the buffer fills
unless the vhost carries `proxy_buffering off;`. That vhost is the per-host
untracked file on cochi (see `cochi host-local divergence`), so the failure
would appear **only on prod** and not reproduce locally. Constraint 4 means
polling costs nothing, and it deletes the whole bug class. WebSocket was
rejected for the same reason plus an upgrade-header hop it doesn't need.

### Cursor is a sequence number, NOT a byte offset

The server keeps an in-memory ring buffer (5000 lines) and numbers lines
monotonically. `since` is that number.

- A **byte offset stops meaning anything the moment the file rotates**, and
  itzg rotates `latest.log` → `YYYY-MM-DD-N.log.gz` on *every server restart* —
  i.e. on every deploy. Offsets are wrong routinely, not rarely.
- `dropped` > 0 tells the client it fell behind the ring and must re-request the
  full buffer rather than silently skip lines. Without this the UI lies by
  omission after any burst (mod crash spam, world load).

### Rotation handling

The tailer stats the path each tick and reopens on **inode change or size
shrink**. Both happen on every deploy. A tailer that only follows an open fd
goes permanently silent after the first restart while still returning HTTP 200 —
the classic version of "the monitor says everything is fine."

### Cold start

On boot, read the tail of `/data/logs/latest.log` to prime the ring. That yields
5000 lines of history immediately with **no database, no persistence, no state
volume**. A monitor restart costs nothing that isn't still on disk. Session
history (joins/leaves) is derived from the same buffer, not stored.

## Dashboard tiles

Everything below is available with **no new privileges**. Uptime Kuma already
owns up/down and uptime *history* — do not rebuild that here.

| Tile | Source | Phase |
|---|---|---|
| Server online + MOTD + icon | `mc-status:8080/v2/status/java/…` over mcnet | 2 |
| Players online (names + NMSR heads) | same, `players.list[]` | 2 |
| Registered players | `mc-status:8080/players` (Drasl-backed) | 2 |
| Disk free on the world volume | `statfs` on `/data` | 2 |
| Backup freshness | newest file in `./backups` (`:ro` mount) | 2 |
| Join/leave session history | parsed from the buffer already tailed | 2 |
| TPS / MSPT | RCON, see below | 3 |

**Deliberately excluded:** service reachability probes (Kuma's job), world-size
walk (expensive, low value, cache-or-nothing), container health (needs the
docker socket).

### `/api/summary` — one response, independently-degrading parts

Every tile has its own TTL cache and its own failure mode (`collect.go`). A
collector that fails keeps serving its last good value and reports the reason in
a sibling `*_err` field, so the page shows *stale with a warning* rather than a
blank tile or a spinner that never resolves. Verified: with mc-status killed,
`server_err` is populated while the disk, backup and log tiles keep working.

The five collectors are refreshed **concurrently**. Sequentially, one request
during an outage would cost the sum of every timeout (~15s with mc-status
hanging and RCON blackholed) — and an outage is precisely when someone is
staring at the page. Measured with a dead dependency: 0.24s.

TTLs: status 15s, roster 60s, disk 60s, backup 60s, TPS 30s. The UI polls
`/api/summary` every 10s and `/api/log` every 2s — tiles move far more slowly
than the log does.

## Join/leave browser notifications (opt-in)

A bell toggle in the header raises a desktop notification when a player joins or
leaves. It rides the **same `sessions[]`** the Recent joins tile renders — no
second data source, no re-parsing of the log in JS. Entirely client-side:
no server change, no new egress, no secret. The container still reaches nothing
outside `mcnet`.

Deliberately **not** a push channel. Chat push (Telegram/Discord/ntfy) would
reach a phone with nothing open, but it needs a CA bundle in the `scratch` image
(this container currently makes no TLS calls at all), a token in `.env`, and
outbound internet egress it does not have today. Browser notifications cost none
of that, at the price of only firing while the page is open — a background tab
is fine, a closed tab or sleeping laptop is not.

### Newness is a seen-set keyed on the log timestamp

The whole difficulty is *not* notifying for history, and history arrives two
ways: the events ring is already populated when the page loads, and **the
cold-start backfill re-reads 5000 log lines on every deploy**, repopulating the
ring with the same events.

So the client keeps a `time|player|action` seen-set in `localStorage` (capped at
the 500 newest keys). That key is stable across both a page reload and a
container restart. A server-side sequence number would **not** be — it resets to
zero on restart, so every historical event would look new and every deploy would
fire a notification burst.

### The baseline flag must flip unconditionally

The first response after a page load is history by definition, so it seeds the
seen-set and notifies nothing. The subtle part, caught by the tests rather than
by reading the code: that flag has to flip **whether or not any fresh events were
found**. A returning user already has every key in `localStorage`, so their first
response yields nothing fresh — flipping the flag lazily would burn the baseline
on their first *real* event instead and silently swallow it.

### Other behaviour

- **Opt-in, off by default.** Permission is requested from the bell click (a
  user gesture), never on page load.
- **A denied permission shows a distinct blocked state**, because "I enabled it
  and nothing happened" is otherwise indistinguishable from a bug.
- **Bursts coalesce**: more than 3 events in one poll (everyone rejoining after a
  restart) becomes one "N joined, M left".
- **While enabled, `/api/summary` polls every 5s instead of 10s** — with this
  feature on, the poll interval *is* the alert latency.
- Events are marked seen even while the toggle is off, so enabling it does not
  replay a backlog.
- Each notification carries a `tag` derived from its event key, so a duplicate
  cannot stack.

### Running as root, on purpose

`/data` is `drwxr-x---` owned by uid 1000 (itzg). A non-root container user
cannot traverse it, so the tailer would read nothing — silently. The container
therefore runs as root, and what actually bounds the damage is that every mount
is `:ro` and no port is published. Do not "harden" this by adding a `USER`
directive without re-checking those permissions first.

## RCON — the one credential, and why it doesn't break constraint 2

TPS is the number you actually want when someone says "the server is lagging",
and **the only source without adding a 165th mod is an RCON query**. So the
monitor holds `RCON_PASSWORD` and dials `minecraft:25575`.

Constraint 2 survives because the UI never sends a command: the poller runs a
**compile-time allowlist** (`neoforge tps`, `list`) on a 30s timer. There is no
code path from an HTTP request to an RCON string. Hardcode the allowlist so a
future refactor can't quietly turn this into a web console.

- **The command is `neoforge tps` on NeoForge 1.21.1**, not `forge tps`.
  Confirmed on the running server; `forge tps` answers *"Unknown or incomplete
  command"*. An unknown-command reply is treated as "TPS unavailable" rather
  than being allowed to kill the poll loop.
- **The output format is NOT the old Forge "Mean tick time / Mean TPS" wording**
  this doc originally assumed. Actual, captured from prod:

  ```
  Overworld: 20.000 TPS (0.298 ms/tick)
  The Nether: 20.000 TPS (0.014 ms/tick)
  Overall: 20.000 TPS (0.357 ms/tick)
  ```

  One line per dimension plus an `Overall:` line, with `§`/ANSI colour codes
  mixed in that have to be stripped. `main_test.go` pins this against the real
  captured string — if a NeoForge update changes the wording, that test fails
  instead of the tile silently reading zero.
- The RCON client is implemented in-repo (`rcon.go`, ~120 lines of the Source
  RCON protocol) rather than pulled in as a dependency, specifically so the
  allowlist is a compile-time constant with no code path from HTTP input to a
  command string. `main_test.go` asserts `op`, `stop`, `say`, `ban` and
  near-misses like `LIST` and a trailing-space `"neoforge tps "` are all
  rejected.
- Alternative considered: the `spark` mod. Better data (real profiling), but a
  165th jar into a pack already over its stated ceiling, plus its own web
  endpoint to secure. Rejected for now; still the upgrade path if RCON TPS
  proves too coarse.

## Auth + ingress

Basic auth lives in **caddy**, not host nginx: `caddy/conf.d/*.caddy` is in git,
the nginx vhosts are per-host untracked drift on cochi.

```caddy
# Monitor — admin-only log stream + dashboard (monitor service). Read-only.
http://monitor.{$BASE_DOMAIN} {
	basic_auth {
		{$MONITOR_USER} {$MONITOR_PASSWORD_HASH}
	}
	reverse_proxy monitor:8080
}
```

- **caddy `basic_auth` needs a bcrypt hash** (`caddy hash-password`). Note this
  is the *opposite* of the filestash trap in `20-files.md`, where bcrypt is the
  hash that silently fails and `$6$` is required. The two services take
  different formats; don't copy a hash between them.
- One user, one password (`MONITOR_USER` / `MONITOR_PASSWORD_HASH` in `.env`).
- **Uptime Kuma does not need an auth exemption.** Kuma is on `mcnet`, so it
  probes `http://monitor:8080/healthz` directly and never traverses caddy. The
  whole vhost stays gated with no public path carve-out.

### Rejected ingress alternatives

- **Apex path `/monitor/*`** (the mc-status precedent — no DNS, no cert). Serving
  a JS app from a subpath is exactly the failure class `20-files.md` documents
  for filestash. Fixable with relative asset paths, but it's a permanent
  footgun tax to save one DNS record.
- **NetBird / VPN-only, no public listener.** Considered and dropped. Worth
  recording *why it was attractive*: zero auth surface, logs never touch the
  internet. And why it was dropped: NetBird gates *reachability*, not
  *identity* — any peer the ACL admits gets the full log stream with no login,
  so the security model silently degrades the moment a second device or person
  joins that network.
- **SSH tunnel to `127.0.0.1:9090`.** Most secure option available. Dropped
  because it defeats checking the server from a phone.

## Rejected: whole-stack logs via the docker socket

Following caddy/drasl/filestash/mc-backup logs is genuinely where a lot of the
debugging value lives (deploy failures surface in caddy and drasl, not in
`latest.log`). It was dropped because constraint 3 limits scope to `latest.log`,
and that removes the need for the socket entirely.

Recorded so the tradeoff isn't relitigated blind:

> Mounting `/var/run/docker.sock` into a container is equivalent to granting
> that container root on the host, and the `:ro` flag does **not** prevent this
> — it makes the socket *file* read-only while the Docker API behind it still
> accepts container-create calls. The mitigation, had we needed it, is
> `tecnativa/docker-socket-proxy` restricted to GET containers+logs.

As designed, the monitor's only mount is `mc_data:/data:ro` (plus `./backups:ro`
in phase 2), and its only secret is `RCON_PASSWORD` in phase 3.

## UI

Single `index.html`, `//go:embed`-ed into the binary. Vanilla JS + CSS, **no
build step, no pnpm in the deploy path**, image stays a two-stage scratch build.
Copy the landing's CSS custom properties so it doesn't look alien.

Behaviour:

- **Autoscroll sticks to bottom only while already at the bottom.** Scroll up
  and it stops fighting you; a "jump to live" button returns.
- **Filter** is plain substring, case-insensitive, client-side over the loaded
  buffer. `/re/` syntax opts into regex.
- **Level chips** (ERROR / WARN / INFO) as quick toggles — log4j lines are
  `[HH:mm:ss] [thread/LEVEL]`, so this is a cheap parse.
- ERROR red, WARN amber, everything else default.

Rejected: an Astro page (reuses landing components but adds a second pnpm build
to `update-all.sh`), and caddy-served static assets (UI and API then version-skew
independently).

## Deploy surface

New/changed files:

- `docker/monitor/{main.go,go.mod,Dockerfile,ui/index.html}`
- `docker-compose.yml` — `monitor` service (no `ports:`)
- `caddy/conf.d/80-monitor.caddy`
- `nginx/ulicraft-caddy.conf.tmpl` — one `server` block + the `:80` redirect name
- `.env.example` — `MONITOR_USER`, `MONITOR_PASSWORD_HASH`, `MONITOR_ENABLED`,
  `LE_SUBDOMAINS += monitor`
- `plan/00-overview.md`, `plan/19-routes.md` — index + route table

Gotchas:

- **`update-all.sh` rolling mode cannot see caddy conf changes.** It has no
  pre-pull baseline for `caddy/conf.d/*.caddy`, so the first deploy needs
  `--hard` — a full `down`/`up`, i.e. **Minecraft world downtime**. Later
  monitor-only changes are rolling (`up -d --build monitor`).
- **DNS + cert are the long pole and live outside this repo.** `monitor.` needs
  an A record, an `LE_SUBDOMAINS` entry, and `tooling/issue-letsencrypt.sh`.
  Nothing works until that lands; do it first.
- **Per-host.** cochi and oxps diverge — gate with `MONITOR_ENABLED` rather than
  assuming both hosts want it.

## Build phases

Ship the thin slice first. Every hard part of this project is plumbing — new
subdomain, cert, nginx vhost, basic auth through two proxies, `mc_data:ro`,
tail-across-rotation. None of it is in the tiles. Once a password-gated page
shows raw log lines, everything after is a small additive commit with a rolling
one-container restart.

### Phase 1 — thin slice: logs only — **DONE, live on cochi 2026-08-01**

- [x] DNS A record for `monitor.${BASE_DOMAIN}`; add to `LE_SUBDOMAINS`; issue cert.
      (DNS already resolved — the zone has a wildcard, so no record was needed.)
- [x] `docker/monitor/` — Go service: ring buffer, tailer (inode/shrink reopen),
      cold-start backfill, `GET /api/log?since=`, `GET /healthz`, embedded UI.
- [x] `ui/index.html` — log pane, sticky autoscroll, filter, level chips.
- [x] compose service (no `ports:`, `mc_data:/data:ro`).
- [x] `caddy/conf.d/80-monitor.caddy` + `MONITOR_USER`/`MONITOR_PASSWORD_HASH`.
- [x] nginx vhost + `MONITOR_ENABLED` gate; render + reload.
- [x] Deploy with `--hard` (world downtime, 0 players online). Verified below.

### Phase 2 — tiles — **DONE**

- [x] `./backups:/backups:ro` mount; backup-freshness + disk-free tiles.
- [x] mc-status reuse: online/MOTD, players online w/ NMSR heads,
      registered count. (Icon skipped — the MOTD and version carry the same
      information in a row of small tiles, and the favicon is a base64 blob.)
- [x] Join/leave session history parsed from the buffer.
- [x] **Fixed a phase-1 bug found while building this:** the UI's level regex
      assumed itzg's console format (`[12:34:56]`), but the *file* uses
      `[01Aug2026 22:55:34.358]`. Every line was classified INFO, so the
      ERROR/WARN chips silently did nothing. Now matches on shape
      (`[…] […/LEVEL]`) instead of on the timestamp.

### Phase 3 — TPS — **DONE**

- [x] RCON client, compile-time allowlist, 30s poll.
- [x] Confirmed `neoforge tps` is the right command, and captured its real
      output format (see *RCON* above — it is not what this doc first assumed).

### Phase 4 — join/leave browser notifications — **DONE, live 2026-08-02**

- [x] Bell toggle, opt-in, persisted in `localStorage`; permission requested from
      the click gesture; blocked state surfaced.
- [x] Seen-set newness detection + unconditional first-response baseline.
- [x] Burst coalescing; 5s poll while enabled.
- [x] Fixed a bug this introduced: the level-chip handler bound to every
      `.chip`, which now includes the bell. Scoped to `.chip[data-level]`.

Verified by extracting the shipped block from `index.html` and running it in node
against stubbed `localStorage`/`Notification`: silent baseline, new event
notifies, re-polling silent, restart-replay silent (same instance *and* after a
reload), burst coalescing counts, the returning-user edge case, toggle-off still
tracking, denied permission silent, seen-set bounded at the 500 newest keys.

**Not covered by that harness:** the actual browser permission prompt and whether
notifications render — those need a real browser and were not tested. Everything
downstream of `Notification.permission === "granted"` is stubbed.

### Follow-ups

- [ ] Uptime Kuma HTTP monitor against `http://monitor:8080/healthz` (internal,
      unauthenticated by construction).
- [ ] Decide whether "search all history" (server-side grep over rotated
      `.log.gz`) is missed. Deferred from v1 on purpose: `latest.log` on a
      164-mod server is mostly boot spam, and the real workflow is watching the
      last few hundred lines filtered by a player name or `ERROR`.

## Verification

A 200 on `/` proves the server is up, not that the page works (see
`spa-curl-200-is-not-verification`). After deploy, check all of:

```bash
# auth actually gates it (expect 401)
curl -s -o /dev/null -w '%{http_code}\n' https://monitor.${BASE_DOMAIN}/

# with creds, expect 200 + a non-empty buffer
curl -s -u "$MONITOR_USER:$MONITOR_PASSWORD" \
  "https://monitor.${BASE_DOMAIN}/api/log?since=0" | jq '.seq, (.lines|length)'

# the port is NOT reachable from outside the compose network
nmap -Pn -p 8080 <host-public-ip>        # expect closed/filtered
docker compose port monitor 8080          # expect an error: no published port
```

Then the one that actually matters, because it is the failure this design is
most likely to hit: **restart the Minecraft server and confirm the stream keeps
moving.** That is the rotation path, it fires on every deploy, and a broken
tailer stays HTTP-200 while going silent.

### Phase 1 results (cochi, 2026-08-01)

The deploy itself exercised the rotation path: monitor came up while the old
`latest.log` still existed, Minecraft then created a new one, and the tailer
reported `rotations: 1` and kept streaming. Verified in prod:

| Check | Result |
|---|---|
| `GET /` no creds / wrong creds / right creds | 401 / 401 / 200 |
| TLS | `CN=monitor.ulicraft.net`, Let's Encrypt |
| Buffer after boot | 908 lines, `tail.ok: true` |
| Stream advancing | +109 lines in 12s, `dropped: 0`, `reset: false` |
| Rotation across the MC restart | `rotations: 1`, stream uninterrupted |
| Port 8080 from the internet | connection refused |
| `docker compose port monitor 8080` | `:0` (nothing published) |
| Host listeners on :8080 | none |

Also exercised locally before deploy, against a synthetic log: backfill capped
to buffer size, partial (unterminated) lines withheld until their newline lands,
`mv`-style rotation, in-place truncation, delete-and-recreate, ring overflow
(`dropped: 15, reset: true`), a cursor from the future, malformed `since` (400),
unknown path (404).

**Known cosmetic gap:** a rotation done as delete-then-recreate (rather than
`mv`) does not increment the `rotations` counter — the tailer takes the
"file was gone, reopen" branch instead of the rotation branch. Streaming is
unaffected; only the displayed count is.

### Phase 2 + 3 results (cochi, 2026-08-01)

Deployed **rolling**, not `--hard` — no caddy conf changed, so only the monitor
container was recreated and Minecraft was never restarted (zero downtime).

Live values on first request: server online, MOTD "Uli LAN party", 1.21.1 ·
**14** registered accounts · **20.000 TPS / 0.307 ms/tick across 6 dimensions** ·
88 GB free, 59% used · newest backup 0h old, 57 archives, 14 GB.

Verified against *real* data rather than assumptions:

| Check | Result |
|---|---|
| `neoforge tps` over the in-repo RCON client | 20.000 TPS, 6 dimensions parsed |
| RCON allowlist rejects `op`/`stop`/`say`/`ban`/`LIST`/`"neoforge tps "` | all rejected (unit test) |
| Wrong RCON password | clean `authentication failed`, other tiles unaffected |
| Session parser vs. real rotated-log join lines | `Patxakita joined/left`, timestamps intact |
| UI level regex vs. real log lines | WARN and ERROR now distinguished (was: all INFO) |
| NMSR avatar URL the heads use | HTTP 200, `image/png` |
| mc-status killed | `server_err` set; disk, backup and log stream unaffected |
| 25 concurrent `/api/summary` under `-race` | 0 data races |
| `/api/summary` with a dead dependency | 0.24s (sequential would stack timeouts) |