Require the platform checks and k3d smoke test in CI for platform changes #5

New issue

Open

opened 2026-11-04 06:01:18 +00:00 by slyons · 0 comments

slyons commented

2026-11-04 06:01:18 +00:00

Owner

Problem

tools/smoke/k3d.sh is the only check that runs the real bootstrap against a real cluster, twice, and proves the platform comes up healthy and idempotent. Today it runs only when someone remembers to run it by hand. Changes to infra/k8s and tools/bootstrap can merge without it, and nothing in CI notices.

The same is true of the cheaper checks for those areas. Presubmit runs only the Bazel checks. shellcheck, tofu fmt/tofu validate and the kustomize | flux envsubst | kubeconform manifest check (all listed in CLAUDE.md) run nowhere automatically.

Proposal

Make both a required part of review for platform changes.

Static checks in presubmit (small, do first). Add a job that runs shellcheck, the OpenTofu checks and the manifest check on every pull request. They need no secrets and take seconds. Pin each tool by version and checksum, as the Bazel setup does.
The smoke test in CI. Run tools/smoke/k3d.sh for every pull request that touches infra/k8s/** or tools/bootstrap/**. Make its status required in branch protection.

Constraints the design must handle

It can't run on the production node. The node has 4 vCPU and 8 GB. A second full platform (Forgejo, Synapse, two Postgres clusters, Flux) running inside CI would starve the real one for six minutes on every run.
It needs a Docker daemon. Jobs deliberately get no Docker socket (docker_host: "-" in infra/k8s/ci/runner.yaml), because a job that drives dind could tamper with other jobs.
It runs untrusted code. Pull requests come from forks, and the smoke test runs the change's own bootstrap with root-equivalent power over its cluster. The machine it runs on must hold nothing worth stealing and must be thrown away afterwards.
The owner holds every key. If CI creates a throwaway server, it needs a cloud token. That token must live in a separate Hetzner project with nothing else in it, be limited to that project, and be unable to reach the platform's project, DNS or state.
Required checks and path filters. A required status that never reports blocks every pull request that doesn't touch those paths. The job must always run and finish quickly as a no-op when the paths are untouched.

Directions to evaluate

Separate runner on an ephemeral VM. A second runner label (e.g. smoke) on a cheap, short-lived Hetzner server, created for the run and deleted afterwards, in its own Hetzner project.
A small, permanent second runner VM. Simpler to operate, at a few euros a month. Each run must still get a fresh Docker host so one pull request can't poison the next.
Run on approval only. Run the smoke test only after a maintainer approves the run (as for a new contributor's first run), so untrusted code never starts it unattended.

Write down what the smoke test still can't cover (its header lists cloud-init, TLS, backups and more) next to the required check, so a green result isn't read as more than it is.

Done when: a pull request that touches infra/k8s or tools/bootstrap can't merge until the static checks and the k3d smoke test pass in CI, and other pull requests aren't slowed down by it.

Design doc

Needed for step 2: where it runs, who holds which token, and how each run is isolated.

### Problem `tools/smoke/k3d.sh` is the only check that runs the real bootstrap against a real cluster, twice, and proves the platform comes up healthy and idempotent. Today it runs only when someone remembers to run it by hand. Changes to `infra/k8s` and `tools/bootstrap` can merge without it, and nothing in CI notices. The same is true of the cheaper checks for those areas. Presubmit runs only the Bazel checks. `shellcheck`, `tofu fmt`/`tofu validate` and the `kustomize | flux envsubst | kubeconform` manifest check (all listed in `CLAUDE.md`) run nowhere automatically. ### Proposal Make both a required part of review for platform changes. 1. **Static checks in presubmit (small, do first).** Add a job that runs shellcheck, the OpenTofu checks and the manifest check on every pull request. They need no secrets and take seconds. Pin each tool by version and checksum, as the Bazel setup does. 2. **The smoke test in CI.** Run `tools/smoke/k3d.sh` for every pull request that touches `infra/k8s/**` or `tools/bootstrap/**`. Make its status required in branch protection. ### Constraints the design must handle - **It can't run on the production node.** The node has 4 vCPU and 8 GB. A second full platform (Forgejo, Synapse, two Postgres clusters, Flux) running inside CI would starve the real one for six minutes on every run. - **It needs a Docker daemon.** Jobs deliberately get no Docker socket (`docker_host: "-"` in `infra/k8s/ci/runner.yaml`), because a job that drives dind could tamper with other jobs. - **It runs untrusted code.** Pull requests come from forks, and the smoke test runs the change's own bootstrap with root-equivalent power over its cluster. The machine it runs on must hold nothing worth stealing and must be thrown away afterwards. - **The owner holds every key.** If CI creates a throwaway server, it needs a cloud token. That token must live in a separate Hetzner project with nothing else in it, be limited to that project, and be unable to reach the platform's project, DNS or state. - **Required checks and path filters.** A required status that never reports blocks every pull request that doesn't touch those paths. The job must always run and finish quickly as a no-op when the paths are untouched. ### Directions to evaluate - **Separate runner on an ephemeral VM.** A second runner label (e.g. `smoke`) on a cheap, short-lived Hetzner server, created for the run and deleted afterwards, in its own Hetzner project. - **A small, permanent second runner VM.** Simpler to operate, at a few euros a month. Each run must still get a fresh Docker host so one pull request can't poison the next. - **Run on approval only.** Run the smoke test only after a maintainer approves the run (as for a new contributor's first run), so untrusted code never starts it unattended. Write down what the smoke test still can't cover (its header lists cloud-init, TLS, backups and more) next to the required check, so a green result isn't read as more than it is. **Done when:** a pull request that touches `infra/k8s` or `tools/bootstrap` can't merge until the static checks and the k3d smoke test pass in CI, and other pull requests aren't slowed down by it. ### Design doc Needed for step 2: where it runs, who holds which token, and how each run is isolated.

slyons added the

labels

2026-11-04 06:01:18 +00:00