Run large apps on their own cluster, separate from the platform #3

New issue

Open

opened 2026-11-04 05:32:44 +00:00 by slyons · 0 comments

slyons commented

2026-11-04 05:32:44 +00:00

Owner

Problem

Tenant apps (today, hello) run on the platform's only node, next to Forgejo, Synapse, CI and every platform secret. Once an app grows, that matters in two ways:

Performance: an app's traffic spike, or an app that misbehaves, competes for CPU, memory and disk with the forge and chat. It can slow down or take down the tools every contributor depends on.
Security: tenants are contained by a namespace-scoped role, admission policies (ClusterIP-only Services, their own hostname only) and default-deny network policies. Those are good controls, but tenants still share a kernel and a node with the forge. A container escape in a tenant reaches the node, and the node holds every platform secret. A separate cluster is a much stronger boundary.

Proposal

Keep this cluster small and stable, running only the platform: forge, chat and CI. Run large apps on one or more separate app clusters, which Flux deploys from the same monorepo. This is how large organizations separate their engineering tools from their products.

What the design needs to cover:

Infrastructure: an OpenTofu environment for app cluster nodes, ideally in its own Hetzner project, so that compromising the app cluster gives no access to the platform's servers, volumes or backups.
Deployment: Flux in the app cluster pulls from the forge with read-only credentials, as the platform cluster does today, so CI never holds cluster credentials.
Images: pulled from the forge registry with read-only credentials.
Tenants: the tenant model (namespace per app, the tenant role, admission policies, network policies) moves to the app cluster. App hostnames in DNS point at it.
Growth beyond one node: storage that isn't tied to a node (Hetzner volumes per claim), a private network between nodes, and a load balancer or ingress node.
Operations: backups, the runbook, and the smoke test for the new cluster.

Cheaper steps that may postpone this, in order:

A bigger server (server_type; resizing takes about a minute of downtime).
Moving CI runners to separate machines, plus a Bazel remote cache. CI will be the heaviest load long before app traffic is.

When: when a real app needs more than the shared node can safely give it, not before.

Design doc

To be written from docs/design-doc-template.md before work starts; this is a large change.

### Problem Tenant apps (today, `hello`) run on the platform's only node, next to Forgejo, Synapse, CI and every platform secret. Once an app grows, that matters in two ways: - **Performance:** an app's traffic spike, or an app that misbehaves, competes for CPU, memory and disk with the forge and chat. It can slow down or take down the tools every contributor depends on. - **Security:** tenants are contained by a namespace-scoped role, admission policies (ClusterIP-only Services, their own hostname only) and default-deny network policies. Those are good controls, but tenants still share a kernel and a node with the forge. A container escape in a tenant reaches the node, and the node holds every platform secret. A separate cluster is a much stronger boundary. ### Proposal Keep this cluster small and stable, running only the platform: forge, chat and CI. Run large apps on one or more separate **app clusters**, which Flux deploys from the same monorepo. This is how large organizations separate their engineering tools from their products. What the design needs to cover: - **Infrastructure:** an OpenTofu environment for app cluster nodes, ideally in its own Hetzner project, so that compromising the app cluster gives no access to the platform's servers, volumes or backups. - **Deployment:** Flux in the app cluster pulls from the forge with read-only credentials, as the platform cluster does today, so CI never holds cluster credentials. - **Images:** pulled from the forge registry with read-only credentials. - **Tenants:** the tenant model (namespace per app, the `tenant` role, admission policies, network policies) moves to the app cluster. App hostnames in DNS point at it. - **Growth beyond one node:** storage that isn't tied to a node (Hetzner volumes per claim), a private network between nodes, and a load balancer or ingress node. - **Operations:** backups, the runbook, and the smoke test for the new cluster. Cheaper steps that may postpone this, in order: 1. A bigger server (`server_type`; resizing takes about a minute of downtime). 2. Moving CI runners to separate machines, plus a Bazel remote cache. CI will be the heaviest load long before app traffic is. **When:** when a real app needs more than the shared node can safely give it, not before. ### Design doc To be written from `docs/design-doc-template.md` before work starts; this is a large change.

slyons added the

labels

2026-11-04 05:32:44 +00:00