Decide whether to keep Hetzner's root-disk backups #6

New issue

Open

opened 2026-11-04 06:03:07 +00:00 by slyons · 0 comments

slyons commented

2026-11-04 06:03:07 +00:00

Owner

Problem

The Hetzner server has provider backups switched on (backups = true in infra/terraform/modules/node-hetzner/main.tf). They cost about 20% of the server price on top. They copy only the root disk, and the root disk holds the LUKS key for the data volume and the node's SSH host key (docs/security.md, Known limits; docs/operations/runbook.md, Backups and restore). Anyone who can restore those backups in the Hetzner project can read all cluster data.

They add almost nothing to recovery:

Lost VM: tofu apply re-creates the server, with the same SSH host key and volume key, from the encrypted OpenTofu state, and cloud-init reattaches and unlocks the data volume (runbook, Disaster recovery).
Lost data: Postgres is archived continuously and volumes are backed up nightly with restic, both to object storage. Hetzner backups don't include the data volume at all.

What they still give: a quick rollback of the root disk after a bad change on the node, such as an update that breaks boot. A fresh tofu apply gives the same result in a few minutes.

Proposal

Decide whether to keep them. If not:

Set backups = false on hcloud_server.node. Check the field against the pinned hetznercloud/hcloud provider version first.
Remove the Hetzner-backups entry from Known limits in docs/security.md and the paragraph about them in the runbook's Backups and restore.
Run tofu apply and check that Hetzner shows backups as disabled and the old backups are deleted.

Done when: the decision is made, and either the backups are gone and the docs no longer mention them, or the docs say why they're kept.

Design doc

Not needed.

### Problem The Hetzner server has provider backups switched on (`backups = true` in `infra/terraform/modules/node-hetzner/main.tf`). They cost about 20% of the server price on top. They copy only the root disk, and the root disk holds the LUKS key for the data volume and the node's SSH host key (`docs/security.md`, *Known limits*; `docs/operations/runbook.md`, *Backups and restore*). Anyone who can restore those backups in the Hetzner project can read all cluster data. They add almost nothing to recovery: - **Lost VM:** `tofu apply` re-creates the server, with the same SSH host key and volume key, from the encrypted OpenTofu state, and cloud-init reattaches and unlocks the data volume (runbook, *Disaster recovery*). - **Lost data:** Postgres is archived continuously and volumes are backed up nightly with restic, both to object storage. Hetzner backups don't include the data volume at all. What they still give: a quick rollback of the root disk after a bad change on the node, such as an update that breaks boot. A fresh `tofu apply` gives the same result in a few minutes. ### Proposal Decide whether to keep them. If not: 1. Set `backups = false` on `hcloud_server.node`. Check the field against the pinned `hetznercloud/hcloud` provider version first. 2. Remove the Hetzner-backups entry from *Known limits* in `docs/security.md` and the paragraph about them in the runbook's *Backups and restore*. 3. Run `tofu apply` and check that Hetzner shows backups as disabled and the old backups are deleted. **Done when:** the decision is made, and either the backups are gone and the docs no longer mention them, or the docs say why they're kept. ### Design doc Not needed.

slyons added the

labels

2026-11-04 06:03:07 +00:00