Running a cluster
Every node serves the whole UI and the whole API. There is no admin node and no node you have to be on to do a particular thing.
Status: shipped and running. This page is an outline of behaviour that is already stable.
How a request is served
- Reads come from the local replica, so they do not cross the network.
- Writes go through the Raft leader, wherever it happens to be.
- Node-local operations, anything that has to touch a specific host's Docker or ZFS, are proxied exactly one hop over mTLS, with the original user asserted so the audit trail names a person rather than a machine.
Joining a node
A node joins with a token generated on an existing node. The token is short-lived and single-use, and issuing it also issues the joining node's cluster-internal mTLS certificate. There is no shared secret to distribute and nothing to copy between hosts by hand.
Quorum
Losing a majority makes the cluster read-only, and it says so in a banner rather than failing one write at a time with an unhelpful error. Reads keep serving from the local replica throughout. Applications on a node that is merely unreachable keep running if the host itself is alive; they just cannot be managed until it comes back.
Migrating an application
Migration is the same code path as backup and restore, which means the restore path is exercised every time an application moves rather than on the day you need it.
- Preflight. Target pool, zone presence and free space are checked first.
- Pre-seed. The bulk of the data is sent while the application is still running.
- Cutover. Stop, incremental send, start.
The address moves with the application, and because the MAC is derived from the address, the rest of the network does not need to learn anything new. The previous dataset is retained for rollback.
Upgrading
Installing the packages is the whole procedure. There is no separate migration step to run, no maintenance mode, and nothing to coordinate by hand.
On one node:
apt update && apt upgrade
On a cluster, the same two commands on each node. Schema migrations are append-only and ship inside the binary, so upgrade every node together when a release adds one; the changelog says which releases do.
What the cluster does on its own
Each node advertises the schema version its binary supports in its heartbeat. The leader watches for that, and the moment every node advertises a pending schema, it raises the cluster's schema epoch itself and checkpoints the result so every replica proves it holds identical state afterwards.
Nodes therefore upgrade one at a time without the cluster tearing: a node running the new binary keeps serving on the old schema until the last one catches up.
- A node that is still down holds the upgrade. That is deliberate: the epoch does not move until every member can honour it. The Nodes view names what it is waiting on.
- The Finalize button is a manual accelerator, nothing more. Left alone, the leader gets there within about a minute of the last node coming up.
- Replica digests are the proof. After finalizing, every node digests its own copy of the store at the same log position; matching digests are what "in sync" means on the Nodes view.
Rolling back
Downgrading a node below the cluster's committed schema epoch is not supported. Migrations are append-only and a raised epoch is not lowered, so plan an upgrade as something you go forward through. Restoring a node from a snapshot re-runs migrations against old state, which is the one path that can diverge a replica; re-seed it from the leader instead.