0.67.2
Guides

Backups and restore

A backup is a zfs send stream taken from a snapshot, not a tarball of the files. Ownership, extended attributes, ACLs, sparseness and recordsize all survive, because none of them are being re-derived on the way out.

Status: shipped and exercised. Local backups, offsite shipping to S3, a NAS, a storage location or another node, per-destination encryption, restore and scheduled restore verification all run on a live cluster.

What a backup run does

  1. Stop the application.
  2. Take a snapshot.
  3. Start the application again.
  4. zfs send from the snapshot, while the application is already running.

The application is down for the snapshot only: seconds, not the length of the transfer. The stream is checksummed and carries a self-describing manifest, so a backup can be identified without the database that produced it.

Where backups go

Local is always primary. Every backup lands on this node's pool first, and that copy is what a restore reads by default.

Each application's backup policy then names exactly one offsite destination. Every backup of that application ships there, the scheduled ones and the manual ones alike.

Kind What it is
s3 An S3-compatible bucket. Anything with an S3 API: Cloudflare R2, MinIO, AWS
nas An SMB or NFS connection, mounted and managed by ProxDock
location A cluster-wide storage location, under its reserved .proxdock/backups subtree
node Another node in the cluster, streamed over the internal mTLS channel

Leaving the destination empty means local only.

Shipping looks after itself

The shipper runs as a scan loop rather than a one-shot after each backup, which makes it self-healing:

  • Changing an application's destination backfills its existing backups to the new place.
  • A failed upload retries after a cooldown rather than being lost.
  • Per-destination retention prunes independently of local retention.
  • Copies at a destination you have stopped using stay until its own retention expires them or you remove them. Changing a dropdown never deletes offsite data.

Encryption

A destination you do not trust with plaintext can be marked encrypted. It is a per-target toggle, applied by the shipper at ship time, so the same backup is plaintext locally and sealed on the way out.

Local artifacts stay plaintext, because a local disk is your own trust domain. Node-to-node copies also ship plain: both ends are the same cluster by definition, and the hop is already mTLS.

The format is an age-style chunked AEAD stream: XChaCha20-Poly1305 over fixed-size chunks, each sealed with a counter nonce derived from a random per-file prefix, with the final chunk flagged. Truncation, reordering and tampering all fail decryption rather than yielding silently wrong bytes.

Keys are 32-byte symmetric keys in a replicated keyring. The newest key encrypts; older ones are kept so backups written before a rotation still decrypt. Each key has a recovery phrase, which is the key itself in printable form: offline insurance for the case where the cluster and its store are both gone.

Write the recovery phrase down somewhere that is not the cluster. It is the only thing that opens an encrypted offsite backup if you have lost everything else.

Restoring

Restore always receives into a fresh dataset and retains the previous one, so a restore that turns out to be wrong is a rollback rather than a loss.

A restore can also run under a new name and a new address, which puts a copy beside the original. That is the way to test a restore without touching the thing you are testing it against.

Restores can read from an offsite copy as well as the local one; the restore dialog lists the copies that survive for that backup and you pick the source.

Browsing without restoring

Individual files can be browsed and extracted from a snapshot without a full restore. This is the common case for "I deleted one config file", and it does not need a spare dataset's worth of space.

Scheduling, retention and verification

Jobs run on a schedule with per-kind retention. Retention also covers the datasets left behind by deletes and migrations, so a cluster does not silently fill up with orphans nobody is tracking.

Scheduled restore verification restores backups on a timer into throwaway clones and reports what it finds. A backup that cannot be restored is a failure worth hearing about on a Wednesday rather than during an outage.

Outline: the retention policy grammar and the interaction between a job's schedule and a manual run still need writing up here.