High availability

A Wharfbase hub normally keeps everything in one SQLite file in its data volume. That is simple, and it means the hub is one container you can’t double: if its host goes down, so does Wharfbase.

For teams that run Wharfbase as shared infrastructure, a hub can instead keep its data in PostgreSQL. Then a second hub can wait next to it as a standby, and take over within seconds when the first one stops. Standby hubs are part of the Team plan. The PostgreSQL store itself works on every plan.

How it works

• One hub leads. Every hub connects to the same PostgreSQL. The one holding a lock in PostgreSQL leads: it serves the UI and the API, talks to the agents, and runs scans, schedules, alerts and backups.
• The others stand by. A standby answers /healthz (alive) and /readyz with 503 (not ready), so the load balancer sends it nothing. It tries for the lock every 2 seconds.
• Taking over. When the leading hub stops or crashes, its lock goes with its connection, and a standby has it within about 2 seconds. It reads the store, becomes ready, and the load balancer starts sending it traffic. The audit log records the takeover (hub.leader).
• A frozen hub. A hub that hangs (a paused VM, a lost network) stops confirming its lock. PostgreSQL ends its connection after 30 seconds, and the standby takes over. Before the hub writes anything again, it confirms its lock, finds it gone, and stops. Two hubs never write at the same time.
• Every server is an agent. Hubs in this setup run with WB_ROLE=hub: a hub that can move between hosts can’t call one host’s Docker its own. Install the agent on each server you want Wharfbase to watch (Settings → Servers), pointing it at the load balancer’s address. In Kubernetes it’s different: every replica reads the same cluster, so replicas there keep discovering it (Kubernetes).

Signed-in users stay signed in through a takeover. A SQL query or a backup running on the hub that stopped is lost and needs starting again.

What you need

• PostgreSQL 15 or later, ideally one you already run with backups and failover of its own. Wharfbase creates its tables in a schema (wharfbase by default) and needs a user that can create it.
• The Team plan, for a hub to take over. Without it, a second hub waits but doesn’t take over (it logs why).
• The same master key on every hub, from a secret: WB_MASTER_KEY_FILE or WB_MASTER_KEY. A key generated inside one hub’s data volume isn’t on the other.
• A shared volume mounted at the same path on every hub, set as WB_SHARED_DIR. Backups, export files and the key agents use to trust the hub (hub.key) live there. On one host a Docker volume does; across hosts use NFS, EFS or a ReadWriteMany claim.
• A load balancer that checks /readyz and sends traffic only to the hub that answers 200. It must pass WebSockets through: the UI and the agents use them.

Each hub keeps its own data volume for local things only: temporary files, SQLite snapshots and the admin socket.

Set it up

deploy/docker-compose.ha.yml runs two hubs, PostgreSQL and a Caddy load balancer on one host, with comments on every part. It’s meant to show the moving parts: for real availability, put the hubs on different hosts and use a highly available PostgreSQL.

mkdir -p secrets
head -c 32 /dev/urandom | base64 > secrets/master.key
head -c 24 /dev/urandom | base64 | tr -d '/+=' > secrets/pg_password
printf 'postgres://wharfbase:%s@postgres:5432/wharfbase' "$(cat secrets/pg_password)" > secrets/store_url
sudo chown 10001:10001 secrets/master.key secrets/store_url
sudo chmod 0400 secrets/master.key secrets/store_url
docker compose -f docker-compose.ha.yml up -d
docker compose -f docker-compose.ha.yml exec hub-a wb setup-token

Open the load balancer’s address, finish the setup, and enter a Team license in Settings → License. wb doctor on the leading hub shows where the store is, that the hub holds the lock, and whether a standby has what it needs.

The settings

Variable	Default	What it does
WB_STORE_URL	(SQLite)	postgres://user:password@host:5432/db, with ?sslmode=require for TLS. Prefer WB_STORE_URL_FILE, so the password stays out of the environment
WB_STORE_URL_FILE		A file holding the URL (a Docker or Kubernetes secret)
WB_STORE_SCHEMA	wharfbase	The schema for Wharfbase’s tables: one per Wharfbase, if several share a database
WB_STORE_POOL	10	Connections each hub keeps to PostgreSQL
WB_SHARED_DIR	the data dir	Backups, exports and hub.key, on the volume every hub shares
WB_ROLE	all	hub for a hub that can stand by (or WB_DOCKER=false, as in Kubernetes)

A single hub can use PostgreSQL too, with any role, and without the Team plan. A second hub on the same store is refused if it watches a Docker of its own (any role but hub, unless WB_DOCKER=false).

In Kubernetes

The Helm chart does all of the above with store.existingSecret and replicaCount: see Standby replicas.

Moving an existing hub to PostgreSQL

wb store migrate copies a running hub’s store into an empty PostgreSQL schema:

docker exec -it -e WB_STORE_URL="postgres://wharfbase:…@db:5432/wharfbase" wharfbase wb store migrate

Pass the URL with -e, not as an argument, so the password doesn’t show in the process list.

• During the copy changes pause; reads go on. The copy is one transaction in PostgreSQL: it’s all there, or nothing is.
• Checked: the rows of every table are counted on both sides, and the audit log’s hash chain is verified on the copy.
• Afterwards the hub only reads: changes are refused (STORE_MOVED) and /readyz says not ready. Set WB_STORE_URL for the container and restart it (docker compose up -d).
• The SQLite file stays in the data volume, untouched since the copy. A file next to it, store-moved.json, keeps Wharfbase from starting on it again by mistake. To go back to SQLite instead, delete that file.

The schema must be empty: wb store migrate won’t merge into an existing store.

Upgrades

docker compose pull && docker compose up -d replaces the hubs one at a time. The new version waits as a standby while the old one still leads. It takes over, and updates the tables, once the old one stops. An old version that finds newer tables refuses to start (SCHEMA_TOO_NEW), so two versions never write to the same store.

Backups

PostgreSQL is yours to back up, with the tools you use for it already (pg_dump, snapshots, point-in-time recovery). The stored passwords are sealed with the master key before they reach PostgreSQL: a dump without the key holds no readable secrets. Back up the master key separately.

The database backups Wharfbase makes (Pro) are in WB_SHARED_DIR/backups.