Clustering

Cluster mode joins several Quazzar Cloud OS machines into one fleet. One node is the primary: it keeps the cluster state, watches the other nodes and decides where apps run. Every other node is a worker that reports its load and runs the apps the primary places on it.

• Nodes join with a one-time token and an admin approval on the primary.
• Nodes talk over an encrypted WireGuard mesh (10.42.0.0/16, the primary is 10.42.0.1).
• Workers send a heartbeat every 10 seconds with CPU, memory, disk and their apps.
• A node that misses 3 heartbeats (30 s) is marked offline and its apps are rescheduled.
• Placement rules pin apps to nodes by label (for example GPU apps on the GPU box).

Before you start

• Two or more machines with Cloud OS installed (the join script installs it if missing).
• The machines can reach each other on the LAN: the primary’s web port (HTTPS) and UDP 51820 for WireGuard.
• wireguard-tools installed on every node (the packages recommend it).
• Ideally the same Cloud OS version everywhere.

Create the cluster

On the machine that should be the primary, open Cluster → Nodes and click Create cluster. On a single machine you get there by picking Cluster (tagged Create) in the dock’s mode picker, or from dock → Infra → Cluster in Server mode (see The desktop). Once the cluster exists, Cluster mode has three spaces: Constellation (the fleet live), Nodes and Placement. This:

1. creates the WireGuard hub interface wg0 (10.42.0.1),
2. registers the machine itself as the primary node,
3. bootstraps Raft on the node’s mesh/LAN address with a stable node id.

The primary is also a normal placement target — apps can run on it.

Add a node

1. On the primary, click Add node. Optionally set labels (for example zone=home) and how long the token stays valid (default 15 minutes).

2. Copy the command and run it on the new machine:

   curl -sfL https://quazzar-01.local/api/cluster/join.sh | sudo sh -s -- \
     --token qz_… --primary https://quazzar-01.local

   The script installs Cloud OS if needed and asks the local node to send a join request. Tokens are single-use and expire; a used or expired token is refused.

3. The machine appears on the primary as pending. Click Approve (or Reject).

4. Within a few seconds the worker receives its mesh address and credentials, brings up WireGuard and starts heartbeating. It shows as online with live CPU / memory / disk.

Already logged in on the worker? You can also run quazzar-cli cluster join --token qz_… --primary https://quazzar-01.local (with sudo on a fresh node).

Node states

A node can also be cordoned (no new apps are placed on it) and draining (being emptied).

Placement rules

A rule says “these apps should run on nodes matching this selector”:

Selector grammar — comma-separated terms, all must match:

• key=value — label present with that value
• key — label present
• !key — label absent

Example: gpu,zone=home,!battery.

Label nodes from the node panel (Labels). Apps without a rule may run on any online, uncordoned node. Among the candidates the scheduler picks the least-loaded node (CPU + memory, fewer apps breaks ties).

When a node goes offline

After 3 missed heartbeats the primary marks the node offline and records a heartbeat lost event. For each app on it:

• Stateless apps (templates without volumes) are redeployed on the best candidate. They show as rescheduling until the new node reports them running. When the old node comes back it is told to remove its stale copy.
• Apps with persistent data are never moved automatically — they show as manual with an event explaining why. They keep running on their node when it returns; to move one, restore it from a backup on another node.

Drain, cordon, wake, remove

• Cordon — stop placing new apps on the node. Uncordon reverses it.
• Drain — cordon + move its stateless apps elsewhere (apps with data are flagged manual). Use it before maintenance.
• Wake — sends a Wake-on-LAN magic packet to the node’s MAC (recorded at join) from the primary’s LAN. The node’s BIOS/NIC must have WoL enabled.
• Remove — reschedules its apps, removes its WireGuard peer and revokes its credential. The node must re-join with a new token.

Rebalance

Rebalance moves stateless apps only where it clearly helps (a node that no longer satisfies an app’s rule, or a large load difference). Use Preview (dry run) to see the planned moves first.

Events

The cluster keeps an event feed: join requests, approvals, nodes lost / recovered, apps moved, manual moves needed, cordon / drain, Wake-on-LAN, rebalances, Raft leader changes and failover drills.

High availability (Raft)

The primary runs Raft. HA → Failover drill measures failover: on a multi-voter cluster it transfers leadership and reports how long a new leader took; on a single voter it measures a commit round-trip. Additional Raft voters are added from the HA page.

Troubleshooting

A node stays pending

Approve it on the primary (Cluster → Nodes). The worker polls every 5 s and joins automatically.

A node shows offline

Check the service on the node (systemctl status quazzar) and that it can reach the primary URL. It returns online on its next heartbeat.

wg-quick errors

Install wireguard-tools. Cloud OS runs wg and wg-quick through its privileged helper (quazzar-setup priv), so no extra sudo rule is needed; the package ships the helper. In containers or labs without WireGuard set QUAZZAR_CLUSTER_NO_WG=1.

Wake does nothing

Enable Wake-on-LAN in the node’s BIOS and NIC (ethtool -s eth0 wol g). The primary and the node must be on the same L2 network.