Staged Rollouts

🔒 Business plan and above. Staged rollouts and the older canary deployments are part of the Business and Enterprise plans. See Control Center features by plan.

A staged rollout moves a change across your fleet a slice at a time. By default the slices are 1 % → 10 % → 50 % → 100 %. The change can be a Cloud OS version, an app or a config value. Each stage has to stay healthy for a soak period before the next one starts. If a health gate is breached, the rollout pauses on its own and records why.

You’ll find rollouts under Operations → Rollouts in the Control Panel. The card at the top shows the selected rollout: one tile per stage (share of the fleet, node count, progress and state), the live value of every health gate next to its threshold, and the Roll back, Pause / Resume and Promote to … buttons. Below it are the list of rollouts (click one to select it), its timeline and its per-node status.

Click New rollout to start one: pick what to roll out (a Cloud OS version, an app or a config value), the stage ladder, the soak time per stage, the health gates, which nodes take part (groups, regions, tags; every online node when nothing is picked) and whether stages promote themselves.

Links such as /operations?rollout=<id> open a specific rollout, and /operations?new=rollout opens the New rollout dialog.

How a rollout runs

1. Pick the nodes. When you create the rollout, the Control Panel takes every online node that matches your filter (groups, regions, tags or an explicit node list). OS rollouts also skip nodes that already run the target version. Offline nodes are left out.
2. Order them. Nodes are sorted by a hash of their id. The order is stable, so the same nodes always go first and your 1 % canaries stay the same from one rollout to the next.
3. Split into stages. Each stage covers a cumulative share of the selected nodes. For example, 5,284 nodes on the default ladder give 53, 529, 2,642 and 5,284 nodes.
4. Dispatch the stage. Every node in the current stage gets the update command (update.apply for OS versions). A node that went offline since selection is skipped.
5. Soak. Once every node of the stage has reported back, the soak timer starts (15 minutes by default). The health gates are checked throughout.
6. Promote. When the soak passes, the rollout either moves to the next stage by itself (auto-promote) or waits for you in awaiting promotion.

Health gates

The gates are evaluated over every node the rollout has touched so far, using fleet metrics and heartbeats since the current stage started.

On a breach the rollout goes to paused. The reason is saved (for example health gate breached: p95 CPU 93.0% >= 80.0%) and added to the rollout timeline. The latest gate values are always shown on the rollout.

Controls

On plans below Business

On Community or Pro, Operations → Rollouts shows an upgrade card with a Business badge and an Upgrade button that opens Billing. Creating a rollout (POST /api/fleet/rollouts) or a canary (POST /api/fleet/canary) returns 402 with the standard upgrade_required body ("feature": "staged_rollouts", "required_plan": "business").

If you downgrade while rollouts are in progress:

• Rollouts that already started keep running to completion. Stages still soak, promote (automatically or by hand) and pause on a health-gate breach. You can still Pause, Resume, Promote and Roll back them.
• No new rollout can start. The New rollout button turns into an upgrade link. A rollout that had not dispatched any node yet (for example one created seconds before the downgrade) is paused with the reason plan no longer includes staged rollouts and can be resumed after you upgrade.
• The rollout list and details stay readable. GET /api/fleet/rollouts adds "gated": true and "required_plan": "business".

Older canary deployments

Canary deployments created at /api/fleet/canary still work as before. They also appear in the rollout list, mapped to two stages (canary share → 100 %) and flagged legacy: true. You can promote or roll back a legacy canary from the rollout endpoints (and from the Rollouts tab, where they carry a legacy badge). Pause and resume are not available for them, so those buttons stay disabled.

API

Below Business, POST /api/fleet/rollouts and POST /api/fleet/canary return 402 upgrade_required, and resuming a rollout that never started returns 402 too. Every other endpoint keeps working so a started rollout can finish.

Create a rollout:

POST /api/fleet/rollouts
{
  "name": "Cloud OS 3.2.1",
  "kind": "os_version",
  "target_version": "3.2.1",
  "stages": [1, 10, 50, 100],
  "soak_sec": 900,
  "gates": { "max_error_rate": 0.005, "max_cpu_p95": 80, "max_failed_boots": 0, "max_heartbeat_loss": 0 },
  "filter": { "group_ids": [], "regions": ["eu-west"], "tags": [] },
  "auto_promote": false
}

For app or config rollouts, set kind to app or config and pass an action: {"command_type": "config.set", "payload": {...}, "rollback_command_type": "config.set", "rollback_payload": {...}}.

Rollout states are running, awaiting_promotion, paused, completed, rolling_back and rolled_back. Each stage in stage_views reports percent, target_nodes (cumulative), nodes (new in this stage), counts per node state, done_pct and a state: done, active, soaking, awaiting_promotion, paused, next, waiting or rolled_back.

Next Steps

• GitOps Drift — see which nodes differ from the configuration in Git
• Governance — fleet policies and compliance
• Fleet Orchestrator — ask “how is the rollout going?” in chat