Fleet Management

Fleet Management is the core feature of the Control Panel. It lets you register Cloud OS instances, monitor their health in real time, organize them with tags and groups, and view aggregated metrics across your entire fleet.

Instance Registration

Cloud OS instances connect to the Control Panel via an mTLS agent. The agent authenticates using a CA-signed certificate, ensuring that only trusted instances can join your fleet.

Registration Flow

1. Generate a registration token in the Control Panel under Fleet > Registration Tokens
2. On the Cloud OS instance, configure the agent with the token and the Control Panel’s agent server address (port 8443)
3. The agent presents the token to the Control Panel
4. The Control Panel validates the token, issues a signed certificate, and registers the instance
5. The instance appears on the Fleet Dashboard as a new server

┌──────────────┐   1. Token    ┌──────────────────┐
│  Control     │◄──────────────│  Cloud OS        │
│  Panel       │               │  Instance        │
│              │  2. mTLS cert │                  │
│  (CA)        │──────────────►│  (Agent)         │
│              │               │                  │
│              │  3. Heartbeat │                  │
│              │◄──────────────│                  │
└──────────────┘   (ongoing)   └──────────────────┘

Registration Tokens

Registration tokens are one-time or multi-use tokens that authorize new instances to join the fleet. You can manage tokens from the Registration Tokens page:

• Create a new token with an optional expiry date and usage limit
• Revoke a token to prevent further registrations
• View token usage history

Client Certificates and Automatic Renewal

The certificate a node receives at registration is valid for 90 days. Nodes renew it on their own: once less than 30 days are left, the agent generates a new key, asks the Control Panel for a fresh certificate over its existing mTLS connection (POST /api/agent/v1/renew) and switches to it without a restart or any action from you. A failed attempt (for example while the Control Panel is unreachable) is retried with increasing delays on later heartbeats, and the current certificate keeps working in the meantime. The node’s certificate expiry is shown on its page in the Control Panel.

A certificate is only renewed for a node that still exists in your fleet: after you delete a node, it drops off at the latest when its current certificate runs out.

Recovering a Node Whose Certificate Expired

A node that was switched off (or ran an older Cloud OS version without automatic renewal) past its certificate’s expiry can no longer connect — the Control Panel rejects it with tls: expired certificate and the node stays Offline. On the node, Settings → Control Center shows a red “Client certificate no longer trusted by Control Panel” banner. To bring it back:

1. In the Control Panel open Fleet → Registration tokens and generate a token.
2. Paste it into the token field inside the red banner and click Re-register — no need to disconnect first. Alternatively run quazzar enroll --cc <your Control Panel URL> --token <token> on the node.

The node re-registers under its existing identity: it keeps its place in the fleet, its tags, group and history, and does not count as an additional node against your plan’s limit. A token from a different workspace cannot take over a node that is registered in yours — that attempt is refused.

The same banner appears when the Control Panel’s CA was rotated; see CA rotation.

Heartbeat Monitoring

Once registered, each instance sends periodic heartbeat signals to the Control Panel. The heartbeat includes system metrics such as CPU usage, memory usage, disk utilization, and uptime.

Node Health: Online, Degraded, Offline

Every screen of the Control Panel (fleet list, dashboard, heatmap, node page, cluster map) uses the same three states:

The fleet table shows why a node is degraded (for example “CPU high” or “app failed”), so you can go straight to the cause.

Fleet Dashboard

The Dashboard is the landing page after login. It shows:

• Headline — number of nodes and the share that is fully healthy (online).
• KPIs — Online, Degraded, Offline, Tenants (your MSP clients, or your workspace), open incidents with the worst severity, and how many nodes already run the newest Cloud OS version. Each number shows its change compared with 24 hours ago (the Control Panel keeps an hourly snapshot of your fleet; deltas appear after the first day).
• Fleet CPU — the median (p50) and 95th percentile (p95) CPU across nodes over the last 24 hours.
• Status donut — online / degraded / offline split.
• Top by CPU — the busiest connected nodes right now.
• Recent alerts — the latest fleet alerts and incidents with their severity and alert rule, and a badge with the number of unacknowledged alerts.
• Regions — node count per region with its health split.
• Resource usage, plan usage and quick actions — fleet averages, uptime and alerts over 7 days, your plan’s node limit, and shortcuts to enroll a node, the fleet, operations and clusters.

Every KPI is clickable: Online / Degraded / Offline open the Fleet list filtered by that status, Open incidents opens Reliability, and the newest-version cell lists the nodes already on it. Roll out starts a rollout of the newest version in Operations. The chart range can be switched between 1 h, 6 h, 24 h and 7 d, and fleet alerts can be acknowledged right from the list.

The dashboard refreshes about every 20 seconds.

Regions

Every node belongs to a coarse region such as eu-west, eu-central, us-east, us-west, ap-south or sa-east. The region is chosen in this order:

1. Set by you — open the node and pick a region, or type a provider code such as eu-west-3 (it is kept as the zone and shown next to the region). A region you set is never overwritten automatically; clear it to go back to automatic detection.
2. Reported by the node — nodes on AWS, Google Cloud, Azure, DigitalOcean and Hetzner Cloud detect their region from the cloud provider. On other hardware you can pin it on the node with the QUAZZAR_REGION (and optionally QUAZZAR_ZONE) environment variable, for example QUAZZAR_REGION=home-lab.
3. Guessed from the location — otherwise the region is derived from the node’s detected city and country.

Nodes without any of these appear under unknown.

Instance List

The Fleet page lists every node with its host name, IP address, region and zone, tenant (for MSP workspaces), CPU / memory / disk usage, Cloud OS version and uptime. The list is paged on the server, so it stays fast with thousands of nodes.

Filtering, Sorting and Search

• Search by host name, IP address, tag or location.
• Status chips — All, Online, Degraded, Offline and Pending (enrolled but never connected; also counted in Offline), with counts.
• Region chips, group, Cloud OS version and, in MSP workspaces, tenant (client).
• Sort by any column (host, IP, region, tenant, CPU, memory, disk, version, uptime, last seen). Versions sort numerically, so 3.10 comes after 3.9; nodes that never connected sort last by last seen.
• Columns — the Columns button shows or hides the optional Last seen (on by default), Location and Tags columns. The choice is remembered in your browser.

The counts on the chips always reflect your search, group and tenant, but not the chip you clicked, so you can hop between them.

Bulk Actions

Select nodes (or use the current filter) to restart an app, update Cloud OS, add or remove tags, move nodes to a group, remove them, or run a command — in one go. When a whole page is selected, Select all N matching extends the action to every node that matches the current filters. Each bulk action becomes a job with live progress; the confirmation links straight to the job. See Fleet Operations.

The ⋯ menu on each row opens the node, edits its group, location, base URL and tags, sets its region, links it to an MSP client, or removes it. Filters live in the page URL, so a filtered view can be bookmarked or shared.

Tags and Groups

Organize your fleet with tags and groups to make filtering and bulk operations easier.

Tags

Tags are key-value pairs assigned to instances. Common tagging strategies include:

You can assign tags when registering an instance or add them later from the server detail page.

Groups

Groups are named collections of instances. Unlike tags, a group is a single label that you can use to target bulk operations. For example, a group called “database-servers” might contain all instances running PostgreSQL.

Metrics Aggregation

The Control Panel aggregates metrics from all registered instances and provides fleet-wide views:

• CPU utilization — average and peak CPU usage across the fleet
• Memory usage — total and per-instance memory consumption
• Disk usage — storage utilization with capacity planning indicators
• Network throughput — aggregate inbound and outbound traffic

These metrics are available on the Fleet Dashboard and can be filtered by tag or group.

Heatmap

The Heatmap shows one cell per node, one row per region, colored by CPU, memory or disk usage (switch with the tabs). Offline nodes are marked, each row shows its node count and average, and in MSP workspaces hovering a cell shows the client. Click a cell to open the node.

Server Detail

Click any node in the Fleet list (or on the Dashboard or Heatmap) to open its detail page. The header shows the node’s status, address, region, tenant and Cloud OS version, with Restart (pick an app), Update, Open in node, Open terminal and a ⋯ menu (install an app, edit info, set region, link to a client, remove). The page has six tabs — Overview, Apps, Metrics, Terminal, Logs and Settings — and includes:

• System info — hostname, OS, kernel, CPU model and cores, RAM, disk size, uptime, Cloud OS agent version, IP addresses, region and zone, tenant, group, cluster and role, last heartbeat, certificate expiry and license plan
• Live metrics — CPU, memory, disk and incoming network sparklines for the last hour
• Apps on this node — every app with its state and, when the node reports it, its CPU and memory use
• Logs — realtime logs from the node (see below)
• Historical charts — area charts showing CPU, Memory, Disk, and Network over time (see below)
• Apps — every app with its image, version, CPU, memory and restart count; restart an app or install a new one from a template
• Settings — tags, the base URL used by Open in node, region, MSP client, group and location, and removing the node

Historical Metric Charts

The node detail page shows area charts for CPU, Memory, Disk I/O, and Network traffic. Use the time-range pills to zoom in or out:

A background janitor rolls raw data into the instance_metrics_1h and instance_metrics_1d rollup tables once per hour and prunes data outside the retention window automatically. No manual configuration is required.

Realtime Logs

The Logs tab streams the node’s logs live, without opening a shell:

• System logs — the Cloud OS service (quazzard) by default, or any systemd units you pick (for example caddy.service, docker.service).
• App logs — the output of one app’s container.
• Filter by minimum level (debug, info, warn, error), start time (for example “last 15 minutes”), number of lines to load first, and a text search. Turn Follow off to load a snapshot instead of a live tail. Pause freezes the view while new lines keep arriving in the background (Resume shows them), and Clear empties it.

Logs travel over the node’s existing secure connection to the Control Panel, so they work even when the node has no public address. The node only reads logs; nothing is changed. Streaming logs requires the same permission as the web terminal (fleet write), and every stream is recorded in the audit log. A node must be connected (agent online) for its logs to be available.

If a system-log stream fails, the error now shows journalctl’s own message. “No journal files were opened due to insufficient permissions” means the node’s service user cannot read the journal: Cloud OS 0.12.1 and later add it to the systemd-journal group on install or upgrade (and restart the service once). On an older node, run sudo usermod -aG systemd-journal quazzar && sudo systemctl restart quazzar.

Terminal

The Terminal tab opens a shell on the node right in the browser. The session travels over the node’s existing secure agent connection to the Control Panel, so it works even when the node has no public address.

• Runs as root: the shell is a root login shell on the host (home /root, a clean environment, the host’s real filesystem), so you can run apt, systemctl, docker and edit files under /etc without sudo. It runs in its own systemd scope, so a Cloud OS restart doesn’t kill what you started. Each open carries a one-time ticket that the Control Panel signs; the node checks it against the Control Panel key it pinned (sudo quazzar-cli admin cp-trust show on the node). The node’s local terminal is deliberately limited, so this tab is the way to get root. An older Control Panel, or a node that refuses the ticket, opens a limited quazzar shell with a banner explaining why.

• Only through this tab: terminal, log-stream, chat, search and family commands can’t be queued as fleet commands (the API, bulk Run command, workflow command steps and rollouts refuse them), so the owner/admin check, the one-time ticket and the audit below always apply. Update the Control Panel before its nodes: an older Control Panel can still queue a terminal command without a ticket. Current nodes answer such a request with the limited quazzar shell (root needs a valid ticket), but nodes on Cloud OS 0.12.1 and earlier still answer it with a root shell.

• Who can use it: workspace owners and admins (a shell is full control of the node). Managers and viewers see an explanation instead. Staff sessions opened from the admin console (“Open as customer”) can’t open a shell.

• The node decides too: each node has a Remote shell from the Control Center switch (Cloud OS Settings → Control Center, on by default for enrolled nodes). When a node’s admin turns it off, the tab says “Remote shell is turned off on this node” and running sessions end at once.

• Limits: a session closes after 15 minutes without activity and after 2 hours at most (self-hosted Control Panels can change this with CC_TERMINAL_IDLE_TIMEOUT and CC_TERMINAL_MAX_SESSION), and at once if the node’s agent disconnects. A node runs at most four remote shells at a time.

• Audit: issuing the connection ticket, the start and the end of every session (with its duration and reason) are recorded in the audit log.

• Recording: if the workspace records terminal sessions (Settings → General, see Workspace Settings), every session is stored as a transcript, and owners see Recorded sessions under the terminal to replay or download each one.

• Reconnect: if the connection drops mid-session the terminal reconnects by itself (up to three times, each time with a new shell); Reconnect starts over after a session ended.

If the node’s agent is offline, the tab says so and offers Try again; Open in node still gives you the terminal in the node’s own interface.

Open in Node (Impersonate)

The Open in node button on the node detail page opens an authenticated Cloud OS session in a new browser tab — without entering a password.

How it works:

1. Click Open in node on any fleet node’s detail page.
2. The Control Panel mints a short-lived, HMAC-signed token bound to that node.
3. The Cloud OS instance validates the token via a secure backend endpoint, creates a session, and redirects you to the OS dashboard.
4. The token is single-use (JTI replay defense) and tied to the specific node (audience pinning) — it cannot be replayed or used against a different node.

Fleet Health

The fleet health section provides an at-a-glance summary of your fleet’s operational state:

• Health score — percentage of instances that are online
• Alerts — instances that have transitioned to warning or offline status
• Trends — historical uptime data showing fleet reliability over time

Next Steps

• Fleet Operations — issue commands to your registered instances
• Governance — define and enforce policies across your fleet
• Billing — manage your subscription and invoices