Reliability
Reliability in the Control Panel sidebar groups everything that tells you whether your fleet is healthy and what to do when it isn’t. It has four tabs: Incidents, SLOs, Alerts and Probes. The red number next to the entry is the count of open incidents.
Incidents
The top of the page shows up to four SLOs as rings. The ring fills with the error budget that is left, and turns amber below 30 % and red once the budget is gone. Click a ring to open that SLO.
Below the rings:
- Incident list — severity (SEV1 critical, SEV2 high, SEV3 moderate, SEV4 low), title, scope (region, cluster or node), assignee and status. Switch between Open, Resolved and All.
- Selected incident — the timeline (alert fired, assignment, notes, acknowledgement, merges, resolution), plus the actions:
- Acknowledge a firing incident, or Resolve it.
- Ask AI orchestrator opens the Fleet Orchestrator chat with the incident, its timeline and the affected nodes already loaded. A new chat suggests a first question; clicking again later returns to the same chat.
- Assignee — pick a team member.
- Add a note — appears on the timeline (Ctrl/⌘ + Enter to add).
- Merge — tick two or more incidents and choose Merge into…. The primary keeps all timeline events; the others are folded into it and resolved.
Links such as /observability/incidents?id=<incident> (used by notifications and the ⌘K search) open that incident directly.
SLOs
An SLO is an availability objective (for example 99.9 %) over a 7, 30 or 90 day window, measured from a probe. Create one with New SLO. Selecting an SLO shows the error budget left as a gauge, the achieved availability, the 1-hour burn rate (1× spends the budget exactly over the window; above 1× it runs out early), the window start and the number of hourly buckets.
Alerts
Alert rules watch live node metrics every minute. When a rule matches, it opens (or joins) an incident and notifies its channels.
| Field | Meaning |
|---|---|
| Scope | Each node (cpu_pct > 90) or Whole fleet (count(disk_pct > 85) / total > 0.1) |
| Expression | A metric (cpu_pct, ram_pct, disk_pct, heartbeat_age_seconds), an operator (>, >=, <, <=, ==) and a number |
| Severity | SEV1–SEV4 |
| Cooldown | Minimum time in seconds before the rule fires again for the same resource |
| Channels | Comma-separated, for example slack:#ops (empty = Control Panel log only) |
| Enabled | Disabled rules are kept but not evaluated |
The form checks the expression before saving. Rules can be edited, switched on or off from the list, and deleted. The Firing card lists alerts that haven’t been acknowledged; Ack silences one until it fires again. Turn on Show acknowledged to see them all.
Probes
A probe is an HTTP check that the Control Panel runs on a schedule: method, URL, expected status, optional text the body must contain, interval (at least 30 seconds) and timeout. Probes can be edited, paused and deleted. Selecting a probe shows its last 25 results with the success count, average latency and a latency sparkline.
API
| Endpoint | Method | Description |
|---|---|---|
/api/incidents?status=open (or resolved, all) | GET | List incidents |
/api/incidents/{id} | GET | Incident with its timeline |
/api/incidents/{id}/acknowledge · /resolve · /note | POST | Change status, add a note ({"text"}) |
/api/incidents/{id}/assignee | PATCH | {"assignee_id"} (empty clears) |
/api/incidents/merge | POST | {"primary_id", "merged_ids"} |
/api/incidents/{id}/ask-ai | POST | Open the incident’s orchestrator chat |
/api/slos, /api/slos/{id}/status | GET | SLOs and their burn status |
/api/slos · /api/slos/{id} | POST · DELETE | Create / delete |
/api/fleet/alerts/rules · /{id} | GET, POST · PUT, DELETE | Alert rules |
/api/fleet/alerts · /api/fleet/alerts/{id}/ack | GET · PUT | Fired alerts, acknowledge |
/api/synthetic/probes · /{id} · /{id}/results?n=25 | GET, POST · PUT, DELETE · GET | Probes and results |
Changes need the fleet:write permission; read-only roles see everything but can’t act.