AI Health
AI → Health (/ai-health) shows whether the AI side of every node works: that its LLM provider answers, that its Memory MCP store finds what it should, and whether its AI canary passes. It needs the fleet read permission; running an eval or changing the eval policy needs fleet write.
Where the data comes from
Every node collects a small AI health snapshot every 5 minutes and sends it with its heartbeat:
| Block | What the node reports |
|---|---|
| Provider | Whether an LLM provider is configured, and whether the default one answers a model listing (with the latency, or the error) |
| Memory MCP | Note count, active MCP keys, hybrid (semantic) or lexical-only search |
| Skills | Skills loaded and enabled |
| Assistant errors | The assistant’s error rate over a recent window (shown once the node reports it) |
| Canary | The latest AI canary run: pass, degraded or fail, with each check’s result, detail and latency (shown once the node runs a canary) |
Nodes on older Cloud OS versions don’t send a snapshot and show as No data. The Control Center keeps each new snapshot for 30 days.
Status
| Status | Meaning |
|---|---|
| Healthy | Provider reachable, canary passing (or none yet) |
| Degraded | The canary reports degraded, or the assistant fails 20 % or more of at least 5 recent requests |
| Failing | The provider is unreachable or the canary failed |
| No provider | No LLM provider is configured on the node |
| No data | The node never reported AI health |
Hover the status badge to see the reasons.
Alerts
The Control Center raises a critical fleet alert and a notification when:
- a node’s AI canary fails twice in a row. One failed run doesn’t alert, and a run counts once even though the node re-sends it on every heartbeat.
- a node’s LLM provider becomes unreachable.
When the node recovers (the next canary run passes, or the provider answers again), the alert is acknowledged automatically and a “recovered” notification says how long the problem lasted. The alerts are also listed under Reliability → Alerts as ai.canary_failed and ai.provider_unreachable.
The page
- KPI strip: healthy, degraded (and nodes without a provider), failing (and open alerts), providers reachable, canaries failing, and the last eval score (the mean score of each node’s latest successful eval). Click a cell to filter the table.
- Table: one row per node, with the provider and its latency, the canary result (passing checks out of total), Memory MCP notes, skills, the last eval and when the node last checked. Filter by status or search by hostname.
- Node drawer: click a row (or open
/ai-health?node=<id>from a notification) to see the reasons, open alerts, provider / memory / skills details, every canary check, the node’s eval runs and its check history over 24 hours, 7 days or 30 days.
Retrieval evals
An eval measures how well a node’s Memory MCP search finds notes. The node takes a fixed sample of its notes (30 by default) and looks each one up twice, through the same ranking it serves to MCP clients:
- title – the note’s title as the query (baseline recall);
- deep – a sentence from the body that is not in the title (whether search reaches facts buried in the content).
For each probe it reports recall@1/3/5/10 (how often the note is in the top k results) and MRR (mean reciprocal rank). The eval score is the mean MRR of the probes, from 0 to 1. The eval is read-only: it does not change note ranking signals.
Click Run eval on a row or in the drawer. The node runs it on its next command poll, usually within seconds. Possible results:
| Result | Meaning |
|---|---|
Score (e.g. 0.72) | The eval ran |
| Too few notes | The node has fewer than 5 notes with enough text |
| Unsupported | The node’s Cloud OS version can’t run evals |
| Error | The eval failed on the node (the error is shown) |
| Timed out | The node did not answer within 30 minutes |
Only one eval runs per node at a time.
Standing evals 🔒
Standing evals (top-right button) re-run the eval on a schedule on every online node with at least 5 notes. Set the interval (hours), the sample size, and an alert floor: when a standing eval scores below it, everyone in the workspace gets a “memory retrieval score dropped” notification. Standing evals need the Pro plan or above; viewing AI health and running evals by hand are free on every plan. See Orbit Pro and Billing & Licensing.