Fleet Orchestrator
The fleet orchestrator is the chat panel of the AI section (Agents & chat in the sidebar). Ask about your fleet in plain language: “which nodes in eu-west are above 85 % disk?”, “how is the 3.2.1 rollout going?”, “what’s burning our error budget?”. The orchestrator looks up the answers with real tools instead of guessing, and shows each lookup in the chat as it runs, as a pill such as fleet.query(region=eu-west-3, disk_gt=85) ✓ (click a pill to see what it found).
Where to find it
- On a wide screen (1280 px or more) the panel is docked on the right of the Agents, Teams and Workflows tabs, so you can ask while you look at the lists. It continues your latest orchestrator conversation (or starts one with your first message); the expand button opens it full size.
- The Chat tab shows it full size, with your conversations on the left (new chat, rename, delete), the node and model to run on (or Auto-route when your plan includes it) and today’s message count against your plan’s daily limit.
/ai-chat?conversation=<id>opens a specific conversation.
The pill in the panel header shows what the conversation covers: scope: all nodes, or scope: incident for a chat opened from an incident.
Where the model runs
The chat uses the AI models your nodes already provide (for example a local Ollama model). The Control Panel sends the conversation to an online node that has a suitable model, or to the node and model you pick. Nothing is sent to a third party unless that node is set up to use one.
Tools
| Tool | What it answers |
|---|---|
fleet.query | Nodes matching a filter: status, region, group, version, tag, host name, disk / CPU / memory above a threshold |
fleet.node | One node in depth: status, version, live metrics, installed apps, open incidents |
incidents.list | Open (or all) incidents, by severity |
slo.status | Availability, remaining error budget and burn rate for each SLO |
rollouts.status | Staged rollouts with stage progress and health-gate values |
bulk.propose | Prepares a bulk action (update the OS, install or restart an app, set a config value) for the nodes matching a filter |
Every tool runs as you. It only sees your workspace, and the permissions of your role apply. Read-only users can ask questions but can’t create proposals.
Proposals need your confirmation
The orchestrator never changes anything by itself. bulk.propose only creates a proposal: the action, the matching nodes and a summary. The chat shows it as a card with the summary, the number of target nodes, a few sample host names, a countdown and Confirm / Reject buttons. Nothing runs until someone with fleet:write confirms it, and a proposal expires after 15 minutes. After confirmation the command goes out to each node, and the card shows the progress (succeeded, failed, running, pending).
Ask AI from an incident
Every incident (Reliability > Incidents) has an Ask AI orchestrator button. It opens an orchestrator chat that already knows the incident: title, severity, scope, timeline and the nodes it affects. The next time you click it on the same incident, you get the same conversation back.
Chat protocol
The browser talks to /api/ai/chat/ws, which is also served at /api/v1/ai/chat/ws. After a {"type": "chat.send", ...} message, the socket streams:
| Event | Fields |
|---|---|
chat.chunk | conversation_id, content: answer text (tool-call markup is filtered out) |
tool.call | conversation_id, call_id, name, args: a tool is starting |
tool | conversation_id, call_id, name, args, ok, result_summary, result, error (if it failed), and proposal (for bulk.propose) |
chat.done | conversation_id, message_id, model, instance_id, tokens_in, tokens_out, tool_calls |
chat.error | conversation_id, error |
API
| Endpoint | Method | Description |
|---|---|---|
/api/ai/chat/* and /api/v1/ai/chat/* | — | Conversations, models, usage, limits and the WebSocket (the two prefixes are equivalent) |
/api/ai/chat/proposals/{id} | GET | A proposal, with progress once it has run |
/api/ai/chat/proposals/{id}/confirm | POST | Run the proposal (fleet:write) |
/api/ai/chat/proposals/{id}/reject | POST | Discard the proposal |
/api/incidents/{id}/ask-ai | POST | Open (or reopen) the incident’s chat. Returns conversation_id, created, affected_nodes, timeline and suggested_prompt. |
The AI lists at /api/ai/agents, /api/ai/teams and /api/ai/workflows also show runs, success_rate (0–1, empty until a run finishes), model and status (active, paused or draft) for each row.
Next Steps
- AI Orchestration — run AI tasks across nodes
- Staged Rollouts