AI Resilience
Cloud OS includes three safety nets that keep the AI assistant trustworthy when something goes wrong: restart recovery for AI channels, duplicate-message protection, and a periodic AI health check (the canary). All three are on by default; each has an emergency kill-switch.
Channel restart recovery
When the Cloud OS service restarts (update, reboot, crash) while the AI is answering a message that came in through an AI channel, the answer is lost. Instead of leaving the user waiting, Cloud OS records every channel request while it is being answered — the channel, the chat or thread, the sender and the start time — and clears the record once the answer is sent.
On the next start, any record left behind by the previous run gets a short reply in the original conversation:
I was restarted while answering your last message, so the answer was lost. Please resend your question.
| Channel | Where the notice goes |
|---|---|
| Telegram | Reply to the interrupted message in the same chat |
| Slack | The same thread |
| Discord | The same channel |
| The same chat | |
| A reply to the sender with the same subject | |
| REST | No notice — the HTTP caller already received an error |
Bot channels reconnect asynchronously after a restart, so undelivered notices are retried about 15 seconds, 1 minute and 5 minutes after start. Requests older than 24 hours, or still undeliverable after three attempts, are dropped (and logged).
Duplicate-message protection
Messaging platforms occasionally deliver the same message twice — Telegram replays updates that were not confirmed before a restart, Slack retries events that were acknowledged slowly, the WhatsApp sidecar can resend. Cloud OS remembers the platform message ID of every message it has handed to the AI (per channel, for 72 hours) and silently ignores a repeat, so the AI never answers the same message twice. Button presses and slash-command interactions have no message ID and are never deduplicated.
AI health check (canary)
Every 4 hours the node runs a self-check of its AI stack and stores the report (the newest 20 are kept):
| Check | Passes when |
|---|---|
memory | A test memory can be written to Memory MCP, found by search, and deleted. It is stored under a private owner and namespace (_canary), never synced, and deleted even if the check fails. |
provider | The provider requests are routed to answers. In the default auto mode local providers get a tiny prompt, paid cloud providers only have their model list fetched — the check never spends tokens. With no provider configured, or with the monthly budget exhausted, the check is skipped. |
mcp | Both MCP servers (the built-in /api/mcp server and Memory MCP) list their tools, and every tool has a unique valid name, an object input schema and a registered handler. |
skills | The skill directories reload cleanly and every skill and skill tool validates. Broken skills make the run degraded (warn), not failed. |
The run status is ok, warn (degraded), fail or skip (nothing configured). Administrators see it under Molly > Settings > AI health check, which also has a Run now button, or through the API:
curl -H "Authorization: Bearer $TOKEN" https://node.example.com/api/ai/health/canary
curl -X POST -H "Authorization: Bearer $TOKEN" https://node.example.com/api/ai/health/canary/runControl Panel reporting
When the node is connected to a Control Panel, the canary reuses the existing node-to-CP channels:
- Alerts. When a run starts failing, the node forwards an alert with rule
ai_canaryand severitywarningwhose message names the failing checks. When it recovers, aninfoalert follows. A failure that persists is not re-sent on every run. If the Control Panel is unreachable when the state changes, the alert is sent on the next run instead of being lost. - Metrics. Every metrics push carries
ai.canary.ok(1healthy or degraded,0failing) andai.canary.failed_checks.
Environment variables
Set these in /etc/quazzar/quazzar.env and restart the service (sudo systemctl restart quazzar.service).
| Variable | Default | Effect |
|---|---|---|
QUAZZAR_AI_CHANNEL_RECOVERY | on | off stops recording in-flight channel requests and sending restart notices |
QUAZZAR_AI_CHANNEL_DEDUP | on | off answers redelivered channel messages again |
QUAZZAR_AI_CANARY | on | off stops the scheduled health check and disables Run now |
QUAZZAR_AI_CANARY_INTERVAL | 4h | Go duration between runs (30m, 12h, …); minimum 5m |
QUAZZAR_AI_CANARY_PROVIDER | auto | auto (prompt local providers, list models on paid ones), prompt (always send the tiny prompt — a few tokens per run on paid providers), list (model list only), off (skip the provider check) |
The kill-switches are for emergencies. With recovery off, users whose answer was lost in a restart get no reply at all; with dedup off, a replayed message is answered twice.
For developers: tool-registration guard
The same validation the mcp check runs is enforced in CI by internal/ai/toolguard: a Go test enumerates every tool Molly, the built-in agents, the bundled skills and both MCP servers register and fails the build on a duplicate name, an invalid input schema or a tool without a handler. LLM-facing tools must also satisfy the provider function-name rule (^[A-Za-z0-9_-]{1,64}$).