Security Hardening

This guide covers key security practices for running Quazzar Cloud OS in production.

Web Application Firewall (WAF)

Cloud OS includes a built-in WAF powered by Coraza, compatible with the OWASP Core Rule Set. The WAF inspects HTTP requests and blocks common attacks like SQL injection, XSS, and path traversal.

• Detect mode — logs threats without blocking. Use this initially to identify false positives.
• Block mode — actively blocks matching requests. Recommended for production.

Enable and configure the WAF from the Security section of the Cloud OS dashboard.

Firewall

Restrict network access to only required ports:

Use your firewall tool (UFW, iptables, cloud security groups) to deny all other incoming traffic.

TLS / HTTPS

When the built-in Caddy reverse proxy is enabled, Cloud OS automatically provisions Let’s Encrypt certificates with strong TLS defaults (TLS 1.2+, HSTS headers, automatic renewal).

For custom certificates (e.g., internal CA), place them in the Cloud OS certificates directory. Cloud OS will use them instead of provisioning from Let’s Encrypt.

API Authentication

Every Cloud OS API route requires authentication — a session (the access_token cookie the web app uses, or Authorization: Bearer <token>), an API key (X-API-Key), the paired Quazzar Mobile app (through its pair tunnel), or the internal service token. One gate checks all of them, so every route that accepts a browser session also accepts an API key or the paired phone — with the role rules below. WebSocket endpoints (/ws, terminals, container exec, app logs) authenticate the same way; browsers send the session cookie with the upgrade. Unauthenticated requests get 401.

System-level operations additionally require the admin role (403 otherwise):

• Admin only: backups (including restore and backup targets), the audit log, user management and the activity log, SSO provider management, the host terminal, in-app terminals and file managers, container exec, the host File Manager (/api/files/* — every route, including listing, reading, previews and sudo elevation), the Server Mode actuators (/api/sysctl/*), the crontab editor (/api/cron), the Database Manager (/api/dbman/*, reads included), container inspect (GET /api/containers/{id} — it returns environment variables, the command line and mounts), every log read (/api/apps/{id}/logs[/search], /api/containers/{id}/logs, /api/logs/* including the system journal, /api/log-forwarders, and the public-tunnel log stream GET /ws/tunnels/{id}), mail administration (/api/mail/relay, /api/mail/stack, /api/mail/mailboxes, reads included), and the Family supervisor routes (see Family below).
• Admin for changes (reads stay open to signed-in users): apps (install, uninstall, start, stop, restart), domains, system services (POST /api/system/services/{name}/action), updates (apply, rollback, channel, schedule), Docker cleanup, security scans, alerting rules and channels, maintenance windows, plugins, OS packages, WAF / compliance / SIEM / air-gap, Control Center registration, marketplace installs and uninstalls (any marketplace write with an install, uninstall, reinstall, remove or import path segment) and approving marketplace submissions, AI workflows (/api/ai/workflows — create, edit, delete, run, schedule), the Server Mode container manager (/api/containers — run, compose deploy, start/stop, images, registries; the container list stays open, it has no environment), disks (mount / unmount), storage pools and snapshots, network settings (DNS, hostname, interfaces, Wi-Fi), reverse-proxy routes (/api/proxy), public tunnels, object storage, the migration agent (/api/agent/v1), and Plex (install, control, uninstall).
• AI administration (admin only): installing, starting, stopping or removing Ollama and LocalAI; pulling and deleting models; changing AI providers, routing rules and the AI budget; adding, removing, connecting or disconnecting MCP servers and calling MCP tools (/api/ai/mcp/* writes); fine-tune jobs, datasets and evaluations (/api/ai/finetune/* writes); installing native voice engines (POST /api/voice/install/{engine}); creating, installing, editing, enabling, scheduling or removing skills; importing agent bundles; installing apps through the import wizard (/api/import/compose/install, /api/import/{casaos,umbrel}/import); and changing the node-wide AI memory store or learned agent knowledge (POST /api/ai/memory, DELETE /api/ai/memory/{id}, DELETE /api/ai/agents/{id}/knowledge/{kid}). Chat, conversations, feedback, reading memory and agent knowledge, model and provider lists, running an installed Docker-sandboxed or instruction-only skill, and compose previews stay open to every signed-in user.

A paired phone reaches the same /api routes as a browser session (with the same role checks; the /api/auth/* account endpoints such as password change and passkey management still need a browser session, apart from QR approval) and acts with its owner’s current role: the role is re-read from the user record on every request, so demoting a user takes effect on their phone immediately and deleting the user cuts the phone off (401).

Every install path counts against the plan’s app limit — the App Store, first-run setup, Molly’s install tools and installs pushed from the Control Panel. On the Community plan the eleventh install is refused (402 upgrade_required in the API, an upgrade_required: … result for Control Panel commands).

First-run setup (POST /api/auth/setup) creates at most one admin even if several setup requests arrive at the same moment; the others get 409.

Only these endpoints are reachable without a session. Each one either carries no private data or checks its own credential:

The cluster-internal workflow endpoint (POST /api/internal/workflow/execute-step) only accepts connections whose TCP peer is on the WireGuard cluster network (10.42.0.0/16; X-Forwarded-For / X-Real-IP headers are ignored for that check) and that carry the primary’s signature made with the receiving node’s own cluster credential (X-Quazzar-Node-Signature, see the API reference). A request from another machine on the cluster network without a valid, fresh, never-seen signature gets 401.

Signing out revokes the session on the server: the old refresh token can no longer mint access tokens. An access token that was already issued stays valid until it expires (at most 15 minutes), so treat a leaked access token as live for that window.

Who can run what on the server

Anything that executes on the host, or changes the system, needs the admin role no matter which way it is reached:

When a non-admin asks Molly or the AI chat for an admin action, nothing runs; the assistant receives a tool result starting with Permission denied: and tells the user an administrator has to do it. A turn’s role is the lower of the session’s role and the user’s current role, so demoting a user takes effect on their next message, and conversations started from a messaging channel (Telegram, …) use the linked user’s current role. Scheduled skills run with the current role of the user who created the schedule — a schedule left behind by a demoted or deleted user can no longer run host commands.

Workflows work the same way. Each one records the admin who created or last saved it (created_by). Whenever it runs — by hand, on its schedule, or with steps delegated to another cluster node — its http steps and node-targeted steps run only if that owner is still an admin. Workflows created before owners were recorded have no owner, so those steps are blocked until an admin opens and saves the workflow once. A blocked step fails with blocked: this workflow has no admin owner…, which is logged, sent as a step_blocked event and shown as the run’s error. In Workflows, click a run in Execution History to see its steps: blocked steps are marked blocked in amber, and a workflow without an owner shows a hint to open and save it as an admin. A run started by hand keeps going after the request that started it returns; each run is bounded by its own 6-hour timeout (it then fails with timed out after …) and can be cancelled from the run view.

The AI Hub memory store (/api/ai/memory) and each agent’s learned knowledge (/api/ai/agents/{id}/knowledge) are node-wide, not per user: memory entries are keyed by namespace and key with no owner and also feed the node’s MCP memory.* tools, and knowledge is learned from everyone’s feedback. Every signed-in user can read them; storing or deleting entries is admin only. (Molly’s memory is different: it is per user, see below.) The GraphQL API no longer has an appLogs query; use the admin-only REST log routes.

Conversations are private

AI Hub conversations (/api/ai/conversations) and Molly conversations (/api/molly/conversations) belong to the user who started them. Each user’s list shows only their own conversations. Reading, continuing (/stream), compacting, renaming, deleting or pricing (/cost) someone else’s conversation answers 404, the same as a conversation that does not exist. Admins get no exception. No admin feature reads other users’ chats. Molly memory entries can only be deleted by their owner. AI Hub conversations created before owners were recorded belong to the node’s first (oldest) admin.

Family: supervisor = admin

On a Family-managed node the supervisor is the node’s admin. Monitoring toggles, policies, activity events, alerts, the daily summary, the daemon heartbeat, live screen view (start, stop and the stream), the enrolment instance id, pair codes, the hardening attestation (PUT) and the Family uninstall/wipe are admin only. Every signed-in user, including a monitored one, can read the transparency pages: GET /api/family/self (what is collected and how long it is kept), GET /api/family/managed-status, GET /api/family/hardening, and the public pre-sign-in banner. None of these return another user’s activity. The Control Panel reaches the supervisor routes over its authenticated agent tunnel.

Mail

Mail administration is admin only, reads included: the outbound relay (/api/mail/relay), the mail stack (/api/mail/stack), and the mailbox list and CRUD (/api/mail/mailboxes). Webmail (/api/mail/status, /provision, /folders, /inbox, /messages/…, /send) stays open to every signed-in user. It always resolves the mailbox from the signed-in user, so nobody can reach another user’s mailbox.

Container details and logs

GET /api/containers/{id} returns a container’s environment variables, command line and mounts, and container logs routinely repeat the same secrets (database passwords, API tokens). Container inspect and every log read are therefore admin only, rather than redacted: hiding environment values would still leave the command line and the logs. The container list (GET /api/containers: name, image, state, ports, labels) stays open.

API keys

An API key acts as the user who created it, with that user’s current role — a key created by a Viewer is a Viewer; demoting the owner demotes their keys immediately, and deleting the owner revokes them (401). A key’s scopes are enforced: * (the default) reaches every route the owner’s role allows, while a scoped key reaches only the matching /api areas — apps, backups, domains, monitoring, settings, users, alerts, vpn, security, updates — and gets 403 {"error":"api key scope does not cover this endpoint"} elsewhere (including key management, personal apps and WebSockets). API keys cannot approve a QR sign-in.

Rate limits and client IPs

Sign-in rate limits (password, 2FA, passkey, QR, approve-on-another-device, unlock) and the cluster-join limits count per client IP address. The client IP is the address the TCP connection comes from, without its port (reconnecting from a new port does not reset the count). An X-Forwarded-For / X-Real-IP header is honoured only when the connection comes from a trusted proxy — loopback by default, which covers the bundled Caddy — and ignored otherwise, so a client cannot pick a fresh identity per attempt. If a load balancer or reverse proxy on another host sits in front of Cloud OS, add it to server.trusted_proxies (see Configuration); otherwise every client shares that proxy’s limit. The same client IP is recorded in sessions and the audit log.

QR-code sign-in

1. The signed-out device calls POST /api/auth/qr/generate and receives a session id and a one-time secret. Only the session id goes into the QR code; the secret never leaves that device.
2. A signed-in device (a browser session or the paired Quazzar Mobile app) scans the code and approves it with POST /api/auth/qr/approve. Approval leaves at most 60 seconds to finish sign-in.
3. The signed-out device polls GET /api/auth/qr/status/{id} — it only ever returns a status, never a token.
4. It then calls POST /api/auth/qr/exchange with the session id and the secret, and receives a normal session (the same cookies as a password sign-in).

Someone who photographs the QR code learns only the session id, which is useless without the secret. A code is valid for 5 minutes, can be approved once and exchanged once. API keys cannot approve a QR sign-in.

Narrowed quazzar privileges

The quazzar service account no longer has broad sudo rights. The package installs /etc/sudoers.d/quazzar (replaced on upgrade; don’t edit it) with exactly these rules, all NOPASSWD, (root), with !requiretty and a fixed secure_path of /usr/sbin:/usr/bin:/sbin:/bin:

List the operations with sudo quazzar-setup priv list. The helper validates every argument: mount points below /mnt or /media (always nosuid,nodev), hostnames as FQDN labels of at most 64 characters, purge of Plex limited to /var/lib/plexmediaserver, protected system directories that can’t be deleted or moved, and no --allow-unauthenticated in updates. The systemd unit’s ReadWritePaths no longer includes /etc/apt/sources.list.d; APT repository files are written only by the helper.

Administrator confirmation

Root-owned files in the File Manager, adding or removing custom APT sources on the Packages page, and hooks of native apps outside the built-in catalogue need the password of root or of a sudo / admin / wheel user, checked with unix_chkpwd nonull (an empty password never works). The confirmation lasts 15 minutes, is bound to the browser session and can’t be obtained with an API key. It is limited to 5 attempts per minute per user (shared with “read with password”), and the helper waits 2 seconds after a wrong password. Installing a native app template that isn’t built in currently shows “administrator password required” from the UI.

Control Center root terminal and CP trust

The root terminal needs a ticket signed with the Control Panel’s ed25519 license key. Tickets are domain-separated, last 90 seconds (120 at most), are bound to the node’s instance id and work once. Only the Quazzar service process, running in the quazzar.service cgroup, can redeem a ticket or pin the key. Each root grant is logged to the journal (tag quazzar-setup, auth.notice) with the signed actor, session and nonce.

The node pins the Control Panel key on trust-on-first-use: the first license it receives live from the Control Panel sets the key. Check it with sudo quazzar-cli admin cp-trust show. If the Control Panel’s key changes (for example a rebuilt Control Panel), run sudo quazzar-cli admin cp-trust reset; the next live license pins the new key. With an old Control Panel, a missing ticket or a refusal, the terminal opens a limited quazzar shell with a banner explaining why.

Known remaining exposure

This narrows what a shell or script running as quazzar can do. It does not make the service itself unprivileged. Plainly:

1. quazzard is still in the docker group, so a compromise of the Quazzar service itself is root-equivalent through the Docker socket. A filtering docker-socket proxy is planned for a later phase. The local terminal has no docker group, which is why it is limited.
2. Code that quazzard runs counts as quazzard. The AI agent’s shell tool, skills and app build steps run as the service. The local terminal owns /var/lib/quazzar as quazzar, so a user of the local terminal can influence the state those features read.
3. The first pin is automatic. Until a node has received its first live license, a process running as quazzar that could steer the service’s Control Center connection could get its own key trusted. A node without a Control Center stays in this state as long as nothing is pinned. Register the node with its Control Center early and check cp-trust show.
4. cgroup limits. On hosts with cgroup v1 only, or where sudo’s PAM moves processes into a session scope (possibly RHEL), the Control Center root terminal falls back to the limited shell.
5. RPM systems lose in-app system upgrades and package checks (they show “unavailable”) until the helper supports dnf. Installing Caddy from apt isn’t supported either; Caddy ships inside the package.
6. Updates report coarser progress than before.

Session signing secret

Session tokens are signed with a per-node random secret kept in /var/lib/quazzar/jwt-secret (mode 0600); the Control Center keeps its own in $CC_DATA_DIR/jwt-secret (/var/lib/quazzar-cp/jwt-secret under the packaged unit). Anyone who knows the secret can forge a session for any user, including admins, so treat the file like a private key.

• QUAZZAR_JWT_SECRET / CC_JWT_SECRET are only needed to pin a secret. Use at least 32 random characters (openssl rand -hex 32).
• Node: a placeholder (CHANGE_ME, changeme, change-me, secret, <64-byte hex string>), an empty value or anything shorter than 32 characters is ignored: the node logs an error and uses the persisted random secret.
• Control Center: unset or a placeholder uses the persisted secret. A value you pinned that is shorter than 32 characters, or a data directory (CC_DATA_DIR, systemd StateDirectory=quazzar-cp) it cannot read or write, stops the Control Center with a message saying what to fix. It never runs on a throwaway key.
• The Control Center audit log is chained with CC_AUDIT_HMAC_KEY, else a strong CC_JWT_SECRET, else $CC_DATA_DIR/audit-hmac-key; never with a placeholder. A workspace whose log was written with a placeholder key gets one audit.rekeyed entry, and only if it was listed, with its newest entry, in $CC_DATA_DIR/audit-placeholder-pending (created just before audit-hmac-key; the list is taken on the first start that runs the migration) or audit-placeholder-migrated (CC_AUDIT_MIGRATE_PLACEHOLDER=1), and that newest entry is unchanged; verification starts there and counts older entries as unverifiable. The entry records the entry before it and how many entries precede it, so a second, copied or moved audit.rekeyed entry, or an entry added before it, deleted before it or put in place of the one right before it, is reported as tampering; the content of older entries is not checked. The database alone can never trigger it: workspaces created later, or changed after the list was taken, are never anchored, so any other mismatch, including entries re-signed with a placeholder later, is reported as tampering.
• Rotating CC_JWT_SECRET changes the audit key too, unless CC_AUDIT_HMAC_KEY is set. Set CC_AUDIT_HMAC_KEY to the current CC_JWT_SECRET first, then rotate.
• Packages before this fix shipped an active CHANGE_ME line. Upgrades comment that exact line out; check with grep JWT_SECRET /etc/quazzar/quazzar.env /etc/quazzar/control-center.env.
• Changing the secret, including this automatic switch away from a placeholder, ends existing sessions. On a node, the browser’s refresh token usually signs people back in without a prompt; Control Center users are signed out and sign in again.

If the placeholder was ever active

A node or Control Center that was reachable while CHANGE_ME signed its sessions may have been accessed by someone who forged a session, including as an administrator. After upgrading, check it and remove anything you don’t recognise:

Node (Cloud OS)

1. Users and roles (Settings → Users): unknown accounts, and users promoted to Admin.
2. API keys and MCP keys: delete keys nobody on your team created.
3. Passkeys and authenticator apps on every admin account: remove devices you don’t recognise.
4. Sessions: sign out sessions you don’t recognise, or all of them.
5. Password resets: resets or password changes nobody asked for.
6. Sign-in history and timeline: sign-ins from unknown IP addresses, and changes nobody made (apps, backups, VPN, shares, settings).
7. Change the passwords of administrators, and of any account that looks wrong.

Control Center

1. Staff accounts (Admin console) and customer portal users: unknown accounts, raised roles.
2. Workspace members and roles, API tokens, security keys, SSO and SCIM settings.
3. Sessions: everyone was signed out by the upgrade; check that nobody unexpected signs in again.
4. Password and MFA resets nobody asked for.
5. Audit log: on the start that creates $CC_DATA_DIR/audit-hmac-key (the first start after leaving the placeholder) each workspace whose log was signed with the placeholder gets one audit.rekeyed entry, recording whether the old chain was still intact; a failed start is finished by the next one from audit-placeholder-pending. If the list cannot be taken (for example the database is unreachable), the Control Center does not start and nothing is stored, so the next start takes the whole list again; to start without migrating, delete audit-placeholder-pending (those workspaces then verify as tampered). If you replaced CHANGE_ME with a strong secret yourself, no key file is created: set CC_AUDIT_MIGRATE_PLACEHOLDER=1 for the upgrade start and remove it after a start logs placeholder migration complete. With a persistent CC_DATA_DIR, audit-placeholder-migrated makes later starts ignore the variable; without one (Helm without cpState.persistent) you must remove it yourself. To run it again, delete audit-placeholder-migrated and set the variable for one more start. Entries after it are verified with the new key. Entries before it were signed with a public key and could have been edited, so read them as untrusted even when the old chain was intact.
6. Change the changeme Postgres password in CC_DATABASE_URL if you still use it.

See Configuration → JWT signing secret.

Two-Factor Authentication

Enable TOTP-based two-factor authentication for the admin account from the Profile page. This is strongly recommended for any internet-facing instance.

Security Scanning

Cloud OS includes a built-in security scanner accessible from the dashboard. It checks:

• TLS configuration and certificate validity
• Security header presence (HSTS, CSP, X-Frame-Options)
• Authentication settings (2FA, password strength)
• Software update status
• Backup encryption status

The scanner produces a security score (0-100) with category breakdowns and remediation guidance for each finding.

Hardening Checklist

• Enable HTTPS (built-in proxy or external)
• Make sure no placeholder JWT secret is active: grep JWT_SECRET /etc/quazzar/*.env
• If a placeholder was ever active, work through If the placeholder was ever active
• Enable WAF in block mode
• Configure firewall to restrict ports
• Enable TOTP two-factor authentication
• Configure encrypted backups with off-site storage
• Run a security scan and address critical findings
• Restrict SSH access to management IPs
• If a proxy other than the bundled Caddy fronts Cloud OS, list it in server.trusted_proxies
• Check the pinned Control Center key with sudo quazzar-cli admin cp-trust show after the node’s first live license
• Give day-to-day users the Operator or Viewer role; keep Admin for people who manage the server
• Set log level to info in production

Tips

• Run security scans regularly and after configuration changes.
• Track the security score over time as a KPI.
• Subscribe to security finding webhooks for automated alerting.
• Keep Cloud OS updated for security patches.