Object Storage

Cloud OS runs Garage  (v2.4.1) so the node ships with a ready-to-use S3-compatible endpoint. Use it as the destination for the built-in backup engine, as a static-site or CDN backend, or as a target for any tool that speaks the S3 API. Orbit Drive stores its files here too.

The page lives at Infra → Object Storage (/storage/objects). The storage is the optional S3 Object Storage component — install or remove it in Settings → Components (see Components).

What you get

All of them listen on the loopback only. Nothing is reachable from the network until you publish it through the reverse proxy.

• Exact size and object count per bucket, refreshed every 30 seconds.
• Access keys with read, write or owner permission per bucket, and one-click revoke.
• One-click Expose via Reverse Proxy for the S3 API.

Buckets

Click New bucket and pick:

• Private — clients must sign every request with an access key.
• Public — anyone can read the bucket’s objects through the website endpoint (read-only). The bucket card shows the public URL and the host name it answers to, for example http://127.0.0.1:3902 with host cdn.web.local. Use this for static assets, never for sensitive data.

To publish a public bucket on a real domain, name the bucket after the domain (for example cdn.example.com) and add a reverse-proxy route for that domain to http://127.0.0.1:3902. Garage serves a bucket for the host <bucket>.web.local and for the host equal to the bucket name.

The trash icon deletes a bucket. A bucket that still holds objects cannot be deleted — empty it first with any S3 client (for example aws s3 rm s3://photos --recursive).

The quazzar-drive bucket holds Orbit Drive’s files. Its card says “Used by Orbit Drive”: it has no delete button and cannot be ticked for an access key (the API answers objstore_bucket_reserved).

Access keys

Each access key is a credential pair you give to one app. When the app is retired, revoke its key without touching anyone else’s.

Click Create key, give it a memorable name (backup-agent, grafana-tempo, ci-pipeline), and tick what it may do in each bucket:

A key without any ticks can’t reach any bucket. Cloud OS then shows the secret exactly once. Copy it now — it can’t be shown again. If you lose it, revoke the key and create a new one.

IAM policy JSON is not supported any more; the API answers objstore_policy_unsupported when a request still sends one.

Endpoint info

The Endpoint panel shows the S3 API address, the website endpoint and the region (us-east-1; the S3 protocol needs some region). There is no web console — manage buckets and keys on this page.

Expose via Reverse Proxy

To reach the S3 API as s3.example.com:

1. Point DNS for the domain at this node.
2. Type the domain into the Endpoint panel and click Expose.
3. Cloud OS adds a route that forwards the domain to http://127.0.0.1:3900, with TLS from the proxy’s automatic Let’s Encrypt certificate.

If the Reverse Proxy Manager is not enabled you’ll see “Reverse Proxy is not enabled on this node” — enable it on Infra → Reverse Proxy first.

Connecting an external tool

AWS CLI:

aws configure set aws_access_key_id GK31c2f218a2e44f485b94239e
aws configure set aws_secret_access_key '<the secret you saved>'
aws --endpoint-url https://s3.example.com --region us-east-1 s3 ls

rclone (rclone config → s3 → provider Other): endpoint https://s3.example.com, region us-east-1, force_path_style = true.

Migrating from MinIO

Earlier versions ran MinIO. After updating to 0.12.1 a node that has MinIO data moves it to Garage by itself — there is nothing to start:

1. The package update installs Garage. Cloud OS checks that the Garage disk has at least 1.1 times the size of the MinIO data free. If not, the node keeps working on MinIO and you get a notification with the space needed; it checks again every hour. When MinIO and Garage share one disk, this means MinIO may use at most about 47% of it — see Not enough room on a shared disk.
2. Copy pass. Every bucket is copied to Garage in the background, four objects at a time, with its content type, metadata and visibility (public MinIO buckets become public Garage buckets). Drive keeps working on MinIO during the copy. Progress shows on the S3 card in Settings → Components (“Migrating to Garage: 1,240 / 3,100 objects”).
3. Catch-up pass. A second pass copies what changed during the first one and removes what was deleted.
4. Verify. Cloud OS compares every object key by key — presence, size and, for objects uploaded in one piece, the content checksum (ETag) — bucket by bucket.
5. Switch. Only when everything matches, Drive switches to Garage. The switch runs under Drive’s lock, so uploads pause for a moment; it gives up after 5 minutes and the node stays on MinIO. Because every such attempt holds up uploads, the next one waits longer after each timeout in a row: 1 hour, 2, 4 and so on, up to 24 hours. When it succeeds you get a notification.
6. MinIO keeps running for 6 more minutes, so downloads and presigned links started just before the switch still finish, and is then stopped.

While the move runs you can read buckets, but creating or deleting buckets and access keys is refused (objstore_migration_in_progress). If a step fails the node stays on MinIO and retries every hour; you get at most one failure notification a day. An interrupted move resumes where it stopped.

If Cloud OS cannot tell whether the move already happened, or MinIO fails to start, Orbit Drive is paused and the S3 card says so (objstore_migration_state_unknown, objstore_minio_unavailable). Quazzar retries on its own, and restarts itself once (at most once an hour) when the question is settled so that Drive comes back. A MinIO that does not start is retried every hour whether or not Garage is running, with at most one notification a day. Only when Cloud OS cannot tell whether the move happened and Garage is not running either is nothing retried before the next start (objstore_migration_state_unknown_next_start).

A node with MinIO buckets on disk always counts as “not moved yet”, even when the minio program or minio.env is gone. So does a MinIO data directory that Cloud OS cannot read (for example after a permissions change or a disk error). Drive then stays paused rather than showing an empty Garage in place of your files, and the S3 card names what is missing. Put it back (reinstalling the previous package restores both; minio.env may be in /etc/quazzar or /var/lib/quazzar) and Quazzar picks it up within the hour. Over existing MinIO data Quazzar never creates new MinIO root credentials.

Access keys are not migrated — MinIO never reveals existing secrets, so old keys stop working after the switch. The notification sent to the node owner lists the old key IDs; create new keys in Storage → Objects and update the apps that used them. Public bucket URLs change to the website endpoint (see Buckets above); update reverse-proxy routes that pointed at port 9000.

Deleting the old MinIO data

The old MinIO data stays on disk until an administrator deletes it. When the move is done, the S3 card in Settings → Components shows “Old MinIO data” with its size and a Delete button, or run:

sudo quazzar objstore purge-minio

Both remove /var/lib/quazzar/objects, minio.env (from /etc/quazzar and, if it lives there, /var/lib/quazzar) and, through quazzar-setup, /usr/local/bin/minio. The delete never crosses into other mounted filesystems. It is refused:

• before the move has finished (objstore_minio_purge_unavailable);
• within 6 minutes of the switch, while MinIO still serves earlier downloads (objstore_minio_purge_too_soon, with the minutes left);
• while Garage is not healthy (objstore_garage_unavailable), because it holds the only copy;
• when the running Garage is not the node the data was moved to (objstore_garage_node_changed);
• when a bucket that received objects in the move is now gone or empty in Garage (objstore_garage_data_mismatch, with the bucket and both counts). Fewer objects than right after the move is fine: deleting files in Orbit Drive deletes objects;
• when files in the MinIO buckets changed after the switch (objstore_minio_changed_after_switch, with the number of files) — see below;
• while a components apply is running, or Quazzar cannot tell whether one is (components_busy).

The Garage node and bucket checks protect the MinIO copy, so they are skipped when /var/lib/quazzar/objects is already gone: Delete then only removes minio.env and the minio program and records that the old data is gone.

Other S3 apps writing to MinIO during the switch

Drive switches to Garage in one step, but for 6 minutes afterwards MinIO keeps running so that downloads and presigned links issued just before the switch still work. An app that talks to MinIO directly (a backup tool with an old access key, a presigned upload link) can still write to MinIO in that time. Those writes are not moved to Garage. Quazzar notices them when you delete the old data: it refuses with objstore_minio_changed_after_switch and keeps everything.

To move them by hand:

1. Start MinIO once more in a terminal (Quazzar stopped it after the 6 minutes):

   sudo -u quazzar sh -c 'set -a; . /etc/quazzar/minio.env; exec minio server /var/lib/quazzar/objects --address 127.0.0.1:9000'

   (minio.env is in /var/lib/quazzar on nodes where /etc/quazzar is read-only.)

2. Copy the changed objects to Garage with any S3 client, for example rclone with one remote for http://127.0.0.1:9000 (the MINIO_ROOT_USER / MINIO_ROOT_PASSWORD from minio.env) and one for http://127.0.0.1:3900 (a key with write access to the bucket; for quazzar-drive use the root key in /etc/quazzar/garage.env), then rclone copy minio:photos garage:photos.

3. Stop MinIO with Ctrl+C.

4. The check still sees the changed files, so remove the old data yourself: sudo rm -rf /var/lib/quazzar/objects. Then click Delete (or run sudo quazzar objstore purge-minio): with the data gone it skips the Garage checks, removes minio.env and the minio program and records that the old data is gone.

When a moved bucket was emptied or deleted

Deleting files in Orbit Drive, or objects in any bucket, after the move does not block deleting the old data. Only a bucket that received objects in the move and is now empty or gone in Garage does (objstore_garage_data_mismatch): Quazzar cannot tell whether you emptied it or its data was lost. If you emptied or deleted it yourself, remove the old data as in step 4 above and finish with Delete.

Not enough room on a shared disk

The move needs free space of 1.1 times the MinIO data on the disk that holds /var/lib/quazzar/garage/data. On a node where MinIO and Garage share one disk, MinIO may therefore use at most about 47% of it; above that the node stays on MinIO and the S3 card shows objstore_not_enough_space. Free up space on the disk, or mount a larger disk at /var/lib/quazzar/garage/data (before the move starts, with Quazzar stopped), then wait for the hourly check or restart Quazzar.

Operational notes

• Where data lives. Object data is in /var/lib/quazzar/garage/data, metadata (SQLite) in /var/lib/quazzar/garage/meta. Back up both directories together while Garage is stopped, or use the backup engine against the S3 API.
• Configuration. /etc/quazzar/garage.toml (mode 0600, /var/lib/quazzar/garage.toml if /etc/quazzar is read-only) is written once on first start and never regenerated — it holds the RPC secret and the admin token. It turns on metadata_fsync, data_fsync and a metadata snapshot every 6 hours (metadata_auto_snapshot_interval = "6h", kept in /var/lib/quazzar/garage/meta). A file written by an earlier build gets exactly these missing settings added on the next start; nothing else in it changes, and a value you set yourself is kept. The internal root key Cloud OS uses is in /etc/quazzar/garage.env (0600); it is not shown in the access-key list and can’t be revoked.
• Process. Quazzar runs garage -c /etc/quazzar/garage.toml server itself and restarts it if it exits (1 s, doubling up to 30 s).
• Binary. The S3 component downloads the official static Garage v2.4.1 build to /usr/local/bin/garage and checks its SHA-256; a mismatch aborts the install and leaves nothing behind. Supported: x86-64 and ARM64.