Live failure: `sudo ./setup.sh garage` → Full reinstall on a box that
already had a working Garage install crashed with:
Error: ApplyClusterLayout returned InternalError (500): Internal
error: Invalid new layout version
Root cause: a "Full reinstall" deliberately never wipes ./data or
./meta (that's real backup-mirror data — Kopia's sync-to s3 target —
and losing it silently on reinstall would be far worse than the
alternative), but the cluster-init step unconditionally re-ran `garage
layout assign` + `layout apply --version 1` every time it was reached.
Garage requires each apply to be exactly previous_version + 1; a node
that already has a committed layout (from the earlier install, still
sitting in the preserved ./meta) rejects a second "1". Fix: check
`garage status` for "NO ROLE ASSIGNED" first and only run the
assign/apply once, matching what the surrounding comment already
claimed happened ("Only ever run once") but the code didn't enforce.
Second, related issue this would have hit immediately after: the same
reused-./meta state almost always means an existing bucket + key from
the earlier install are still sitting in Garage's storage. The fresh
flow was about to silently create a brand-new bucket/key and overwrite
.env to point at those instead — orphaning any real data already in
the old bucket (nothing left on disk pointing at it, even though it's
still physically stored). Now: when the layout is already applied,
list existing buckets and require an explicit y/n (default n) before
creating new ones, with recovery instructions for reconnecting to an
existing bucket by hand instead.
Third, the actual reason a full reinstall was reached at all: Update
mode never backfills .env fields added to this script after someone's
initial install (GARAGE_S3_API_PORT, needed by services/backup.sh to
read an instance remotely) since Update deliberately never touches
.env otherwise — the only other path was the now-unsafe fresh
reinstall. Update now backfills just that missing key by reading the
real port back out of the already-written docker-compose.yml, so a
future .env schema addition doesn't force this tradeoff again.
Verified with standalone harnesses (not the live install, mocked
`garage status`/bucket-list output and .env/docker-compose.yml
fixtures): all four layout-state branches (fresh node, existing
buckets + decline, existing buckets + confirm, existing role but no
buckets), and both backfill cases (missing key added, existing key
left alone). Caught and fixed a real bug in the first draft of the
port-extraction regex during this testing — grep -oE '^[0-9]+' never
matched because the captured group still had its surrounding quotes.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01H4k6J1qXXyYxhGEgnJaMvn
Extends the "ADDITIONAL MIRROR" section (previously SFTP-only) with a
type choice: SFTP, or S3 against a Garage instance already running on
that box. For the S3 path, this script never asks the operator to retype
a bucket name or key — it SSHes to the destination, reads
~/docker/garage/.env directly (the real, currently-configured values,
generated once by services/garage.sh and never touched again on its own
Update runs), and uses those for the dry-run verification and the
persisted mirror args. If Garage isn't installed there yet, it says so
plainly with the exact install command instead of failing cryptically or
silently skipping.
Also removed the last hardcoded suggestions from services/garage.sh
itself ("kopia-backup" / "kopia" as fixed prompt defaults) — replaced
with a freshly-generated suggestion each run (timestamp-suffixed), so
nothing about the bucket/key name is a fixed string baked into this repo
at any point in the chain; it's always the operator's actual choice, read
back live wherever it's needed.
Verified end-to-end against a mocked ssh (returning realistic
~/docker/garage/.env content) covering both outcomes: Garage installed
with a real bucket/key correctly parsed, dry-run run, and persisted; and
Garage missing, correctly warning with the install command and leaving
backup.conf untouched either way.
Garage's real CLI output pads labels with extra spaces for column
alignment ("Key ID: GKxxxx"), not a single space like the
mocked test used ("Key ID: GKxxxx") — the fixed ": " field separator left
that padding stuck to the parsed value, so .env ended up with access
key/secret strings carrying leading whitespace inside the quotes.
Confirmed live by the user right after install. This would have broken S3
auth outright once actually used, since access keys have to match exactly.
Switched to ':[[:space:]]+' as a regex field separator, which consumes
however many spaces are actually there instead of assuming exactly one.
Verified against both the single-space and padded/aligned formats — both
now produce the identical clean value with no leading whitespace.
MinIO's open-source community edition is dead: console GUI stripped May
2025, Docker images stopped publishing October 2025, repo formally
archived April 2026, with MinIO redirecting everyone to their paid AIStor
product. Verified this directly before building anything, since recommending
a since-abandoned image would have been worse than the SFTP problem this
was meant to solve.
Garage (Deuxfleurs) is the actively-maintained small-scale self-hosted
replacement — single Rust binary, purpose-built for exactly this "one
lightweight node" use case (as opposed to SeaweedFS, which targets large
object counts / large-scale deployments, more machinery than a single
backup-mirror target needs).
services/garage.sh follows this repo's standard service template: port
scanning for the S3 API/RPC/admin ports, an RPC secret generated once and
never touched again on Update, and a one-time cluster init sequence
(layout assign/apply, bucket create, key create, bucket allow) gated on
whether .env already has a saved access key — Update reruns skip all of it
and just refresh the image.
Primary intended use: a local S3-compatible target for services/backup.sh's
additional-mirror Kopia sync, so a local mirror can reuse the exact same
sync-to s3 code path already proven reliable for the Backblaze B2 mirror,
instead of Kopia's separate, less-exercised SFTP backend that's been the
source of today's connection troubleshooting.
Verified end-to-end against a mocked environment (fake docker exec
returning realistic `garage status`/`garage key create` output) — caught
and fixed a real off-by-one in the status-output parsing this way (grabbed
the column-header row's literal "ID" instead of the actual node ID; output
has a title line, then a header line, then the data row). Also validated
the generated docker-compose.yml with real `docker compose config` in both
the no-network and network-created cases.
Claude