Gitea previously only existed bundled inside the full ai-stack service
(Ollama/ComfyUI/InvokeAI/etc. all together) — no way to get just a git
server without the rest of that heavy stack. This adds it as its own
lightweight service, reusing the vendored gitea-github-sync.sh but not
any of ai-stack's other components.
- Auto-creates a Gitea admin account and API token via the container's
own CLI (no manual web setup wizard).
- Asks GitHub token, sync direction (GitHub->Gitea / Gitea->GitHub /
both), and whether to install a systemd timer for automatic sync —
prints manual commands instead if declined.
- Own systemd unit for the timer rather than the vendor script's
built-in --install-timer, since that always runs both directions
with no way to pin a single direction.
pstn_personal_ring dialed the owner and hung up regardless of
DIALSTATUS, so no-answer/busy calls to a personal DID just dropped
silently instead of offering voicemail. Now falls to
VoiceMail(<owner>@default,u) on anything but ANSWER, gated by the
owner's existing voicemail=yes/no flag in pstn-permissions.conf.
The shared ring-group and group-owned personal DID inbound paths have
the same gap but no single owning extension to pick a mailbox for —
left as-is pending a decision on what that should do.
AST_CONFIG() silently returns an empty string instead of erroring when
asterisk.conf's [options] section lacks live_dangerously = yes, so a
correct tier_out=full in pstn-permissions.conf still evaluates as no
permission — every outbound/ring-group call gets denied with nothing
in the logs pointing at the real cause. Easy Asterisk's vendor default
ships without this set. Now applied automatically on fresh install and
on every "update" rebuild, restarting only when the file actually
changes.
Asterisk DO->IONOS migration (resolved), web-based extension messaging
(not started), Pi-hole (done), VPS-as-VPN-endpoint with encrypted DNS
(idea stage). Temporary -- delete once these are finished or turned into
real issues/PRs.
restore replaces every file under the Asterisk directory with a fresh
extraction from the archive (rm -rf + mv in the new tree) -- every file is
a new inode, so any POSIX ACL grants the Security Dashboard holds on them
(_secdash_grant_asterisk_access's setfacl access to .env/config/logs/spool)
are gone, and were never on the new files to begin with. Confirmed live:
surfaces as "No permission to read .../.env" in the Extensions tab right
after a restore, on a box where the dashboard was already working fine.
The restore script can't fix this itself (runs standalone, no access to
services/*.sh's functions or the dashboard's service-user name), so it now
prints a reminder to re-run `sudo ./setup.sh security-dashboard` at the end
of a restore when the dashboard's systemd unit is present, plus a matching
note in the generated README.
Filters by IP/range, scenario, ASN, carrier name, or country in a single
free-text field -- asked for so a phone's current IP or its network/carrier
name can be searched directly instead of scanning the full ban list by eye.
DEVICE_MARKER_RE expected "; === Device: NAME [AA:marker] (category) ===",
but device_config's own template (further down this file) generates
"; === Device: NAME (category) [AA:marker] ===" -- category parens before
the AA tag, not after. The regex never matched a real device comment, so
list_extensions() silently returned [] for every device on every install,
and /api/pstn-permissions served {"extensions": []} regardless of what was
actually in pstn-permissions.conf. That's why the Extensions tab's
Messaging/Voicemail checkboxes always rendered unchecked after a save +
reload even though the file itself had messaging=yes/voicemail=yes written
correctly -- the JS falls back to an all-default row when the endpoint
returns nothing. ea_list_devices() and the rename-device code parse the
same comment via string-splitting/a differently-shaped regex and were
already correct; this was the one broken parser.
The prior fix (930233c) wrote noload lines for app_voicemail_imap.so/
app_voicemail_odbc.so into config/asterisk/modules.conf at install time, but
vendor's docker/entrypoint.sh regenerates /etc/asterisk/modules.conf
unconditionally on every container start (same bind-mounted file) and
clobbers it within seconds — confirmed live on a fresh install with the
prior fix in place. Move the noload patch into
_asterisk_refresh_vendor_files()'s sed pass over the vendor template
instead, alongside the existing logger.conf/cert-regen patches, so it
survives the container's own regeneration. Drop the now-dead host-side
_asterisk_write_modules_conf() and its call sites.
Live-discovered bug, present on every install using this script, not
specific to any one box or extension: the easy-asterisk image ships
app_voicemail.so, app_voicemail_imap.so, and app_voicemail_odbc.so all
autoloading by default -- three alternative storage backends for the
SAME application (VoiceMail, VoiceMailMain, VMAuthenticate,
VoiceMailPlayMsg, VMSayName, the VM_INFO function, several AMI
actions), which collide registering those names against each other on
every single Asterisk start. This box's own container log showed the
exact signature on every restart: "Already have an application
'VoiceMail'" (and every sibling) followed by "app_voicemail.c:15897
load_module: Failure registering applications, functions or tests" --
app_voicemail never actually finished loading. Confirmed against
Asterisk's own documentation this session rather than assumed: this
is a known multi-backend conflict ("administrators should enable only
one module at a time"), not something specific to this repo's config.
Fix: new _asterisk_write_modules_conf, called from both the fresh-
install and update paths (matching voicemail-dialplan.conf's own
call-site pattern) alongside the other config/asterisk files, all
sharing the already-bind-mounted ./config/asterisk:/etc/asterisk
volume -- no new mount needed. noloads the two backends this repo
never configures (no IMAP/ODBC settings are ever written anywhere in
this script), leaving only the plain file-based app_voicemail.so
(the one voicemail.conf's [default] mailboxes actually target) to
load cleanly. Regenerated on every install/update, unlike
voicemail.conf, since modules.conf carries no per-install state of
its own -- consistent with how messaging-dialplan.conf/voicemail-
dialplan.conf are already handled, and added to their same chmod 644
line.
Requires a container restart to take effect on an existing install
(re-run `sudo ./setup.sh asterisk` -> Update, which regenerates this
file, then restart the container once).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01H4k6J1qXXyYxhGEgnJaMvn
Live blocker: user's actual migration is DigitalOcean droplet
(asterisk-digital-ocean/, container easy-asterisk-do) -> IONOS (plain
asterisk/, container easy-asterisk) -- exactly the case this script
didn't handle. The archive's own top-level directory name and
docker-compose.yml reflect whichever layout produced it
(_asterisk_resolve_layout's two known layouts). The previous restore
extracted straight into $PARENT_DIR, which recreates whatever name is
baked into the archive -- restoring a droplet archive onto a fresh
non-droplet install would land the data at a *second*,
wrongly-named directory (asterisk-digital-ocean) alongside the
freshly-installed one it was meant to replace, with docker-compose.yml
still naming the old project/container(s). Every service that resolves
Asterisk's layout by directory/container name (security-dashboard.sh,
pstn-trunk.sh, CrowdSec's Asterisk acquisition, Caddy) would get
confused by having two candidate layouts on disk, one of them stale
and half-wired.
Fix: extract into a scratch staging directory first. If the archived
docker-compose.yml's container_name differs from this run's own
$CONTAINER (baked in at generation time, so always correct for
whichever layout THIS box's install actually uses), rewrite the
project name, container name, and coturn container name in place
(coturn's is always "$CONTAINER-coturn" on both known layouts, so no
lookup table needed) before the data ever lands at $HERE -- never
lets the archive's own naming leak through. A same-layout restore
(most common case, or two droplet boxes, or two plain boxes) detects
no mismatch and skips the rewrite entirely, unchanged from before.
Verified against the real generated script (extracted from the
heredoc, not reimplemented): a droplet-flavored archive restored onto
a fresh plain-layout box lands at the correct single directory with
no stray second directory, and docker-compose.yml's name/container_name/
coturn container_name all correctly rewritten to the plain layout
(confirmed by diffing the actual restored file, not just checking for
absence of errors); a same-layout restore (droplet archive onto a
droplet box) confirmed to skip the rewrite entirely; the pre-existing
external-IP patch (previous commit) still fires correctly stacked on
top of the layout fix; and the extraction-failure rollback path (a
corrupt/unreadable archive) still restores the pre-restore install
untouched, verified via a marker file surviving the rollback.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01H4k6J1qXXyYxhGEgnJaMvn
User confirmed via a live pjsip.conf on their actual DO box: both
external_media_address and external_signaling_address are literal
IPs, written by easy-asterisk at first container start (not something
this repo's install script controls directly). A straight restore of
a backup archive from a DIFFERENT box onto the new IONOS box would
leave the OLD DigitalOcean IP baked into pjsip.conf — dialplan and
PJSIP device credentials would come back fine, but RTP media (and
likely SIP signaling/registration) would stay broken, silently, since
nothing in the restore path previously touched these values.
Fix: after extracting the archive, `restore` reads the archive's own
external_signaling_address as "old IP", detects this host's actual
current public IP (same DO-metadata -> ifconfig.me -> hostname -I
fallback chain services/asterisk.sh's own install already uses), and
if they differ, rewrites every occurrence across config/ and .env
(fixed-string match, not a regex, so the IP's dots can't be
misinterpreted). Deliberately does NOT touch spool/, logs/, or lib/ —
those hold voicemail messages and call recordings, and a blind text
substitution across binary audio would corrupt it. A restore onto the
same host (e.g. rolling back a bad config change, no IP change)
leaves every file untouched — the check only fires on an actual
mismatch.
Verified against the real generated script (extracted verbatim from
the heredoc, not a reimplementation) with a full mock backup/restore
cycle: built a fixture archive with pjsip.conf's three transport
blocks (udp/tcp/tls) and .env's TURN_SERVER all hardcoded to a fake
"old box" IP, plus a fake binary voicemail file; restored it onto a
mocked "new box" with a different detected IP via a stubbed curl.
Confirmed every occurrence in both pjsip.conf and .env was correctly
rewritten to the new IP, and confirmed via byte-for-byte comparison
that the binary voicemail file was completely untouched. Separately
verified the same-IP case (mocked curl returning the archive's own
IP) makes no changes at all, matching a same-host config rollback.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01H4k6J1qXXyYxhGEgnJaMvn
This is what's actually needed for a DigitalOcean -> IONOS Asterisk
migration (item #1 of the user's original 4-item list): move the
whole stack to different hardware entirely, not restore onto the
box the offsite mirror already targets.
dr_bringup_kopia.sh only ever scanned DEST_NAMES for restorable
snapshots — those are always local filesystem Kopia repos
(services/backup.sh creates them with `repository create filesystem
--path=...`), meaning they only exist on whichever box originally ran
the backup. On a genuinely new box, every one of them fails to
connect and there's nothing left to restore from — the script's own
header comment only covered the case where "the spare box IS the box
the primary's mirror targets" (i.e. already holds a copy of the repo
data), not a fresh, unrelated box.
Fix: also try REMOTE_TYPE/REMOTE_ARGS (the offsite Backblaze/S3
mirror, if configured) as a same-shaped destination named "offsite",
reusing DEST_default_PASSWORD since sync-to always mirrors that exact
same encrypted repo. Connects once into a fresh local config file
scoped to this DR run, then folds into the existing per-destination
scan/restore loop unchanged — "offsite" just becomes another entry in
_DEST_ARR. Documented the actual migration workflow in the header
comment, including the BACKUP_CONF override so copying the old box's
backup.conf over doesn't clobber the new box's own freshly-configured
one.
Verified against the real script (not a reimplementation) with mocked
kopia/docker binaries and a crafted backup.conf, covering: local dest
unreachable + offsite connects successfully (the actual migration
shape) with correct service/path discovery; a real (non---list)
restore run confirmed it selects the latest of multiple snapshots by
startTime and issues the correct `kopia restore <snapshot> <path>`
call; offsite connect failing (bad REMOTE_ARGS) warns and degrades to
"no restorable sources found" instead of crashing; REMOTE_TYPE=none
skips the new code path entirely with no behavior change (regression
check against the pre-existing local-only case).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01H4k6J1qXXyYxhGEgnJaMvn
User's actual question: Backblaze B2's web console lets them browse a
bucket as folders/files; Garage has no equivalent by default, so after
switching an additional backup mirror from Backblaze-only to also
target local Garage, they had no way to visually confirm data landed
there the way they could on Backblaze. "S3 storage is opaque, you
can't browse it" was true of Garage's *own* CLI, but wrong as a
blanket statement — Backblaze's browsability comes from a client (its
web console) layered on top of the same kind of object storage, and
Garage has an actively-maintained equivalent (khairul169/garage-webui,
1.1k stars, "integrated objects/bucket browser") that gives the same
experience against Garage's S3 API.
services/garage-webui.sh (new): standard service-template Docker
service. Requires an existing services/garage.sh install (checks for
$DOCKER_DIR/garage/.env, errors with instructions if missing — this
is a browser for an existing instance, not a replacement). Reaches
Garage over host.docker.internal (both containers' ports are already
published to the host — simpler and more robust than trying to join
garage's own Compose-project-scoped default network by name). Has its
own login (AUTH_USER_PASS, bcrypt via a throwaway `docker run --rm
httpd:alpine htpasswd` — same $ -> $$ escaping services/wg-easy.sh
already uses for its own bcrypt PASSWORD_HASH, verified here against a
real docker compose config run: unescaped, Compose tries to interpolate
$2y$05... as variable references and silently corrupts the value with
a "not set" warning; escaped, it passes through intact with no
warning), so it doesn't need Authelia gating by default.
Prerequisite fix in services/garage.sh: its admin API (bucket/key
management, object listing — the thing garage-webui talks to) has
been running with zero authentication since this service was first
built, because admin_token was never set in garage.toml. Nothing in
this repo called that API before now, so it went unnoticed; adding a
real consumer is what surfaced it. Fixed: generate admin_token
(openssl rand -base64 32) alongside the existing rpc_secret, persist
GARAGE_ADMIN_TOKEN/GARAGE_ADMIN_PORT to .env for garage-webui to read
locally (never sent over SSH, unlike the S3 credentials backup.sh
reads remotely). Update mode backfills admin_token into an existing
garage.toml (+ restarts just the garage container to apply it) for
anyone who installed before this change, same backfill-not-break
approach as the GARAGE_S3_API_PORT fix from the previous commit.
Verified: bash -n on both files; docker compose config against real
Docker Compose for both the primary garage.toml/.env generation (with
the new admin_token/GARAGE_ADMIN_PORT fields) and the new
garage-webui docker-compose.yml; the bcrypt-escaping behavior
specifically (proved via a minimal repro that unescaped $ corrupts
the value with a warning, escaped does not); the admin_token/
GARAGE_ADMIN_PORT Update-mode backfill logic against old- and
new-style .env/garage.toml fixtures, including idempotency (running
it twice adds nothing a second time); and the credential-parsing
regexes in garage-webui.sh against both a complete .env fixture and
an old one missing the new fields (confirms the "run garage's Update
first" error path actually triggers rather than proceeding with
blanks).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01H4k6J1qXXyYxhGEgnJaMvn
Live failure: `sudo ./setup.sh garage` → Full reinstall on a box that
already had a working Garage install crashed with:
Error: ApplyClusterLayout returned InternalError (500): Internal
error: Invalid new layout version
Root cause: a "Full reinstall" deliberately never wipes ./data or
./meta (that's real backup-mirror data — Kopia's sync-to s3 target —
and losing it silently on reinstall would be far worse than the
alternative), but the cluster-init step unconditionally re-ran `garage
layout assign` + `layout apply --version 1` every time it was reached.
Garage requires each apply to be exactly previous_version + 1; a node
that already has a committed layout (from the earlier install, still
sitting in the preserved ./meta) rejects a second "1". Fix: check
`garage status` for "NO ROLE ASSIGNED" first and only run the
assign/apply once, matching what the surrounding comment already
claimed happened ("Only ever run once") but the code didn't enforce.
Second, related issue this would have hit immediately after: the same
reused-./meta state almost always means an existing bucket + key from
the earlier install are still sitting in Garage's storage. The fresh
flow was about to silently create a brand-new bucket/key and overwrite
.env to point at those instead — orphaning any real data already in
the old bucket (nothing left on disk pointing at it, even though it's
still physically stored). Now: when the layout is already applied,
list existing buckets and require an explicit y/n (default n) before
creating new ones, with recovery instructions for reconnecting to an
existing bucket by hand instead.
Third, the actual reason a full reinstall was reached at all: Update
mode never backfills .env fields added to this script after someone's
initial install (GARAGE_S3_API_PORT, needed by services/backup.sh to
read an instance remotely) since Update deliberately never touches
.env otherwise — the only other path was the now-unsafe fresh
reinstall. Update now backfills just that missing key by reading the
real port back out of the already-written docker-compose.yml, so a
future .env schema addition doesn't force this tradeoff again.
Verified with standalone harnesses (not the live install, mocked
`garage status`/bucket-list output and .env/docker-compose.yml
fixtures): all four layout-state branches (fresh node, existing
buckets + decline, existing buckets + confirm, existing role but no
buckets), and both backfill cases (missing key added, existing key
left alone). Caught and fixed a real bug in the first draft of the
port-extraction regex during this testing — grep -oE '^[0-9]+' never
matched because the captured group still had its surrounding quotes.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01H4k6J1qXXyYxhGEgnJaMvn
Previously, if the additional-mirror S3/Garage check couldn't find
~/docker/garage/.env on the remote box, it just warned and silently
dropped the mirror — forcing a full re-run (and re-entering every
already-answered prompt: destinations, passwords, schedule, B2,
DR-spare, etc.) once Garage was actually installed.
Wrap the S3/SFTP branch in a loop so the "Garage isn't installed yet"
case now offers a real 3-way choice:
1) install Garage in another session, then retry the same .env check
without leaving this script
2) fall back to SFTP for this one mirror, reusing the already-resolved
destination host/port/user/mirror-name with no re-prompting
3) skip just this mirror (default — safe for UNATTENDED, which
resolves to this automatically since prompt_text returns its
default without blocking)
Everything else install_backup() has already collected lives outside
this loop, so none of it is at risk regardless of which of the three
exits it via.
Verified against a standalone harness reproducing the state machine
with a mocked ssh (empty .env vs. populated .env after a simulated
install) and prompt_text, covering all three interactive choices, the
blank/Enter default, and UNATTENDED mode (confirms the blocking
"press Enter to retry" read is unreachable there since prompt_text
resolves choice 1's prompt to default "3" without waiting on stdin).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01H4k6J1qXXyYxhGEgnJaMvn
Extends the "ADDITIONAL MIRROR" section (previously SFTP-only) with a
type choice: SFTP, or S3 against a Garage instance already running on
that box. For the S3 path, this script never asks the operator to retype
a bucket name or key — it SSHes to the destination, reads
~/docker/garage/.env directly (the real, currently-configured values,
generated once by services/garage.sh and never touched again on its own
Update runs), and uses those for the dry-run verification and the
persisted mirror args. If Garage isn't installed there yet, it says so
plainly with the exact install command instead of failing cryptically or
silently skipping.
Also removed the last hardcoded suggestions from services/garage.sh
itself ("kopia-backup" / "kopia" as fixed prompt defaults) — replaced
with a freshly-generated suggestion each run (timestamp-suffixed), so
nothing about the bucket/key name is a fixed string baked into this repo
at any point in the chain; it's always the operator's actual choice, read
back live wherever it's needed.
Verified end-to-end against a mocked ssh (returning realistic
~/docker/garage/.env content) covering both outcomes: Garage installed
with a real bucket/key correctly parsed, dry-run run, and persisted; and
Garage missing, correctly warning with the install command and leaving
backup.conf untouched either way.
Garage's real CLI output pads labels with extra spaces for column
alignment ("Key ID: GKxxxx"), not a single space like the
mocked test used ("Key ID: GKxxxx") — the fixed ": " field separator left
that padding stuck to the parsed value, so .env ended up with access
key/secret strings carrying leading whitespace inside the quotes.
Confirmed live by the user right after install. This would have broken S3
auth outright once actually used, since access keys have to match exactly.
Switched to ':[[:space:]]+' as a regex field separator, which consumes
however many spaces are actually there instead of assuming exactly one.
Verified against both the single-space and padded/aligned formats — both
now produce the identical clean value with no leading whitespace.
MinIO's open-source community edition is dead: console GUI stripped May
2025, Docker images stopped publishing October 2025, repo formally
archived April 2026, with MinIO redirecting everyone to their paid AIStor
product. Verified this directly before building anything, since recommending
a since-abandoned image would have been worse than the SFTP problem this
was meant to solve.
Garage (Deuxfleurs) is the actively-maintained small-scale self-hosted
replacement — single Rust binary, purpose-built for exactly this "one
lightweight node" use case (as opposed to SeaweedFS, which targets large
object counts / large-scale deployments, more machinery than a single
backup-mirror target needs).
services/garage.sh follows this repo's standard service template: port
scanning for the S3 API/RPC/admin ports, an RPC secret generated once and
never touched again on Update, and a one-time cluster init sequence
(layout assign/apply, bucket create, key create, bucket allow) gated on
whether .env already has a saved access key — Update reruns skip all of it
and just refresh the image.
Primary intended use: a local S3-compatible target for services/backup.sh's
additional-mirror Kopia sync, so a local mirror can reuse the exact same
sync-to s3 code path already proven reliable for the Backblaze B2 mirror,
instead of Kopia's separate, less-exercised SFTP backend that's been the
source of today's connection troubleshooting.
Verified end-to-end against a mocked environment (fake docker exec
returning realistic `garage status`/`garage key create` output) — caught
and fixed a real off-by-one in the status-output parsing this way (grabbed
the column-header row's literal "ID" instead of the actual node ID; output
has a title line, then a header line, then the data row). Also validated
the generated docker-compose.yml with real `docker compose config` in both
the no-network and network-created cases.
The additional-mirror "Remote path for the repo" prompt always suggested
a generic ~/backups/kopia-mirror default, unrelated to wherever the
operator already pointed the DR-spare sync. Requested directly: default
to that same location instead, in its own /kopia-data subdirectory so
Kopia's repository files don't end up visually mixed in with the two
plain config files (backup.conf, README.md) the DR-spare sync writes
straight into DR_SYNC_PATH itself.
Falls back to the original generic default when DR_SYNC_PATH isn't set
(no DR-spare configured yet). Verified the path computation handles a
DR_SYNC_PATH with or without a trailing slash correctly (no double slash),
and the unset case still falls back as before.
The additional-mirror setup already resolves user/hostname through ssh -G
so a ~/.ssh/config alias works, but never extracted port — Kopia's sftp
storage backend doesn't read ~/.ssh/config at all and defaults to 22
regardless of what the alias actually configures. Confirmed live: this
produced "server unexpectedly closed connection: unexpected EOF" on the
dry-run verification — Kopia connecting to the right host on the wrong
port, not a credentials or host-key issue, which is exactly why plain
`ssh main` kept working the entire time this was being debugged (it reads
the alias's Port line correctly).
Now parses `port` out of the same ssh -G output, defaults to 22 if absent
(matching ssh's own default), and passes --port= through to both the
dry-run check and the persisted EXTRA_MIRROR_ARGS string — the latter
matters as much as the former, since that's what every actual scheduled
sync reuses afterward, not just the one-time verification.
Verified the parsing against three cases: a custom-port alias, a
default-port alias, and an unresolvable alias — all three resolve to the
correct port with no manual intervention needed.
extras/fix_pikapods_dump.py patches two confirmed Adminer PostgreSQL-export
bugs that otherwise make a PikaPods Mattermost migration fail outright:
unquoted enum-label DEFAULT values (Postgres reads the bare label as a
column reference and rejects the CREATE TABLE) and boolean columns
serialized as bare 0/1 instead of true/false (Postgres doesn't implicitly
cast integers to boolean). Boolean columns are discovered by actually
parsing each CREATE TABLE in the dump rather than working from a
hand-curated list — Postgres only reports the first bad column per failed
row, so a list built from error output alone would likely be incomplete.
Already verified earlier this session against a real local Postgres 16
instance; reviewed now for anything needing redaction before committing —
it's a generic text-processing tool with no hostnames, credentials, file
paths, or personal data in it, so nothing needed changing.
Cross-referenced from the generated migrate-from-pikapods.sh's header
comment (services/mattermost.sh) so anyone hitting a CREATE TYPE/CREATE
TABLE or boolean-column import error is pointed at the fix instead of
having to rediscover it.
The generated migrate-from-pikapods.sh (services/mattermost.sh's existing
"Migrating from an existing Mattermost instance?" prompt on fresh installs)
already correctly parameterizes PROJECT_DIR/MM_CONTAINER/DB_CONTAINER per
instance — no bug there. What it missed: after rsync/cp-ing files in from
the export, it never touched ownership, so the imported ./data landed
owned by whoever ran the script instead of the fixed UID 2000
mattermost/mattermost-team-edition runs as. Every file write then failed
with permission denied — confirmed live as the actual cause of a
client-side "stream closed" error on image/file uploads after a real
migration.
Adds chown -R 2000:2000 ./data right after the copy step, and a root
check up front since chowning to an arbitrary UID needs it (docker/psql
access already implied running as root in practice, just never enforced
explicitly). Usage lines updated to say `sudo` to match.
Verified by reconstructing the exact generated script from the real
source heredocs (head + variable substitution + body, the same three
pieces the actual cat/cat>> sequence produces) and syntax-checking the
result — root check and chown both land in the right place, and
PROJECT_DIR/MM_CONTAINER/DB_CONTAINER still resolve correctly per instance.
Root-caused a live "stream closed" image-upload failure to
data/20260814/.../mkdir: permission denied — the volumes weren't owned by
the fixed UID 2000 mattermost/mattermost-team-edition runs as, most likely
left that way by the PikaPods data import. The install script already
chown -R 2000:2000's these on every run (fresh or update), so re-running
the installer would have fixed it — but that still means remembering to
re-run it every time ownership drifts for any reason, including causes
this repo doesn't control (a future migration, a manual restore, anything
that copies files in as a different UID).
Added a small mattermost-fix-perms init container (busybox, chown, exit)
that the mattermost service now depends on via
condition: service_completed_successfully. Runs on every `docker compose
up` — including a plain host reboot, since restart: unless-stopped brings
the stack back on its own — so this self-heals permanently instead of
needing a human to notice and fix it by hand again.
Verified the generated compose file (with representative variable values)
against real `docker compose config`: valid YAML, and the dependency graph
correctly shows mattermost waiting on both db (service_healthy) and
mattermost-fix-perms (service_completed_successfully).
Two independent, requested changes:
- services/pihole.sh: new standalone service, Pi-hole v6 (the image moved
entirely to a TOML-based /etc/pihole config — the old WEBPASSWORD env var
and separate /etc/dnsmasq.d volume are both gone; uses
FTLCONF_webserver_api_password and FTLCONF_dns_listeningMode=ALL
instead). Deliberately not wired into wg-easy or any other VPN — a device
has to be pointed at it manually (per-device or via router DHCP). DNS
itself (53/tcp+udp) is never scanned/moved since shifting it off the
standard port would defeat the point; a port_in_use check warns instead
of blocking, since the common case (systemd-resolved on 127.0.0.53 only)
doesn't actually collide with Pi-hole binding the host's real interfaces.
Web admin UI is Caddy-fronted like everything else in this repo. Added to
the README services table.
- services/wg-easy.sh: default VPN/web ports moved from 51820/51821 to
51830/51831. Netbird's own WireGuard listener also defaults to exactly
51820 — installing both on one box means wg-easy's existing scan-and-move
logic would silently shift its port every time, which is harder to
predict/document than just not starting on the collision in the first
place. The scan itself is unchanged and still moves both ports further if
even the new default is taken.
Tested pihole.sh's full standalone install flow (no-Caddy and
Caddy-present-locally cases) against a mocked environment, validating both
generated docker-compose.yml files with `docker compose config`, and
confirmed the reinstall-mode gate correctly no-ops on a second run in
unattended mode.
ensure_coturn_user() auto-installs shared coturn (services/coturn.sh) any
time $DOCKER_DIR/coturn doesn't exist — correct behavior for "first
service that needs TURN", wrong behavior for "an operator deliberately
decided every consumer should run its own dedicated coturn instead and
removed the shared one on purpose". The function had no way to tell those
two states apart, so deleting ~/docker/coturn didn't actually retire it —
the next service to call this function (a Mattermost reinstall, a fresh
Asterisk install) would silently bring it right back.
touch ~/docker/.coturn-retired now short-circuits the function straight to
the existing "no TURN available, caller degrades gracefully" return path,
before it ever looks at install_coturn. Every existing consumer (Asterisk,
Mattermost) already handles that path correctly today — it's exactly what
happens if shared coturn simply fails to install — so this needed no
changes on the consumer side, only closing the gap in the shared function.
Verified with a mock: with the flag present, install_coturn is never
invoked and the function returns empty COTURN_HOST/rc=1 as expected.
Requested after a rerun silently reset DR_SYNC_PATH (fixed separately) —
auditing the rest of install_backup() turned up the same class of bug in
several other places, one of them worse than the one that prompted this:
- Default destination repo path defaulted to $ACTUAL_HOME/backups/... even
when the real configured repo was somewhere else entirely (this user's
actual path is /root/backups/kopia-backup) — accepting the shown default
on a rerun would have pointed the installer at the wrong location.
- Extra (non-"default") destinations weren't preserved AT ALL on a rerun —
skipping "Add more destinations?" silently dropped every extra
destination, and anything mapped to it, from the rewritten backup.conf.
- The per-service destination-assignment prompt always showed "[default]"
regardless of the service's actual existing mapping.
- ntfy URL/token always started blank, silently disabling notifications on
any rerun where they weren't retyped.
- The schedule prompt always defaulted to option 1 (daily 02:00) instead of
reading back whatever OnCalendar was actually already running.
- B2's four sub-fields (bucket/endpoint/key ID/secret) always started
blank even when reconfiguring an already-working REMOTE_TYPE=s3 setup —
a mispaste on any one of the four meant retyping all four blind, since
there was nothing to fall back to per-field (the existing REMOTE_ARGS was
already preserved as a whole on a blank/failed attempt, just not offered
back as individual editable defaults).
All six read the same way: pull the existing value from backup.conf (or,
for the schedule, from the live systemd timer unit — schedule isn't stored
in backup.conf) and use it as the prompt default, so accepting the default
keeps what's already there instead of silently reverting it. Verified all
six against a mock backup.conf + timer fixture with pre-existing values for
every field this touches.
Known remaining gap: KEEP_LATEST (retention count) still isn't read back —
doing so correctly needs the repo already connected, which happens later
in this same function's flow. Flagging rather than rushing a reorder here.
Two stacked bugs, found together when re-running the backup installer to
add an SFTP mirror silently reverted a previously-set absolute
DR_SYNC_PATH back to the script's tilde-based default, which then failed
outright:
1. services/backup.sh never read DR_SYNC_HOST/DR_SYNC_PATH back from an
existing backup.conf before prompting (every other setting in this file
does — passwords, mirrors). Accepting the prompt defaults on a rerun
silently reset both to blank/"~/docker/backup" instead of keeping what
was already configured. Fixed by reading them back the same way
DEST_*_PASSWORD already does.
2. extras/backup_kopia.sh's DR-spare sync wraps the remote path in single
quotes for its `ssh host "mkdir -p '...'"` / `"chmod 600 '.../...'"`
commands. Single-quoting a leading ~ stops the remote shell from
expanding it at all, so it looked for a literal directory named "~"
instead of the home directory — breaking the script's own DEFAULT
DR_SYNC_PATH ("~/docker/backup") for anyone who actually used it.
rsync's own transfer step has separate, correct tilde handling, which is
why the sync itself "succeeded" while the follow-up chmod couldn't find
the file. Fixed with a small _dr_remote_quote() helper that keeps a
leading ~/ outside the quotes while still safely quoting the rest of
the path.
Verified the quoting fix by parsing the exact constructed command string
in bash directly — a plain '~/docker/backup' stays literal (the bug),
~/'docker/backup' correctly expands to $HOME/docker/backup (the fix).
_backup_ensure_root_ssh_key() only ever checked for /root/.ssh/id_ed25519
or id_rsa by exact filename. Root can already SSH to the DR-spare/mirror
host just fine in practice (proven by this same script's own DR-spare sync
succeeding), just via a key with some other name — so the function had no
way to see that and always fell through to offering a copy-from-user-home
or brand-new ssh-keygen, both unnecessary.
Now takes the target host as an optional argument. When given, it tests
root's SSH access to that host as-is first and resolves the actual key via
`ssh -G <host>` (which expands ~/.ssh/config the same way the SFTP-dest
resolution earlier in this file already does) before falling back to the
copy/generate prompts. Both call sites (DR-spare, SFTP mirror) now pass
their respective host.
Verified against a mock ssh: an already-working non-default-named key gets
detected and reused with no prompts, and the original copy/generate
fallback still triggers correctly when SSH genuinely doesn't work yet.
The freshly-added raw-error logging paid off immediately: the box's spare
sync was failing every run with "scp: Connection closed" while plain ssh
exec to the same host worked fine. That split (ssh exec OK, scp specifically
rejected) matches modern OpenSSH's default scp-over-SFTP transfer hitting a
restriction on the remote side that a plain exec or rsync's own protocol
don't trigger.
Swapped the scp step for rsync -a over the same ssh options, keeping the
ssh mkdir -p before it (rsync doesn't create missing destination
directories) and the ssh chmod after. Verified the exact command/quoting
against mocked ssh/rsync binaries — array expansion and remote path
handling both check out.
Last night's fully-failed backup run (0/20, "repository not found" on
every service) showed the real gap: categorize_error() clearly saw real
content in $_ERR (it matched a specific pattern, not the generic
fallback), but log_raw_error()'s separate re-read of the same file moments
later came back empty on every single failure — so the raw-error logging
added earlier this session produced nothing when it mattered most.
Fixed by reading $_ERR into a variable exactly once per failure and
passing that string to both categorize_error() and log_raw_error(),
instead of two independent file reads. Verified against a mock harness
reproducing the same call pattern (three simulated failures in a loop,
single shared error file) — both the categorized reason and the raw
stderr text now come through on every iteration.
Doesn't explain why last night's repo access failed in the first place
(disk and mount checks came back clean) — but the next time it happens,
this will actually surface the real kopia error instead of losing it.
Asterisk's whole state (dialplan, pjsip devices, voicemail, recordings,
.env with its coturn credential, docker-compose.yml) already lives under
one self-contained directory, so asterisk-standalone-backup.sh just tars
it — with stop/restart safety around the tar since voicemail/spool write
continuously, and a move-aside-then-extract restore that rolls back
automatically if extraction fails. Written into the install directory at
both fresh-install and update time via _asterisk_write_standalone_backup_script().
Output defaults to ~/asterisk-backups/, deliberately outside ~/docker/, so
a Kopia backup of the box doesn't also back up a backup-of-itself. Meant
for a quick pre-change snapshot or moving this PBX to a new host without
standing up the full backup stack first.
Documented in the generated README's new "Standalone backup/restore"
section. Tested against a mocked EA_DIR (fake docker/docker compose,
config/spool/voicemail files) confirming backup produces a correct tar and
restore replaces content correctly with rollback on extraction failure.
Confirmed live and cross-checked against a real, documented Kopia issue
(kopia/kopia#5329): the walkthrough previously told the operator to scope
the Application Key to just the bucket they created — the more
security-conservative default, and correct for B2's own S3-compatible
API in general. But Kopia specifically needs the listBuckets capability
even though it only ever touches the one configured bucket, and B2's
basic "Add a New Application Key" web form doesn't expose a way to grant
listBuckets on a bucket-restricted key — only an account-wide ("All")
key gets it through that form. Without it, the connection fails with
B2's unhelpful "Cannot access bucket" error, which doesn't point at the
actual missing capability at all.
Updated the guidance to "All" with the reasoning inline, and a note that
single-bucket scoping is still possible for anyone willing to create the
key via B2's CLI/API directly (b2_create_key with an explicit
capabilities list including listBuckets) rather than the basic web form
this walkthrough is written for.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01H4k6J1qXXyYxhGEgnJaMvn
Two separate fixes from a live report.
1. The DR-spare and SFTP-mirror sections both checked ONLY /root/.ssh
for a key, missing the common case: the person running `sudo
./setup.sh backup` already has a key under their own home directory
(used interactively, quite possibly already authorized on the target
box), while root — who actually runs the scheduled systemd service —
has none. Confirmed live: "the computer has the ssh key for the sudo
user on the box" produced "No SSH key found for root" with no
inline way to do anything about it beyond a pointer to go set one up
elsewhere and re-run.
Factored both call sites into one shared _backup_ensure_root_ssh_key()
that checks root first, then offers to reuse the sudo user's existing
keypair (copied into /root/.ssh with correct ownership/permissions,
root:root 600) before falling back to generating a brand new one —
reusing an existing key can work immediately if it's already
authorized on the target, where a fresh key needs a new ssh-copy-id
round-trip regardless. Verified all three branches (root already has
a key, root has none but the user does and accepts reuse, neither
exists and one gets generated) against a mocked filesystem.
2. The B2 dry-run failure message read like it could be about missing
input even when every field was non-empty — confirmed there's no
code path where non-blank-but-wrong values actually trigger the
separate "Left blank" message (the two are on disjoint branches), but
the dry-run failure text itself didn't rule that out or point at the
actual likely cause. Now echoes back what was entered (bucket,
endpoint, Key ID — never the secret) so it's easy to eyeball against
B2's own confirmation screen, states plainly that this is a rejection
of non-blank input, and names the most likely cause directly: pairing
the Key ID from one Application Key with the Secret from a different
one, which is easy to do after creating more than one while
troubleshooting.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01H4k6J1qXXyYxhGEgnJaMvn
Confirmed live: a real failure ("WARNING: spare sync failed — error —
see system logs on ubuntu") didn't match any of categorize_error()'s
known patterns, fell into its generic catch-all bucket, and the actual
stderr text that would have explained it was sitting in a mktemp'd file
this script deletes on exit (trap ... EXIT) — so there was nothing in
"system logs" to actually go check. The categorization was silently
discarding the one piece of information that would have diagnosed the
problem.
Added log_raw_error(), called right after every categorize_error() site
(5 of them: two snapshot-failure paths, the primary REMOTE_TYPE mirror,
the new EXTRA_MIRROR_NAMES loop, and the DR-spare sync) — logs the raw
stderr text (truncated to 500 chars) into the same log stream as
everything else, so it survives past the run that produced it instead
of being deleted with the temp file. categorize_error()'s short bucket
label is untouched and still used for FAILED_SVCS/notification text,
which should stay concise — this adds the detail alongside it, not
instead of it.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01H4k6J1qXXyYxhGEgnJaMvn
Confirmed: Kopia's sync-to sftp has its own SFTP client and doesn't read
~/.ssh/config the way the system ssh/scp binaries do — so an alias set
up via wg-easy's sync-ssh-aliases.sh (or any ~/.ssh/config Host entry)
worked fine for the DR-spare connectivity check (which shells out to
real ssh) but silently failed for this mirror: a plain @-split on an
alias like "main" (no @ present) produced --host=main, a name that only
resolves inside ~/.ssh/config, not real DNS. The dry-run check correctly
rejected it and the mirror was never saved — no error surfaced beyond
that, so it looked like nothing happened.
Now resolves the destination through `ssh -G` before building the Kopia
flags — the same mechanism ssh itself uses to expand config aliases —
and falls back to the previous plain @-split only if that comes back
empty. Verified against three cases: a bare alias (resolves via a mock
~/.ssh/config Host block), an explicit user@ip (passes through
unchanged), and an unrecognized name (falls back to a sane literal
hostname rather than erroring).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01H4k6J1qXXyYxhGEgnJaMvn
Requested: mirror to Backblaze B2 AND directly to the IONOS spare box
over Tailscale, at the same time, not one or the other. REMOTE_TYPE/
REMOTE_ARGS was hardcoded to a single mirror target — extending it to a
list would have meant redesigning the one thing that already works and
was already verified against real B2 credentials, so this adds a
separate, additive mechanism instead: EXTRA_MIRROR_NAMES, a space-
separated list, with per-entry MIRROR_<name>_TYPE/_ARGS (same argument
shape as REMOTE_ARGS). An existing B2-only backup.conf keeps working
completely unchanged if this new section is skipped.
install_backup() gets a new "ADDITIONAL MIRROR" prompt after the
existing B2 section: offers a direct SFTP mirror (Kopia's sync-to sftp,
not the deprecated b2 provider — same reasoning as the S3/B2 choice
already made), defaults the destination to whatever was typed at the
DR-spare prompt above (same box, same purpose, no reason to ask twice),
checks passwordless SSH and an SSH key exist first, then verifies with a
--dry-run against the just-created 'default' repo before saving it —
same "don't save something broken" discipline as the B2 flow. Verified
against a mock backup.conf that install-side writes and worker-side
reads agree on the exact format, and that reusing an existing mirror
name reconfigures it instead of duplicating it in the name list.
One correction while researching sync-to sftp's flags: unlike plain ssh,
Kopia doesn't shell out to the system SSH client, so it needs an
explicit --keyfile and --known-hosts path rather than picking up
whatever `ssh` already trusts automatically — checked Kopia's own docs
for the exact flags before writing this, same as the earlier S3 case.
extras/backup_kopia.sh's worker loops through EXTRA_MIRROR_NAMES after
the existing REMOTE_TYPE mirror step, running sync-to for each
destination against each additional mirror and folding failures into
the same FAILED_SVCS/notification reporting the primary mirror already
uses. Verified end-to-end against a mock backup.conf and a stubbed
kp_for: both the B2 and the new SFTP mirror get called in sequence with
the correct arguments.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01H4k6J1qXXyYxhGEgnJaMvn
Claude