Requested after a rerun silently reset DR_SYNC_PATH (fixed separately) —
auditing the rest of install_backup() turned up the same class of bug in
several other places, one of them worse than the one that prompted this:
- Default destination repo path defaulted to $ACTUAL_HOME/backups/... even
when the real configured repo was somewhere else entirely (this user's
actual path is /root/backups/kopia-backup) — accepting the shown default
on a rerun would have pointed the installer at the wrong location.
- Extra (non-"default") destinations weren't preserved AT ALL on a rerun —
skipping "Add more destinations?" silently dropped every extra
destination, and anything mapped to it, from the rewritten backup.conf.
- The per-service destination-assignment prompt always showed "[default]"
regardless of the service's actual existing mapping.
- ntfy URL/token always started blank, silently disabling notifications on
any rerun where they weren't retyped.
- The schedule prompt always defaulted to option 1 (daily 02:00) instead of
reading back whatever OnCalendar was actually already running.
- B2's four sub-fields (bucket/endpoint/key ID/secret) always started
blank even when reconfiguring an already-working REMOTE_TYPE=s3 setup —
a mispaste on any one of the four meant retyping all four blind, since
there was nothing to fall back to per-field (the existing REMOTE_ARGS was
already preserved as a whole on a blank/failed attempt, just not offered
back as individual editable defaults).
All six read the same way: pull the existing value from backup.conf (or,
for the schedule, from the live systemd timer unit — schedule isn't stored
in backup.conf) and use it as the prompt default, so accepting the default
keeps what's already there instead of silently reverting it. Verified all
six against a mock backup.conf + timer fixture with pre-existing values for
every field this touches.
Known remaining gap: KEEP_LATEST (retention count) still isn't read back —
doing so correctly needs the repo already connected, which happens later
in this same function's flow. Flagging rather than rushing a reorder here.
Two stacked bugs, found together when re-running the backup installer to
add an SFTP mirror silently reverted a previously-set absolute
DR_SYNC_PATH back to the script's tilde-based default, which then failed
outright:
1. services/backup.sh never read DR_SYNC_HOST/DR_SYNC_PATH back from an
existing backup.conf before prompting (every other setting in this file
does — passwords, mirrors). Accepting the prompt defaults on a rerun
silently reset both to blank/"~/docker/backup" instead of keeping what
was already configured. Fixed by reading them back the same way
DEST_*_PASSWORD already does.
2. extras/backup_kopia.sh's DR-spare sync wraps the remote path in single
quotes for its `ssh host "mkdir -p '...'"` / `"chmod 600 '.../...'"`
commands. Single-quoting a leading ~ stops the remote shell from
expanding it at all, so it looked for a literal directory named "~"
instead of the home directory — breaking the script's own DEFAULT
DR_SYNC_PATH ("~/docker/backup") for anyone who actually used it.
rsync's own transfer step has separate, correct tilde handling, which is
why the sync itself "succeeded" while the follow-up chmod couldn't find
the file. Fixed with a small _dr_remote_quote() helper that keeps a
leading ~/ outside the quotes while still safely quoting the rest of
the path.
Verified the quoting fix by parsing the exact constructed command string
in bash directly — a plain '~/docker/backup' stays literal (the bug),
~/'docker/backup' correctly expands to $HOME/docker/backup (the fix).
_backup_ensure_root_ssh_key() only ever checked for /root/.ssh/id_ed25519
or id_rsa by exact filename. Root can already SSH to the DR-spare/mirror
host just fine in practice (proven by this same script's own DR-spare sync
succeeding), just via a key with some other name — so the function had no
way to see that and always fell through to offering a copy-from-user-home
or brand-new ssh-keygen, both unnecessary.
Now takes the target host as an optional argument. When given, it tests
root's SSH access to that host as-is first and resolves the actual key via
`ssh -G <host>` (which expands ~/.ssh/config the same way the SFTP-dest
resolution earlier in this file already does) before falling back to the
copy/generate prompts. Both call sites (DR-spare, SFTP mirror) now pass
their respective host.
Verified against a mock ssh: an already-working non-default-named key gets
detected and reused with no prompts, and the original copy/generate
fallback still triggers correctly when SSH genuinely doesn't work yet.
The freshly-added raw-error logging paid off immediately: the box's spare
sync was failing every run with "scp: Connection closed" while plain ssh
exec to the same host worked fine. That split (ssh exec OK, scp specifically
rejected) matches modern OpenSSH's default scp-over-SFTP transfer hitting a
restriction on the remote side that a plain exec or rsync's own protocol
don't trigger.
Swapped the scp step for rsync -a over the same ssh options, keeping the
ssh mkdir -p before it (rsync doesn't create missing destination
directories) and the ssh chmod after. Verified the exact command/quoting
against mocked ssh/rsync binaries — array expansion and remote path
handling both check out.
Last night's fully-failed backup run (0/20, "repository not found" on
every service) showed the real gap: categorize_error() clearly saw real
content in $_ERR (it matched a specific pattern, not the generic
fallback), but log_raw_error()'s separate re-read of the same file moments
later came back empty on every single failure — so the raw-error logging
added earlier this session produced nothing when it mattered most.
Fixed by reading $_ERR into a variable exactly once per failure and
passing that string to both categorize_error() and log_raw_error(),
instead of two independent file reads. Verified against a mock harness
reproducing the same call pattern (three simulated failures in a loop,
single shared error file) — both the categorized reason and the raw
stderr text now come through on every iteration.
Doesn't explain why last night's repo access failed in the first place
(disk and mount checks came back clean) — but the next time it happens,
this will actually surface the real kopia error instead of losing it.
Asterisk's whole state (dialplan, pjsip devices, voicemail, recordings,
.env with its coturn credential, docker-compose.yml) already lives under
one self-contained directory, so asterisk-standalone-backup.sh just tars
it — with stop/restart safety around the tar since voicemail/spool write
continuously, and a move-aside-then-extract restore that rolls back
automatically if extraction fails. Written into the install directory at
both fresh-install and update time via _asterisk_write_standalone_backup_script().
Output defaults to ~/asterisk-backups/, deliberately outside ~/docker/, so
a Kopia backup of the box doesn't also back up a backup-of-itself. Meant
for a quick pre-change snapshot or moving this PBX to a new host without
standing up the full backup stack first.
Documented in the generated README's new "Standalone backup/restore"
section. Tested against a mocked EA_DIR (fake docker/docker compose,
config/spool/voicemail files) confirming backup produces a correct tar and
restore replaces content correctly with rollback on extraction failure.
Confirmed live and cross-checked against a real, documented Kopia issue
(kopia/kopia#5329): the walkthrough previously told the operator to scope
the Application Key to just the bucket they created — the more
security-conservative default, and correct for B2's own S3-compatible
API in general. But Kopia specifically needs the listBuckets capability
even though it only ever touches the one configured bucket, and B2's
basic "Add a New Application Key" web form doesn't expose a way to grant
listBuckets on a bucket-restricted key — only an account-wide ("All")
key gets it through that form. Without it, the connection fails with
B2's unhelpful "Cannot access bucket" error, which doesn't point at the
actual missing capability at all.
Updated the guidance to "All" with the reasoning inline, and a note that
single-bucket scoping is still possible for anyone willing to create the
key via B2's CLI/API directly (b2_create_key with an explicit
capabilities list including listBuckets) rather than the basic web form
this walkthrough is written for.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01H4k6J1qXXyYxhGEgnJaMvn
Two separate fixes from a live report.
1. The DR-spare and SFTP-mirror sections both checked ONLY /root/.ssh
for a key, missing the common case: the person running `sudo
./setup.sh backup` already has a key under their own home directory
(used interactively, quite possibly already authorized on the target
box), while root — who actually runs the scheduled systemd service —
has none. Confirmed live: "the computer has the ssh key for the sudo
user on the box" produced "No SSH key found for root" with no
inline way to do anything about it beyond a pointer to go set one up
elsewhere and re-run.
Factored both call sites into one shared _backup_ensure_root_ssh_key()
that checks root first, then offers to reuse the sudo user's existing
keypair (copied into /root/.ssh with correct ownership/permissions,
root:root 600) before falling back to generating a brand new one —
reusing an existing key can work immediately if it's already
authorized on the target, where a fresh key needs a new ssh-copy-id
round-trip regardless. Verified all three branches (root already has
a key, root has none but the user does and accepts reuse, neither
exists and one gets generated) against a mocked filesystem.
2. The B2 dry-run failure message read like it could be about missing
input even when every field was non-empty — confirmed there's no
code path where non-blank-but-wrong values actually trigger the
separate "Left blank" message (the two are on disjoint branches), but
the dry-run failure text itself didn't rule that out or point at the
actual likely cause. Now echoes back what was entered (bucket,
endpoint, Key ID — never the secret) so it's easy to eyeball against
B2's own confirmation screen, states plainly that this is a rejection
of non-blank input, and names the most likely cause directly: pairing
the Key ID from one Application Key with the Secret from a different
one, which is easy to do after creating more than one while
troubleshooting.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01H4k6J1qXXyYxhGEgnJaMvn
Confirmed live: a real failure ("WARNING: spare sync failed — error —
see system logs on ubuntu") didn't match any of categorize_error()'s
known patterns, fell into its generic catch-all bucket, and the actual
stderr text that would have explained it was sitting in a mktemp'd file
this script deletes on exit (trap ... EXIT) — so there was nothing in
"system logs" to actually go check. The categorization was silently
discarding the one piece of information that would have diagnosed the
problem.
Added log_raw_error(), called right after every categorize_error() site
(5 of them: two snapshot-failure paths, the primary REMOTE_TYPE mirror,
the new EXTRA_MIRROR_NAMES loop, and the DR-spare sync) — logs the raw
stderr text (truncated to 500 chars) into the same log stream as
everything else, so it survives past the run that produced it instead
of being deleted with the temp file. categorize_error()'s short bucket
label is untouched and still used for FAILED_SVCS/notification text,
which should stay concise — this adds the detail alongside it, not
instead of it.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01H4k6J1qXXyYxhGEgnJaMvn
Confirmed: Kopia's sync-to sftp has its own SFTP client and doesn't read
~/.ssh/config the way the system ssh/scp binaries do — so an alias set
up via wg-easy's sync-ssh-aliases.sh (or any ~/.ssh/config Host entry)
worked fine for the DR-spare connectivity check (which shells out to
real ssh) but silently failed for this mirror: a plain @-split on an
alias like "main" (no @ present) produced --host=main, a name that only
resolves inside ~/.ssh/config, not real DNS. The dry-run check correctly
rejected it and the mirror was never saved — no error surfaced beyond
that, so it looked like nothing happened.
Now resolves the destination through `ssh -G` before building the Kopia
flags — the same mechanism ssh itself uses to expand config aliases —
and falls back to the previous plain @-split only if that comes back
empty. Verified against three cases: a bare alias (resolves via a mock
~/.ssh/config Host block), an explicit user@ip (passes through
unchanged), and an unrecognized name (falls back to a sane literal
hostname rather than erroring).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01H4k6J1qXXyYxhGEgnJaMvn
Requested: mirror to Backblaze B2 AND directly to the IONOS spare box
over Tailscale, at the same time, not one or the other. REMOTE_TYPE/
REMOTE_ARGS was hardcoded to a single mirror target — extending it to a
list would have meant redesigning the one thing that already works and
was already verified against real B2 credentials, so this adds a
separate, additive mechanism instead: EXTRA_MIRROR_NAMES, a space-
separated list, with per-entry MIRROR_<name>_TYPE/_ARGS (same argument
shape as REMOTE_ARGS). An existing B2-only backup.conf keeps working
completely unchanged if this new section is skipped.
install_backup() gets a new "ADDITIONAL MIRROR" prompt after the
existing B2 section: offers a direct SFTP mirror (Kopia's sync-to sftp,
not the deprecated b2 provider — same reasoning as the S3/B2 choice
already made), defaults the destination to whatever was typed at the
DR-spare prompt above (same box, same purpose, no reason to ask twice),
checks passwordless SSH and an SSH key exist first, then verifies with a
--dry-run against the just-created 'default' repo before saving it —
same "don't save something broken" discipline as the B2 flow. Verified
against a mock backup.conf that install-side writes and worker-side
reads agree on the exact format, and that reusing an existing mirror
name reconfigures it instead of duplicating it in the name list.
One correction while researching sync-to sftp's flags: unlike plain ssh,
Kopia doesn't shell out to the system SSH client, so it needs an
explicit --keyfile and --known-hosts path rather than picking up
whatever `ssh` already trusts automatically — checked Kopia's own docs
for the exact flags before writing this, same as the earlier S3 case.
extras/backup_kopia.sh's worker loops through EXTRA_MIRROR_NAMES after
the existing REMOTE_TYPE mirror step, running sync-to for each
destination against each additional mirror and folding failures into
the same FAILED_SVCS/notification reporting the primary mirror already
uses. Verified end-to-end against a mock backup.conf and a stubbed
kp_for: both the B2 and the new SFTP mirror get called in sequence with
the correct arguments.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01H4k6J1qXXyYxhGEgnJaMvn
Requested after a live failure: pasting into the hidden Application Key
field silently captured nothing (terminal/SSH-client dependent), and the
only symptom was a generic "one or more fields left blank" warning after
all four prompts had already gone by — no way to tell which field, or
even that the paste itself was the problem rather than something else.
Each of the four fields now echoes its character count right after entry
(never the value for the hidden Application Key field, just its length),
so a failed paste is visible immediately instead of discovered several
prompts later. The blank-field warning now also names exactly which
field(s) were empty instead of a generic message.
Verified against the user's actual reported case: bucket/endpoint/key-ID
entered normally, Application Key came back empty — reproduces as
"(0 characters entered)" on that line and "Left blank: Application Key"
in the warning, both confirmed against a second case where all four
fields are present and it passes through cleanly.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01H4k6J1qXXyYxhGEgnJaMvn
Requested: don't push the operator toward installing wg-easy if they
already have a different mesh VPN (Netbird or Tailscale) running —
detect any of the three first, and only offer a choice when none are
present.
Detection checks wg-easy's own directory (this repo's install marker),
then falls back to checking whether the netbird/tailscale binaries exist
AND their systemd services are actually active — not just installed,
since an installed-but-never-connected client isn't a usable path to the
spare box either. wg-easy takes priority if somehow more than one is
present, since it's this repo's own chain-installable option.
When none are detected, offers a numbered choice: wg-easy (chain-installs
via the existing declare -F guard), Netbird, or Tailscale (both via their
official curl-pipe-sh installers — verified the current URLs against
each vendor's own docs rather than guessing, since a wrong URL here would
be a bad thing to ship). Both third-party options still need a manual
follow-up step this script can't complete unattended (Netbird needs a
setup key from the operator's account, Tailscale needs an interactive
auth link) — the success message says so rather than implying the
install alone finishes the job.
Verified the detection branching against all the cases that matter:
nothing present, only wg-easy's directory, only Netbird active, only
Tailscale active, and multiple present at once (wg-easy correctly wins).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01H4k6J1qXXyYxhGEgnJaMvn
Requested improvement: the disaster-recovery spare prompt in backup.sh
already ran a live connectivity check and, on failure, printed manual
instructions (set up wg-easy separately if the spare isn't reachable,
run ssh-keygen/ssh-copy-id yourself) — but never offered to do any of it
right there, even though every piece is safe to automate inline.
Now, when the passwordless SSH check fails:
- If wg-easy isn't installed yet, offers to chain-install it (guarded
with declare -F install_wg-easy, same pattern asterisk.sh already uses
for security-dashboard/pstn-trunk) — covers the common case where the
spare is a home box with no port-forward and no path there at all yet,
not just a missing key.
- If root has no SSH key, offers to generate one (ssh-keygen -t ed25519).
- Offers to run ssh-copy-id against the spare interactively right there
— it prompts for the spare's login password itself, so this script
never touches or sees that password, just invokes the real command
inline instead of telling the operator to go run it themselves after.
- Re-runs the connectivity check after ssh-copy-id succeeds, so the
install flow reports the actual current state instead of the
pre-fix failure message.
Verified the has-a-key detection (the part most likely to have a subtle
&&/|| precedence bug) against all four cases — no key, only id_ed25519,
only id_rsa, both — behaves correctly in each.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01H4k6J1qXXyYxhGEgnJaMvn
Confirmed live: backup.sh has no update/fresh distinction and re-runs
every prompt on every invocation, including the repository password
prompt — which always minted a fresh (typed or auto-generated) password
regardless of whether a repo already existed at that destination's path.
Re-running the installer (to add a destination, configure the new B2
offsite mirror, or just by habit) then fails to connect to the real,
already-populated repo with "invalid repository password", because the
repo's actual password is permanently whatever was set the first time
and nothing read that back.
Each destination's password is now read back from the existing
backup.conf (if that destination name was already configured there)
before falling through to prompt/auto-generate — same pattern already
applied to REMOTE_TYPE/REMOTE_ARGS, EMBEDDED_COTURN_SLOT, and everywhere
else in this session that re-running a script with no update/fresh gate
turned out to silently regenerate something it shouldn't have. Verified
against a mock backup.conf: an existing destination's password is reused
verbatim, and a genuinely new destination name still falls through to
fresh generation correctly.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01H4k6J1qXXyYxhGEgnJaMvn
Answers a direct ask: offsite mirroring existed only as a REMOTE_TYPE/
REMOTE_ARGS placeholder in backup.conf with a comment pointing at
`kopia repository sync-to --help` — no interactive setup at all, B2 or
otherwise.
Checked before building anything: Kopia's dedicated `sync-to b2`
provider is marked [DEPRECATED] on kopia.io's own command reference.
B2 also offers an S3-compatible endpoint (s3.<region>.backblazeb2.com,
same application key works as the access/secret key pair), and Kopia's
`sync-to s3` provider isn't deprecated — so this targets that path
instead of building on a command on its way out.
What's now automated vs. guided, deliberately split:
- Bucket creation and the application key are walked through as console
steps, not automated. Object Lock specifically is a one-time,
bucket-creation-only decision with a real tradeoff (undeletable-by-
design vs. genuinely can't delete early) that shouldn't be silently
flipped either way by a script on someone's behalf.
- Once the operator has a bucket + endpoint + scoped application key
(B2 requires a key scoped to one bucket, not the account master key —
noted in the walkthrough), this becomes mechanical: run a
`sync-to s3 --dry-run` against the just-created 'default' repo to
verify the credentials actually work, and only then write
REMOTE_TYPE=s3 / REMOTE_ARGS into backup.conf. A bad bucket name or
key leaves REMOTE_TYPE at "none" with a clear error instead of saving
a broken config that fails silently at 2am.
- Encryption isn't a separate step — Kopia already encrypts client-side
with the repository password set earlier in this same flow; called
that out explicitly since it was asked about as if it needed its own
setup step.
Also fixed a regression the new prompt would otherwise have caused:
backup.sh has no update/fresh distinction and re-asks everything on
every run, so an already-configured offsite mirror is now read back
from the existing backup.conf and preserved by default — answering "no"
on a re-run no longer silently resets REMOTE_TYPE to "none".
Verified the control flow (not just bash -n) against a mock kopia
binary and stubbed prompts: good credentials wire up REMOTE_TYPE/
REMOTE_ARGS correctly, a rejected credential leaves REMOTE_TYPE at
"none" rather than saving something broken, an existing configured
value survives a "no" answer on re-run, and blank fields skip cleanly
without attempting a dry-run at all.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01H4k6J1qXXyYxhGEgnJaMvn
Answers a direct ask: the automated restore-verify test
(extras/test_backup_kopia.sh — verifies the latest snapshot, restores it
over a moved-aside copy, compares, rolls back, reports PASS/FAIL, sends
an ntfy notification) was already fully non-interactive and already
wired to a systemd timer/cron fallback by install_backup() — it just
had no schedule choice at all, hardcoded to weekly (Saturday 03:00).
Every service in this test stops briefly while its data gets moved
aside and restored back, same interruption profile as the main backup
job — so the schedule is a real tradeoff (more frequent verification vs.
more frequent blips), not a free "always pick the most frequent" choice.
Gave it the same Weekly/Monthly/Custom shape the main backup schedule
prompt above it already offers, instead of a single hardcoded option.
Also added an explicit "run the first test now?" prompt right after
scheduling it — otherwise choosing Monthly means waiting up to a month
before finding out whether the test even works, rather than getting
that initial confirmation immediately and then settling into the
chosen cadence.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01H4k6J1qXXyYxhGEgnJaMvn
Found while adding a live scanner to the coturn-slot code: WEB_PORT and
CALLS_UDP_PORT were scanned unconditionally, before the reinstall-mode
prompt even ran and before anything stopped the currently-running
container. On an "Update" run that meant find_free_port would see this
instance's OWN already-published port as occupied and silently shift it
to the next free one — every plain update could have moved the service's
port out from under already-configured Caddy routes, bookmarks, and the
Calls plugin's client config, without the operator asking for that.
services/asterisk.sh already gets this right for WEB_ADMIN_PORT: update
reads the existing port back from .env (no rescan), fresh scans from the
plain default only after stopping the old container. Brought Mattermost
in line with the same shape — the port resolution moved from before the
reinstall-mode block to after it, so MODE is known and, for a fresh
install/"Full reinstall", the old containers are already stopped by the
time it scans.
WEB_PORT/CALLS_UDP_PORT are now also written to .env directly (they
weren't before), with a fallback to parse them from the existing
MM_SERVICESETTINGS_LISTENADDRESS / docker-compose.yml port mapping for
installs made before this change — so an update on an already-running
instance doesn't regress just because its .env predates the new
variables. Verified the explicit-var, fallback-parse, and priority-order
(explicit wins over fallback) cases against a mock before shipping.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01H4k6J1qXXyYxhGEgnJaMvn
Requested check: the slot-allocation scheme added in the previous commit
only checked against OTHER mattermost*/.env files on the box, not
against what's actually listening. A slot whose numbers happen to be
free by that bookkeeping could still be squatted by something this
script doesn't track (a manually-run process, an unrelated service) —
this box already learned that lesson once, from Asterisk and
Mattermost's embedded coturn ranges overlapping without either side
knowing.
Only a NEWLY assigned slot gets the live check — an already-cached slot
(read back from this instance's own .env) is trusted as-is, since a live
conflict on an already-configured, already-running instance's own port
is a real problem to report, not something to silently route around by
moving that instance's TURN port out from under it.
Can't scan the full 200-port relay range port-by-port (large ranges use
the offset scheme instead of scanning per CLAUDE.md's port-collision
section) — checks the control port plus both relay-range boundaries as
the practical middle ground. Verified against a mock: a candidate slot
whose control port is already bound gets skipped in favor of the next
free one.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01H4k6J1qXXyYxhGEgnJaMvn
Follow-up to the Asterisk/Mattermost relay-range overlap fix: that fix
only handled the two-service collision, and left a documented gap for
what happens when a second (or third...) Mattermost instance also falls
back to embedded coturn — they'd have collided with each other on the
same fixed 3479/49253-49452 numbers, same bug, different pair.
find_free_port-style scanning doesn't work for the relay range itself —
it's a scan for a single free port, not a free contiguous 200-port
block — so this follows the same fixed-offset-per-instance approach
CLAUDE.md documents for traccar.sh's large port range instead. Each
instance gets an integer slot (control port = 3479 + slot, relay range
= 49253 + slot*200 through +199) computed once as the smallest slot
number not already claimed by another mattermost*/.env on the box, then
cached in that instance's own .env as EMBEDDED_COTURN_SLOT so it reads
back the same value on every later update or full reinstall instead of
potentially landing on a different slot (which would silently move an
already-configured instance's TURN port out from under it — the same
"never touch what's already the box's answer" rule everything else in
update mode already follows).
Verified the allocation logic against a mock: first instance gets slot
0, a second gets slot 1 without stepping on the first, both instances
keep their own slot across a simulated re-run, and a third new instance
correctly lands on the next free slot (2) rather than reusing either.
Threaded the computed port/range through every place that used to
hardcode 3479/49253/49452: the coturn compose block, the UFW rule
(now also labeled with the instance suffix, matching this file's other
UFW comments), and the Calls-plugin TURN config text in the generated
README/System-Console instructions.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01H4k6J1qXXyYxhGEgnJaMvn
Confirmed live on a box that retired the shared coturn service in favor
of each service running its own dedicated/embedded coturn permanently:
Mattermost's embedded-coturn fallback used relay range 49153-49352,
which overlaps Asterisk's embedded coturn range (49152-49252) by ~100
UDP ports. Both run network_mode: host, so with shared coturn out of
the picture this is the exact same collision CLAUDE.md documents as the
original, already-fixed-once bug that the shared coturn service was
built to solve in the first place — reintroduced here because
Mattermost's embedded-coturn fallback path apparently never got checked
against Asterisk's numbers when it was written.
Moved Mattermost's embedded relay range to 49253-49452 (same 200-port
width, now contiguous with and non-overlapping Asterisk's 49152-49252).
Updated the docker-compose command flags, the matching UFW rule, and
added a comment explaining the offset so it doesn't drift back into
collision — and noting the known residual gap this doesn't cover: two
Mattermost instances *both* falling back to embedded coturn at once
would still collide with each other on these same fixed numbers. Not
fixed here since it requires more than one Mattermost instance to be
running without shared coturn at the same time, which isn't this box's
situation; flagged in-code for whoever hits it.
Also made asterisk.sh's generated README port table stop unconditionally
claiming a TURN relay range it isn't actually publishing when the shared
coturn service (not this install's own container) is fronting TURN
instead — it now branches on USE_EMBEDDED_COTURN, which the function
already receives as a parameter.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01H4k6J1qXXyYxhGEgnJaMvn
Extends the security-dashboard prototype to the shared-coturn trio, since
these three are exactly the case that pattern was built for — a fresh
reinstall of any of them today just overwrote files in place without
stopping old containers first, and coturn's own fresh path never made an
informed choice about the consumer credentials/database it happens to
leave alone (safe today, but by omission rather than design).
- asterisk.sh / mattermost.sh: "Full reinstall" now stops the existing
containers (`docker compose down`) before falling through to the normal
install flow, and asks a single explicit question — delete stored data
(PBX config/spool/voicemail for Asterisk; Postgres db/uploads/config/
plugins for Mattermost) — defaulting to preserve. Their shared-coturn
TURN credential is deliberately left alone either way (reused from
cache via ensure_coturn_user(), same as update) — it's not this
service's own data, and coturn already handles that continuity.
Mattermost's existing "_db_has_data" check already reads the
filesystem to decide whether to reuse or regenerate DB_PASS, so the
wipe/preserve choice composes with that for free — no separate flag
needed. Asterisk's warns to re-run pstn-trunk afterward if data is
wiped, since that's what actually goes stale (its dialplan patch),
not the fabricated "AMI secret" framing an earlier draft of this
warning used before I checked the actual code.
- coturn.sh: "Full reinstall" now lists which consumers are currently
registered (from users/*.env) and asks explicitly whether to also
wipe TURN credentials and the user database, instead of silently
preserving them as an unexamined side effect of never deleting the
directory. Defaults to preserve. If the operator does choose to wipe,
the running container is restarted afterward — it holds the old,
now-deleted turndb file open, so new turnadmin writes to the fresh
file would otherwise go unseen until a restart anyway. Every affected
consumer already self-heals a missing credential on its own next
Update run via ensure_coturn_user()'s existing cache-miss path — no
changes needed there, just confirmed it covers this case.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01H4k6J1qXXyYxhGEgnJaMvn
Two-part change discussed and scoped in this session before touching
anything:
1. Rename "Reinstall in place" (r) -> "Update" (u) and "Full install" (f)
-> "Full reinstall" everywhere the prompt appears: lib/common.sh's
shared prompt_reinstall_mode(), plus the three services that carry
their own duplicated standalone-stub copy of it for standalone
execution (asterisk.sh, coturn.sh, wordpress.sh — per this repo's
documented standalone-bootstrap pattern). Internal state values
(update/fresh/cancel) are unchanged, so no other service's case
statement needed touching. docs/anveo-direct-setup-guide.md's `r`
reference updated to `u` to match. attic/asterisk-digital-ocean.sh
deliberately left alone — this repo's own policy is to not backport
fixes into attic/.
2. security-dashboard.sh's "Full reinstall" now does a real teardown
before reinstalling — stops and removes the systemd unit, sudoers
grant, Caddy site block, and secdash system user, then proceeds
through the normal fresh-install flow — instead of just overwriting
files in place while leaving the old service running underneath.
Prototype for a pattern discussed for other services later: split the
destructive question out explicitly ("also delete
dashboard-admins.conf — per-admin extension scoping?", default n) so
full reinstall doesn't silently discard state a plain "start over"
request wouldn't expect to lose. Verified the backup/restore mechanics
(mktemp, copy out before teardown, copy back after) against a mock
under `set -u` for both the preserve and wipe paths before shipping.
Update mode was already the strongest existing example of surfacing
newer optional prompts (its "Reconfigure Caddy protection?" /
"Reconfigure per-admin scoping?" sub-prompts already cover every setting
fresh-install offers) — no changes needed there for this service.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01H4k6J1qXXyYxhGEgnJaMvn
Direct follow-up to the previous commit's ensure_coturn_user() fix: that
fix is useless for Asterisk specifically unless something actually calls
ensure_coturn_user("asterisk") again, and the update ("Reinstall in
place") branch returns 0 well before the fresh-install path's call to it
— only "Full install" reached it, which re-prompts everything (droplet
detection, domain, etc.) just to fix a credential re-registration.
Added the same call to the update path, gated on NOT having an embedded
coturn (checked via the existing _HAD_EMBEDDED_COTURN detection) — calling
it unconditionally would silently chain-install the shared coturn service
for a box deliberately running Asterisk's own dedicated coturn, exactly
the kind of silent update-time migration CLAUDE.md's coturn guidance
warns against. .env stays untouched either way (self-heal re-registers
with the same cached password, never generates a new one).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01H4k6J1qXXyYxhGEgnJaMvn
Answers a direct question from this session: no, reinstalling
asterisk/mattermost did NOT fix a coturn user missing from the live
database, because ensure_coturn_user() only ever calls turnadmin -a in
the else branch — reached only when the cache file (users/<consumer>.env)
is MISSING. A stale-but-present cache file (exactly what a coturn
container/volume recreation without preserving ./db leaves behind, per
this session's real diagnosis) looked identical to a healthy one and was
trusted blindly, so every consumer's installer kept silently reusing
credentials that no longer existed in coturn's database.
Now checks the cached username against coturn's actual live user list on
every call, and re-registers it with the same cached password if it's
missing — the same self-heal pattern this repo already applies elsewhere
(Beszel's compose patch, Vaultwarden's SMTP half-state, FMD's chown).
Re-uses the turnadmin -l log-noise filter from tools/coturn-test-check.sh
(a real "user[realm]" line never contains a space; at least one coturn
build writes its own startup log lines to stdout, not stderr, so a bare
2>/dev/null doesn't catch them).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01H4k6J1qXXyYxhGEgnJaMvn
Live run surfaced it: this coturn build writes its own startup log lines
("INFO SQLite connection was closed.", "INFO log file opened: ...") to
turnadmin -l's STDOUT, not stderr — 2>/dev/null never caught them, so
they got parsed as if they were usernames, producing nonsensical
"Database has user '2026-...INFO SQLite connection was closed.'" warnings
on a real run. A genuine "user[realm]" line never contains a space; every
log line does, so filtering on that is a simple, build-independent fix.
Also diagnosed the actual underlying failure this surfaced: coturn's live
user database was genuinely empty (both 'asterisk' and 'mattermost' had
cached credential files but neither was registered in the DB) — exactly
the container/volume-recreated-without-db drift this script's consumer
cross-check exists to catch, confirmed against a real run.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01H4k6J1qXXyYxhGEgnJaMvn
Latest live run showed the test getting killed by its own `timeout 10`
before turnutils_uclient printed any result — just two startup INFO
lines, no error. That's the coturn/coturn Docker image's turnutils_uclient
(apparently a newer build with structured "LEVEL component: message"
logging, different from the older packaged version available for local
testing) taking longer than 10s to complete, not a real failure.
Bumped both scripts' timeout to 20s, and now check for timeout(1)'s own
exit code (124) separately from a real reported error — reported as WARN
with a suggested manual command to re-run with more time and see the full
result, instead of lumping "still running" in with "actually failed."
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01H4k6J1qXXyYxhGEgnJaMvn
Not another guess: installed coturn locally (apt-get install coturn) and
ran the actual server + turnutils_uclient against it to verify this
before shipping, since the last two rounds shipped based on reading the
usage text alone and both turned out incomplete.
-e 127.0.0.1 satisfies turnutils_uclient's "-e or -y required" check, but
then fails allocation with "channel bind: error 403 (Forbidden IP)" —
services/coturn.sh never sets --allow-loopback-peers, so loopback as a
peer address is correctly rejected by a real coturn instance, and the
previous fix's own comment about "loopback is always reachable" missed
that reachable and permitted aren't the same thing.
-y ("client-to-client") sidesteps this: it negotiates both ends of a real
relay through the server itself, needs no separate peer address, and
works fine over loopback. Verified directly against a real local
instance: exits 0 with real packet-loss/RTT stats on valid credentials,
and correctly fails ("Cannot complete Allocation", exit 255) on a wrong
password — so it's still a meaningful pass/fail, not just "didn't crash."
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01H4k6J1qXXyYxhGEgnJaMvn
Another real failure from a live run: turnutils_uclient refuses to run at
all without either -e <peer> or -y — a bare auth-only invocation isn't
enough for it to actually attempt anything. Add -e 127.0.0.1 to both
tools/pstn-test-check.sh's and tools/coturn-test-check.sh's invocations;
loopback is always reachable since the test already runs via `docker exec`
inside the coturn container itself, and it lets the test actually prove
data relays through the allocation, not just that auth succeeded.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01H4k6J1qXXyYxhGEgnJaMvn
Real bug caught from a live run: the coturn allocation test passed -t -T
(TCP/TLS) to turnutils_uclient, but services/coturn.sh always starts
coturn with --no-tls --no-dtls — requesting an encrypted/TCP transport
against a server that never offered one fails the allocation outright
("Cannot complete Allocation"), misreporting a config problem that didn't
exist. Dropped both flags in both tools/pstn-test-check.sh and
tools/coturn-test-check.sh so the test matches what the server actually
supports (plain UDP).
Also, from user feedback on the same run:
- warn()/fail() now collect their messages into arrays; the Summary
section prints a "Needs attention" recap of every FAIL/WARN together
at the end, instead of leaving the user to scroll back through a long
run to find what needs fixing.
- The softphone-setup block now offers to reprint itself one extension
at a time (paced with a keypress between each) after the main run, so
a long device list isn't lost in the scrollback either. Factored the
per-extension print into print_ext_info() so the full run and this
reprint can't drift apart. Guarded with `[ -t 0 ]` so it's skipped
automatically when the script isn't run interactively.
Verified via a fuller mock harness (fake docker/curl/systemctl/getent,
non-TTY stdin) that: the corrected turnutils_uclient invocation reports
success, the recap correctly lists FAIL before WARN, and the interactive
reprint prompt is skipped without hanging when stdin isn't a terminal.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01H4k6J1qXXyYxhGEgnJaMvn
Two follow-ups on the PSTN health check:
- New "coturn (TURN relay for Asterisk)" section reads Asterisk's own
TURN_* values from its .env (not re-derived) and runs a live TURN
allocation against whichever coturn Asterisk is actually configured to
use — the shared instance, or its own embedded per-Asterisk coturn if
that's what this box has (detected via the same "grep -q '^ coturn:'
docker-compose.yml" check CLAUDE.md's migration guidance describes).
Proves what Asterisk itself would use at call time, complementing
tools/coturn-test-check.sh's broader multi-consumer check.
- New "Softphone setup" section parses pjsip.conf directly and prints
per-extension SIP server/username/password/port/transport, plus TURN
credentials for any extension with ice_support=yes — the same values
Sipnetic's "Add Account" screen needs, computed here so a client isn't
installed just to read them out of the Security Dashboard.
Also fixed a bug caught while building a mock test harness to verify both
additions: the extension-registration parser grabbed state via a fixed
field position ($3), silently truncating multi-word states like "Not in
use" down to "Not". Replaced with a regex that captures everything
between the extension and the trailing "N of inf" — verified against both
single- and multi-word states.
And a real syntax bug caught by bash -n before this ever shipped: an
apostrophe inside a ${VAR:-default} expansion ("this box's IP") opens an
unterminated single-quote context even inside double quotes — reworded
to avoid the apostrophe entirely rather than fight bash's parser.
Full mock run (fake docker/curl/systemctl/getent, real pjsip.conf/.env
fixtures matching the actual generated format) confirmed both new
sections and the registration fix all produce correct output end to end.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01H4k6J1qXXyYxhGEgnJaMvn
Three follow-ups from live testing on this session's actual VPS:
- tools/pstn-test-check.sh's SMS section printed the Forward-to-URL value
to configure but never said what to do next — add the "text this DID,
then watch journalctl -u sms-inbound -f" step right after it.
- docs/pstn-sms-test-checklist.md: the "which box actually handled this"
question has a simple answer (a DID's inbound routing targets exactly
one IP:port, so there's no ambiguity to resolve, only a portal setting
to confirm) — written up so it doesn't need re-deriving. Also fixed the
--list example to cd into the repo first; ./setup.sh is a relative path
and silently fails with "command not found" from any other directory,
confirmed live in this session.
- New tools/coturn-test-check.sh: health-checks the shared coturn instance
(services/coturn.sh) and every consumer registered against it (Asterisk,
any number of Mattermost instances) — container/identity, each cached
consumer credential cross-checked against coturn's own live user
database (catches the container/volume-recreated-without-db drift case),
UFW rules for both the TURN port and the relay range, a capacity
explanation reasoned from the actual port-range math instead of a guess,
and a real TURN allocation test per consumer via turnutils_uclient —
the only way to prove credentials + port range + firewall all actually
work together, not just that each looks right in isolation. Deliberately
does not attempt a concurrent load test, since that would consume real
relay ports other services may be actively using.
Verified the turnadmin -l output parsing, UFW rule matching, and the
empty-array-under-set–u loop pattern against mock data before shipping.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01H4k6J1qXXyYxhGEgnJaMvn
Server-side config was fully verifiable already; what wasn't is the
provider-account side (Anveo's authorized-IP list, DID routing, SMS
forward-URL) since that lives entirely outside this box. Rather than
leave "go check the portal" as a vague pointer, compute and print the
exact values each portal field needs to match: this box's public IP,
the trunk DID (from .pstn-trunk.env), and the SMS forward URL read
straight from /opt/sms-inbound/settings.env (SMS_FORWARD_URL) instead of
making the user reconstruct or hunt for a value the installer already
generated and stored. Anveo-specific field-by-field checklist when
PROVIDER_NAME matches; generic fallback otherwise.
Verified the .pstn-trunk.env / settings.env sourcing against mock files
matching the real generated format, including the literal $[from]$-style
Anveo placeholders in SMS_FORWARD_URL, which must survive `source` under
`set -u` without triggering bash's legacy $[...] arithmetic expansion —
same guard pattern services/pstn-trunk.sh's own update path already uses.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01H4k6J1qXXyYxhGEgnJaMvn
docs/pstn-sms-test-checklist.md's manual steps (registration, trunk
reachability, dialplan contexts, kill-switch state, usage-alert timer
health, recent call/message activity) are all things a script can check
directly instead of re-typed by hand each time — and re-typing them is
exactly what led to the container-name mistake in the prior commit.
tools/pstn-test-check.sh auto-detects the container/directory the same
way the checklist doc now does, runs every automatable check, and prints
PASS/WARN/FAIL per item plus a summary. What it can't cover — actually
placing a call or sending a text — still needs the checklist doc.
Caught during testing against real command output pasted in this
session: the endpoint-parsing loop matched pjsip's own column-header
line ("<Endpoint/CID...> <State...>") as if it were a real endpoint row,
producing a bogus result. Fixed by skipping any row whose parsed
extension starts with "<".
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01H4k6J1qXXyYxhGEgnJaMvn
Claude