Two stacked bugs, found together when re-running the backup installer to
add an SFTP mirror silently reverted a previously-set absolute
DR_SYNC_PATH back to the script's tilde-based default, which then failed
outright:
1. services/backup.sh never read DR_SYNC_HOST/DR_SYNC_PATH back from an
existing backup.conf before prompting (every other setting in this file
does — passwords, mirrors). Accepting the prompt defaults on a rerun
silently reset both to blank/"~/docker/backup" instead of keeping what
was already configured. Fixed by reading them back the same way
DEST_*_PASSWORD already does.
2. extras/backup_kopia.sh's DR-spare sync wraps the remote path in single
quotes for its `ssh host "mkdir -p '...'"` / `"chmod 600 '.../...'"`
commands. Single-quoting a leading ~ stops the remote shell from
expanding it at all, so it looked for a literal directory named "~"
instead of the home directory — breaking the script's own DEFAULT
DR_SYNC_PATH ("~/docker/backup") for anyone who actually used it.
rsync's own transfer step has separate, correct tilde handling, which is
why the sync itself "succeeded" while the follow-up chmod couldn't find
the file. Fixed with a small _dr_remote_quote() helper that keeps a
leading ~/ outside the quotes while still safely quoting the rest of
the path.
Verified the quoting fix by parsing the exact constructed command string
in bash directly — a plain '~/docker/backup' stays literal (the bug),
~/'docker/backup' correctly expands to $HOME/docker/backup (the fix).
The freshly-added raw-error logging paid off immediately: the box's spare
sync was failing every run with "scp: Connection closed" while plain ssh
exec to the same host worked fine. That split (ssh exec OK, scp specifically
rejected) matches modern OpenSSH's default scp-over-SFTP transfer hitting a
restriction on the remote side that a plain exec or rsync's own protocol
don't trigger.
Swapped the scp step for rsync -a over the same ssh options, keeping the
ssh mkdir -p before it (rsync doesn't create missing destination
directories) and the ssh chmod after. Verified the exact command/quoting
against mocked ssh/rsync binaries — array expansion and remote path
handling both check out.
Last night's fully-failed backup run (0/20, "repository not found" on
every service) showed the real gap: categorize_error() clearly saw real
content in $_ERR (it matched a specific pattern, not the generic
fallback), but log_raw_error()'s separate re-read of the same file moments
later came back empty on every single failure — so the raw-error logging
added earlier this session produced nothing when it mattered most.
Fixed by reading $_ERR into a variable exactly once per failure and
passing that string to both categorize_error() and log_raw_error(),
instead of two independent file reads. Verified against a mock harness
reproducing the same call pattern (three simulated failures in a loop,
single shared error file) — both the categorized reason and the raw
stderr text now come through on every iteration.
Doesn't explain why last night's repo access failed in the first place
(disk and mount checks came back clean) — but the next time it happens,
this will actually surface the real kopia error instead of losing it.
Confirmed live: a real failure ("WARNING: spare sync failed — error —
see system logs on ubuntu") didn't match any of categorize_error()'s
known patterns, fell into its generic catch-all bucket, and the actual
stderr text that would have explained it was sitting in a mktemp'd file
this script deletes on exit (trap ... EXIT) — so there was nothing in
"system logs" to actually go check. The categorization was silently
discarding the one piece of information that would have diagnosed the
problem.
Added log_raw_error(), called right after every categorize_error() site
(5 of them: two snapshot-failure paths, the primary REMOTE_TYPE mirror,
the new EXTRA_MIRROR_NAMES loop, and the DR-spare sync) — logs the raw
stderr text (truncated to 500 chars) into the same log stream as
everything else, so it survives past the run that produced it instead
of being deleted with the temp file. categorize_error()'s short bucket
label is untouched and still used for FAILED_SVCS/notification text,
which should stay concise — this adds the detail alongside it, not
instead of it.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01H4k6J1qXXyYxhGEgnJaMvn
Requested: mirror to Backblaze B2 AND directly to the IONOS spare box
over Tailscale, at the same time, not one or the other. REMOTE_TYPE/
REMOTE_ARGS was hardcoded to a single mirror target — extending it to a
list would have meant redesigning the one thing that already works and
was already verified against real B2 credentials, so this adds a
separate, additive mechanism instead: EXTRA_MIRROR_NAMES, a space-
separated list, with per-entry MIRROR_<name>_TYPE/_ARGS (same argument
shape as REMOTE_ARGS). An existing B2-only backup.conf keeps working
completely unchanged if this new section is skipped.
install_backup() gets a new "ADDITIONAL MIRROR" prompt after the
existing B2 section: offers a direct SFTP mirror (Kopia's sync-to sftp,
not the deprecated b2 provider — same reasoning as the S3/B2 choice
already made), defaults the destination to whatever was typed at the
DR-spare prompt above (same box, same purpose, no reason to ask twice),
checks passwordless SSH and an SSH key exist first, then verifies with a
--dry-run against the just-created 'default' repo before saving it —
same "don't save something broken" discipline as the B2 flow. Verified
against a mock backup.conf that install-side writes and worker-side
reads agree on the exact format, and that reusing an existing mirror
name reconfigures it instead of duplicating it in the name list.
One correction while researching sync-to sftp's flags: unlike plain ssh,
Kopia doesn't shell out to the system SSH client, so it needs an
explicit --keyfile and --known-hosts path rather than picking up
whatever `ssh` already trusts automatically — checked Kopia's own docs
for the exact flags before writing this, same as the earlier S3 case.
extras/backup_kopia.sh's worker loops through EXTRA_MIRROR_NAMES after
the existing REMOTE_TYPE mirror step, running sync-to for each
destination against each additional mirror and folding failures into
the same FAILED_SVCS/notification reporting the primary mirror already
uses. Verified end-to-end against a mock backup.conf and a stubbed
kp_for: both the B2 and the new SFTP mirror get called in sequence with
the correct arguments.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01H4k6J1qXXyYxhGEgnJaMvn
- dr_bringup.sh: bound every kopia call and docker compose up with a
timeout so one stuck service can't stall the rest of the batch, and
only exit non-zero if literally nothing came up — a partial recovery
is a partial success, not a failed run.
- backup_kopia.sh: optional DR_SYNC_HOST/DR_SYNC_PATH in backup.conf
scp's backup.conf + README.md to a spare box over SSH after every
successful backup, so dr_bringup.sh is ready there with no manual
copy step.
- backup.sh: prompts for the spare's SSH destination, verifies
connectivity at install time instead of failing silently at 2am, and
writes ~/docker/backup/README.md (this service never had one) so the
synced copy documents every command listed above.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NQkdAn3iG5A4WoqU9FHMaN
- extras/test_backup_kopia.sh: stop → restore → compare → roll back test for Kopia
- extras/test_backup_borg.sh: stop → extract → compare → roll back test for Borg
- backup workers: timing (duration), service count, and pre-flight disk check (< 512 MB warns)
- backup workers: ntfy notifications include count, duration, and per-failure detail
- services/backup.sh: install test_backup_kopia.sh + optional weekly test timer
- services/borg-backup.sh: install test_backup_borg.sh + optional weekly test timer
extras/test_backup.sh — new unified test script (Kopia + Borg):
• Stops container, moves live data aside, restores latest backup,
compares restored vs live with diff -rq (content, not timestamps),
moves live data back and restarts container
• PASS = restore succeeded; diff output is informational (files changed
since last backup are normal)
• FAIL = restore command failed or target empty after restore
• --list flag, CLI service arg, interactive picker
• Handles both full-service dirs and sub-path sources (gaming-backup)
• Cleanup trap always restores live data even on error
• Sends ntfy notification on pass and fail
extras/backup_kopia.sh, backup_borg.sh, backup_gaming.sh:
• ntfy_send() + categorize_error() helpers added
• Each snapshot/archive failure captures stderr and categorizes:
disk full, remote unreachable, repository not found, wrong passphrase,
permission denied, unknown error
• Single ntfy notification at end: success (low priority) or failure
(urgent) with per-service failure reasons listed
• backup_borg.sh: changed 2>&1 | pipe to 2>"$_ERR" | so stdout logs
cleanly and stderr is captured for error categorization
services/backup.sh, borg-backup.sh, gaming-backup.sh:
• New ntfy prompt section in installer (URL + optional token)
• NTFY_URL / NTFY_TOKEN written to backup.conf
• test_backup.sh copied from extras/ into service dir
• Summary updated to show test_backup.sh commands and ntfy URL
https://claude.ai/code/session_019XgsQ13XKm4Zj3cNsDNwHj
Replace embedded heredocs in the three backup service installers with
cp from versioned source files in extras/:
extras/backup_kopia.sh — Kopia worker (was inline in services/backup.sh)
extras/backup_borg.sh — Borg worker (was inline in services/borg-backup.sh)
extras/backup_gaming.sh — gaming saves worker (was inline in services/gaming-backup.sh)
extras/restore_kopia.sh — unified Kopia restore (multi-dest + single-dest)
extras/restore_borg.sh — unified Borg restore with destination picker
Each installer now does `cp extras/<script>.sh $DIR/<script>.sh` instead of
writing the script inline. Workers and restore scripts are now readable in the
repo rather than buried in heredocs.
Restore scripts are installed flat into the service directory root:
~/docker/backup/backup_kopia.sh ~/docker/backup/restore_kopia.sh
~/docker/borg-backup/backup_borg.sh ~/docker/borg-backup/restore_borg.sh
~/docker/gaming-backup/backup_gaming.sh ~/docker/gaming-backup/restore_kopia.sh
The new restore scripts handle destination selection internally, so a single
script replaces the old per-destination restore/<dest>/ layout.
Also fixes `local` used outside a function in restore_kopia.sh and
restore_borg.sh (destination picker loop), and removes the now-superseded
extras/restore_kopia_backup.sh and extras/restore_borg_backup.sh.
https://claude.ai/code/session_019XgsQ13XKm4Zj3cNsDNwHj
Claude