This is what's actually needed for a DigitalOcean -> IONOS Asterisk
migration (item #1 of the user's original 4-item list): move the
whole stack to different hardware entirely, not restore onto the
box the offsite mirror already targets.
dr_bringup_kopia.sh only ever scanned DEST_NAMES for restorable
snapshots — those are always local filesystem Kopia repos
(services/backup.sh creates them with `repository create filesystem
--path=...`), meaning they only exist on whichever box originally ran
the backup. On a genuinely new box, every one of them fails to
connect and there's nothing left to restore from — the script's own
header comment only covered the case where "the spare box IS the box
the primary's mirror targets" (i.e. already holds a copy of the repo
data), not a fresh, unrelated box.
Fix: also try REMOTE_TYPE/REMOTE_ARGS (the offsite Backblaze/S3
mirror, if configured) as a same-shaped destination named "offsite",
reusing DEST_default_PASSWORD since sync-to always mirrors that exact
same encrypted repo. Connects once into a fresh local config file
scoped to this DR run, then folds into the existing per-destination
scan/restore loop unchanged — "offsite" just becomes another entry in
_DEST_ARR. Documented the actual migration workflow in the header
comment, including the BACKUP_CONF override so copying the old box's
backup.conf over doesn't clobber the new box's own freshly-configured
one.
Verified against the real script (not a reimplementation) with mocked
kopia/docker binaries and a crafted backup.conf, covering: local dest
unreachable + offsite connects successfully (the actual migration
shape) with correct service/path discovery; a real (non---list)
restore run confirmed it selects the latest of multiple snapshots by
startTime and issues the correct `kopia restore <snapshot> <path>`
call; offsite connect failing (bad REMOTE_ARGS) warns and degrades to
"no restorable sources found" instead of crashing; REMOTE_TYPE=none
skips the new code path entirely with no behavior change (regression
check against the pre-existing local-only case).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01H4k6J1qXXyYxhGEgnJaMvn
extras/fix_pikapods_dump.py patches two confirmed Adminer PostgreSQL-export
bugs that otherwise make a PikaPods Mattermost migration fail outright:
unquoted enum-label DEFAULT values (Postgres reads the bare label as a
column reference and rejects the CREATE TABLE) and boolean columns
serialized as bare 0/1 instead of true/false (Postgres doesn't implicitly
cast integers to boolean). Boolean columns are discovered by actually
parsing each CREATE TABLE in the dump rather than working from a
hand-curated list — Postgres only reports the first bad column per failed
row, so a list built from error output alone would likely be incomplete.
Already verified earlier this session against a real local Postgres 16
instance; reviewed now for anything needing redaction before committing —
it's a generic text-processing tool with no hostnames, credentials, file
paths, or personal data in it, so nothing needed changing.
Cross-referenced from the generated migrate-from-pikapods.sh's header
comment (services/mattermost.sh) so anyone hitting a CREATE TYPE/CREATE
TABLE or boolean-column import error is pointed at the fix instead of
having to rediscover it.
Two stacked bugs, found together when re-running the backup installer to
add an SFTP mirror silently reverted a previously-set absolute
DR_SYNC_PATH back to the script's tilde-based default, which then failed
outright:
1. services/backup.sh never read DR_SYNC_HOST/DR_SYNC_PATH back from an
existing backup.conf before prompting (every other setting in this file
does — passwords, mirrors). Accepting the prompt defaults on a rerun
silently reset both to blank/"~/docker/backup" instead of keeping what
was already configured. Fixed by reading them back the same way
DEST_*_PASSWORD already does.
2. extras/backup_kopia.sh's DR-spare sync wraps the remote path in single
quotes for its `ssh host "mkdir -p '...'"` / `"chmod 600 '.../...'"`
commands. Single-quoting a leading ~ stops the remote shell from
expanding it at all, so it looked for a literal directory named "~"
instead of the home directory — breaking the script's own DEFAULT
DR_SYNC_PATH ("~/docker/backup") for anyone who actually used it.
rsync's own transfer step has separate, correct tilde handling, which is
why the sync itself "succeeded" while the follow-up chmod couldn't find
the file. Fixed with a small _dr_remote_quote() helper that keeps a
leading ~/ outside the quotes while still safely quoting the rest of
the path.
Verified the quoting fix by parsing the exact constructed command string
in bash directly — a plain '~/docker/backup' stays literal (the bug),
~/'docker/backup' correctly expands to $HOME/docker/backup (the fix).
The freshly-added raw-error logging paid off immediately: the box's spare
sync was failing every run with "scp: Connection closed" while plain ssh
exec to the same host worked fine. That split (ssh exec OK, scp specifically
rejected) matches modern OpenSSH's default scp-over-SFTP transfer hitting a
restriction on the remote side that a plain exec or rsync's own protocol
don't trigger.
Swapped the scp step for rsync -a over the same ssh options, keeping the
ssh mkdir -p before it (rsync doesn't create missing destination
directories) and the ssh chmod after. Verified the exact command/quoting
against mocked ssh/rsync binaries — array expansion and remote path
handling both check out.
Last night's fully-failed backup run (0/20, "repository not found" on
every service) showed the real gap: categorize_error() clearly saw real
content in $_ERR (it matched a specific pattern, not the generic
fallback), but log_raw_error()'s separate re-read of the same file moments
later came back empty on every single failure — so the raw-error logging
added earlier this session produced nothing when it mattered most.
Fixed by reading $_ERR into a variable exactly once per failure and
passing that string to both categorize_error() and log_raw_error(),
instead of two independent file reads. Verified against a mock harness
reproducing the same call pattern (three simulated failures in a loop,
single shared error file) — both the categorized reason and the raw
stderr text now come through on every iteration.
Doesn't explain why last night's repo access failed in the first place
(disk and mount checks came back clean) — but the next time it happens,
this will actually surface the real kopia error instead of losing it.
Confirmed live: a real failure ("WARNING: spare sync failed — error —
see system logs on ubuntu") didn't match any of categorize_error()'s
known patterns, fell into its generic catch-all bucket, and the actual
stderr text that would have explained it was sitting in a mktemp'd file
this script deletes on exit (trap ... EXIT) — so there was nothing in
"system logs" to actually go check. The categorization was silently
discarding the one piece of information that would have diagnosed the
problem.
Added log_raw_error(), called right after every categorize_error() site
(5 of them: two snapshot-failure paths, the primary REMOTE_TYPE mirror,
the new EXTRA_MIRROR_NAMES loop, and the DR-spare sync) — logs the raw
stderr text (truncated to 500 chars) into the same log stream as
everything else, so it survives past the run that produced it instead
of being deleted with the temp file. categorize_error()'s short bucket
label is untouched and still used for FAILED_SVCS/notification text,
which should stay concise — this adds the detail alongside it, not
instead of it.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01H4k6J1qXXyYxhGEgnJaMvn
Requested: mirror to Backblaze B2 AND directly to the IONOS spare box
over Tailscale, at the same time, not one or the other. REMOTE_TYPE/
REMOTE_ARGS was hardcoded to a single mirror target — extending it to a
list would have meant redesigning the one thing that already works and
was already verified against real B2 credentials, so this adds a
separate, additive mechanism instead: EXTRA_MIRROR_NAMES, a space-
separated list, with per-entry MIRROR_<name>_TYPE/_ARGS (same argument
shape as REMOTE_ARGS). An existing B2-only backup.conf keeps working
completely unchanged if this new section is skipped.
install_backup() gets a new "ADDITIONAL MIRROR" prompt after the
existing B2 section: offers a direct SFTP mirror (Kopia's sync-to sftp,
not the deprecated b2 provider — same reasoning as the S3/B2 choice
already made), defaults the destination to whatever was typed at the
DR-spare prompt above (same box, same purpose, no reason to ask twice),
checks passwordless SSH and an SSH key exist first, then verifies with a
--dry-run against the just-created 'default' repo before saving it —
same "don't save something broken" discipline as the B2 flow. Verified
against a mock backup.conf that install-side writes and worker-side
reads agree on the exact format, and that reusing an existing mirror
name reconfigures it instead of duplicating it in the name list.
One correction while researching sync-to sftp's flags: unlike plain ssh,
Kopia doesn't shell out to the system SSH client, so it needs an
explicit --keyfile and --known-hosts path rather than picking up
whatever `ssh` already trusts automatically — checked Kopia's own docs
for the exact flags before writing this, same as the earlier S3 case.
extras/backup_kopia.sh's worker loops through EXTRA_MIRROR_NAMES after
the existing REMOTE_TYPE mirror step, running sync-to for each
destination against each additional mirror and folding failures into
the same FAILED_SVCS/notification reporting the primary mirror already
uses. Verified end-to-end against a mock backup.conf and a stubbed
kp_for: both the B2 and the new SFTP mirror get called in sequence with
the correct arguments.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01H4k6J1qXXyYxhGEgnJaMvn
- dr_bringup.sh: bound every kopia call and docker compose up with a
timeout so one stuck service can't stall the rest of the batch, and
only exit non-zero if literally nothing came up — a partial recovery
is a partial success, not a failed run.
- backup_kopia.sh: optional DR_SYNC_HOST/DR_SYNC_PATH in backup.conf
scp's backup.conf + README.md to a spare box over SSH after every
successful backup, so dr_bringup.sh is ready there with no manual
copy step.
- backup.sh: prompts for the spare's SSH destination, verifies
connectivity at install time instead of failing silently at 2am, and
writes ~/docker/backup/README.md (this service never had one) so the
synced copy documents every command listed above.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NQkdAn3iG5A4WoqU9FHMaN
restore_kopia.sh is interactive and one-service-at-a-time, which doesn't
scale to standing up a cold spare box quickly during a real outage.
dr_bringup.sh restores every service's latest snapshot (or one named
service) and runs docker compose up -d with no prompts, so a full-stack
recovery is one command instead of N interactive restores.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NQkdAn3iG5A4WoqU9FHMaN
Mount retroarch/shaders and retroarch/overlays from the game drive into
~/.config/retroarch/{shaders,overlays} for both ES-DE and standalone
RetroArch (sibling to the existing cores mount). Anything pulled from the
Online Updater or set in the Quick Menu now persists across sessions and
survives a Wolf/OS reinstall, so a CRT shader or bezel set is configured
once. Dirs are pre-created on the game drive; storage summary and README
updated, including a fallback note to repoint Settings -> Directory if
RetroArch ever writes elsewhere.
gaming-backup: add opt-in snapshots for emulator BIOS (bios/) and the
RetroArch shaders/overlays dirs (cores excluded — re-downloadable). New
BACKUP_BIOS and BACKUP_RA_SHADERS flags wired through the prompts,
backup.conf, summary, and the worker script.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XV8mwKLFwUKt94dAKh3k79
- extras/test_backup_kopia.sh: stop → restore → compare → roll back test for Kopia
- extras/test_backup_borg.sh: stop → extract → compare → roll back test for Borg
- backup workers: timing (duration), service count, and pre-flight disk check (< 512 MB warns)
- backup workers: ntfy notifications include count, duration, and per-failure detail
- services/backup.sh: install test_backup_kopia.sh + optional weekly test timer
- services/borg-backup.sh: install test_backup_borg.sh + optional weekly test timer
extras/test_backup.sh — new unified test script (Kopia + Borg):
• Stops container, moves live data aside, restores latest backup,
compares restored vs live with diff -rq (content, not timestamps),
moves live data back and restarts container
• PASS = restore succeeded; diff output is informational (files changed
since last backup are normal)
• FAIL = restore command failed or target empty after restore
• --list flag, CLI service arg, interactive picker
• Handles both full-service dirs and sub-path sources (gaming-backup)
• Cleanup trap always restores live data even on error
• Sends ntfy notification on pass and fail
extras/backup_kopia.sh, backup_borg.sh, backup_gaming.sh:
• ntfy_send() + categorize_error() helpers added
• Each snapshot/archive failure captures stderr and categorizes:
disk full, remote unreachable, repository not found, wrong passphrase,
permission denied, unknown error
• Single ntfy notification at end: success (low priority) or failure
(urgent) with per-service failure reasons listed
• backup_borg.sh: changed 2>&1 | pipe to 2>"$_ERR" | so stdout logs
cleanly and stderr is captured for error categorization
services/backup.sh, borg-backup.sh, gaming-backup.sh:
• New ntfy prompt section in installer (URL + optional token)
• NTFY_URL / NTFY_TOKEN written to backup.conf
• test_backup.sh copied from extras/ into service dir
• Summary updated to show test_backup.sh commands and ntfy URL
https://claude.ai/code/session_019XgsQ13XKm4Zj3cNsDNwHj
Replace embedded heredocs in the three backup service installers with
cp from versioned source files in extras/:
extras/backup_kopia.sh — Kopia worker (was inline in services/backup.sh)
extras/backup_borg.sh — Borg worker (was inline in services/borg-backup.sh)
extras/backup_gaming.sh — gaming saves worker (was inline in services/gaming-backup.sh)
extras/restore_kopia.sh — unified Kopia restore (multi-dest + single-dest)
extras/restore_borg.sh — unified Borg restore with destination picker
Each installer now does `cp extras/<script>.sh $DIR/<script>.sh` instead of
writing the script inline. Workers and restore scripts are now readable in the
repo rather than buried in heredocs.
Restore scripts are installed flat into the service directory root:
~/docker/backup/backup_kopia.sh ~/docker/backup/restore_kopia.sh
~/docker/borg-backup/backup_borg.sh ~/docker/borg-backup/restore_borg.sh
~/docker/gaming-backup/backup_gaming.sh ~/docker/gaming-backup/restore_kopia.sh
The new restore scripts handle destination selection internally, so a single
script replaces the old per-destination restore/<dest>/ layout.
Also fixes `local` used outside a function in restore_kopia.sh and
restore_borg.sh (destination picker loop), and removes the now-superseded
extras/restore_kopia_backup.sh and extras/restore_borg_backup.sh.
https://claude.ai/code/session_019XgsQ13XKm4Zj3cNsDNwHj
New service: borg-backup (backup group)
- Installs borgbackup from apt
- Same coverage as Kopia backup: full ~/docker/<service>/ snapshots
- Minecraft: flush world (save-all), archive, no downtime
- All others: stop → archive → restart
- Multi-destination support with per-service routing
- SSH remote repos supported natively (user@host:/path)
- Per-service pruning: keep-daily/weekly/monthly
- repo compact after each run
- Systemd timer (with cron fallback)
- repokey-blake2 encryption; key export reminder in summary
- worker.sh: run / list / info subcommands
New restore helper: extras/restore_borg_backup.sh
- Installed to ~/docker/borg-backup/restore/<dest>/
- Interactive: lists services → pick archive → inspect (/tmp) or restore in-place
- In-place restore: stops container, moves current data aside, extracts, restarts
- Old data kept as .pre-restore-DATE with rollback instructions printed
https://claude.ai/code/session_019XgsQ13XKm4Zj3cNsDNwHj
Place datapacks_<ver>.zip or crafting_tweaks_<ver>.zip in extras/datapacks/
before running the Minecraft installer and they will be detected automatically.
- Pattern match: *datapack* → unzip into datapacks-download/; *craft* → copy as-is
- Version extracted from filename; compared against selected MC version at picker step
- Early banner notice when ZIPs are absent (where to put them and naming convention)
- If ZIPs present: show found files + version check, offer "Use pre-packaged VT?" (default y)
- Accepting skips the individual datapack picker entirely
- Manual fallback instructions updated to hint at extras/datapacks/ for next time
- Removed broken VT share# links from manual instructions
- Added extras/datapacks/README.md (naming convention + download sources)
- Added .gitignore: extras/datapacks/*.zip never committed
https://claude.ai/code/session_019XgsQ13XKm4Zj3cNsDNwHj
- extras/restore_kopia_backup.sh: interactive restore tool — lists all
snapshot sources, pick source + snapshot, choose inspect (restore to
/tmp) or restore-in-place (moves current data aside as .pre-restore-DATE
for easy rollback, auto-stops/starts the associated Docker service)
- services/backup.sh: copy restore script to ~/docker/backup/ during
install; update summary to show restore command
- CLAUDE.md: add note to update README services table when adding a
service; add Authelia section documenting which services have built-in
auth, the import-authelia Caddy pattern for no-auth services, and
remember_me_duration config for kiosk/persistent sessions
https://claude.ai/code/session_019XgsQ13XKm4Zj3cNsDNwHj
Adds sync_cc as an extras service module:
- extras/sync_cc.py: the Python tool (3196 lines) — 8 modes: SYNC,
GENERATE, BATCH, RENAME (TMDB), EXTRACT, REMUX, EMBED, BURNSUBS
- services/sync-cc.sh: installs system deps (python3, ffmpeg, mkvtoolnix,
ccextractor), pip installs openai-whisper + ffsubsync, copies the script
to ~/sync-cc/, prompts for TMDB API key → .env, creates /usr/local/bin/sync-cc
wrapper so users run it from any directory containing video/SRT files
Heavy optional deps (easyocr, pgsreader) are installed on first use by the
script itself. GPU (CUDA/MPS) is used automatically if detected.
https://claude.ai/code/session_01Y4dMKtkqkpvmgDKoRdzhTG
Claude