root/ubuntu-post-install
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Support multiple simultaneous offsite mirrors, not just one

Browse Source 
Requested: mirror to Backblaze B2 AND directly to the IONOS spare box
over Tailscale, at the same time, not one or the other. REMOTE_TYPE/
REMOTE_ARGS was hardcoded to a single mirror target — extending it to a
list would have meant redesigning the one thing that already works and
was already verified against real B2 credentials, so this adds a
separate, additive mechanism instead: EXTRA_MIRROR_NAMES, a space-
separated list, with per-entry MIRROR_<name>_TYPE/_ARGS (same argument
shape as REMOTE_ARGS). An existing B2-only backup.conf keeps working
completely unchanged if this new section is skipped.

install_backup() gets a new "ADDITIONAL MIRROR" prompt after the
existing B2 section: offers a direct SFTP mirror (Kopia's sync-to sftp,
not the deprecated b2 provider — same reasoning as the S3/B2 choice
already made), defaults the destination to whatever was typed at the
DR-spare prompt above (same box, same purpose, no reason to ask twice),
checks passwordless SSH and an SSH key exist first, then verifies with a
--dry-run against the just-created 'default' repo before saving it —
same "don't save something broken" discipline as the B2 flow. Verified
against a mock backup.conf that install-side writes and worker-side
reads agree on the exact format, and that reusing an existing mirror
name reconfigures it instead of duplicating it in the name list.

One correction while researching sync-to sftp's flags: unlike plain ssh,
Kopia doesn't shell out to the system SSH client, so it needs an
explicit --keyfile and --known-hosts path rather than picking up
whatever `ssh` already trusts automatically — checked Kopia's own docs
for the exact flags before writing this, same as the earlier S3 case.

extras/backup_kopia.sh's worker loops through EXTRA_MIRROR_NAMES after
the existing REMOTE_TYPE mirror step, running sync-to for each
destination against each additional mirror and folding failures into
the same FAILED_SVCS/notification reporting the primary mirror already
uses. Verified end-to-end against a mock backup.conf and a stubbed
kp_for: both the B2 and the new SFTP mirror get called in sequence with
the correct arguments.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01H4k6J1qXXyYxhGEgnJaMvn
This commit is contained in:
Claude
2026-08-14 03:20:49 +00:00
parent f36cb2c394
commit b99ca32438
2 changed files with 124 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
extras/backup_kopia.sh
+21
View File
@@ -170,6 +170,27 @@ if [ "${REMOTE_TYPE:-none}" != "none" ] && [ -n "${REMOTE_TYPE:-}" ]; then
done
fi
# Additional mirrors (EXTRA_MIRROR_NAMES) — run alongside REMOTE_TYPE above,
# not instead of it, so a box can mirror to e.g. Backblaze B2 AND directly
# to a spare over SSH/SFTP at the same time.
for mirror_name in ${EXTRA_MIRROR_NAMES:-}; do
_mtype_var="MIRROR_${mirror_name}_TYPE"
_margs_var="MIRROR_${mirror_name}_ARGS"
_mtype="${!_mtype_var:-}"
_margs="${!_margs_var:-}"
[ -n "$_mtype" ] || continue
for dest in ${DEST_NAMES:-default}; do
log "Mirroring '$dest' to '$mirror_name' ($_mtype)..."
# shellcheck disable=SC2086
if ! kp_for "$dest" repository sync-to "$_mtype" $_margs 2>"$_ERR"; then
_reason="$(categorize_error "$(cat "$_ERR")")"
log "WARNING: mirror '$mirror_name' failed for '$dest' — $_reason"
FAILED_SVCS+=("mirror[$mirror_name/$dest]: $_reason")
rc=1
fi
done
done
# ── Keep a spare box's copy of backup.conf + README current ─────────────────
# Runs after the data itself is backed up (and mirrored, if configured) so a
# sync never ships config pointing at a repo state that isn't actually there
services/backup.sh
+103
View File
@@ -732,6 +732,99 @@ install_backup() {
fi
fi
# ── 7c. Additional mirror — direct SFTP to another box (optional) ────────
# REMOTE_TYPE/REMOTE_ARGS above is ONE offsite mirror. This adds any
# number of FURTHER ones that all run after every backup too — e.g.
# Backblaze B2 AND a spare box reachable over Tailscale/wg-easy/Netbird,
# simultaneously, not one instead of the other. Stored as its own list
# (EXTRA_MIRROR_NAMES + MIRROR_<name>_TYPE/_ARGS per entry) so it's
# additive on top of the existing single-mirror REMOTE_TYPE mechanism
# rather than replacing it — an existing B2-only backup.conf keeps
# working unchanged if this section is skipped.
local EXTRA_MIRROR_NAMES=""
declare -A EXTRA_MIRROR_TYPE=() EXTRA_MIRROR_ARGS=()
if [ -f "$CONF_FILE" ]; then
EXTRA_MIRROR_NAMES="$(grep '^EXTRA_MIRROR_NAMES=' "$CONF_FILE" 2>/dev/null | cut -d= -f2- | tr -d '"')"
local _emn
for _emn in $EXTRA_MIRROR_NAMES; do
EXTRA_MIRROR_TYPE["$_emn"]="$(grep "^MIRROR_${_emn}_TYPE=" "$CONF_FILE" 2>/dev/null | cut -d= -f2- | tr -d '"')"
EXTRA_MIRROR_ARGS["$_emn"]="$(grep "^MIRROR_${_emn}_ARGS=" "$CONF_FILE" 2>/dev/null | cut -d= -f2- | sed -E 's/^"(.*)"$/\1/')"
done
fi
echo ""
echo "═══════════════════════════════════════════════════════"
echo " ADDITIONAL MIRROR — direct to another box (optional)"
echo "═══════════════════════════════════════════════════════"
echo ""
echo " Mirrors to a SECOND destination over SSH/SFTP, in addition to (not"
echo " instead of) the offsite mirror above."
if [ -n "$EXTRA_MIRROR_NAMES" ]; then
echo ""
echo " Already configured: $EXTRA_MIRROR_NAMES — kept either way; answering"
echo " yes below only adds another one, it doesn't replace these."
fi
echo ""
local _ADD_SFTP_MIRROR=""
local _sftp_default_host="${DR_SYNC_HOST:-}"
prompt_yn " Add a direct SFTP mirror to another box$( [ -n "$_sftp_default_host" ] && echo " (e.g. $_sftp_default_host, same as the DR-spare above)")? (y/n):" "n" _ADD_SFTP_MIRROR
if [[ "$_ADD_SFTP_MIRROR" =~ ^[Yy]$ ]]; then
local _SFTP_DEST=""
prompt_text " SSH destination, user@host:" "$_sftp_default_host" _SFTP_DEST
if [ -z "$_SFTP_DEST" ]; then
log_warning " No destination entered — skipping this mirror."
else
local _SFTP_USER="${_SFTP_DEST%%@*}" _SFTP_HOSTNAME="${_SFTP_DEST#*@}"
local _SFTP_PATH="" _MIRROR_NAME=""
prompt_text " Remote path for the repo:" "~/backups/kopia-mirror" _SFTP_PATH
_SFTP_PATH="${_SFTP_PATH:-~/backups/kopia-mirror}"
prompt_text " Short name for this mirror (letters/numbers/underscores):" "spare" _MIRROR_NAME
_MIRROR_NAME="${_MIRROR_NAME:-spare}"
_MIRROR_NAME="${_MIRROR_NAME//[^a-zA-Z0-9_]/_}"
# sync-to sftp doesn't shell out to the system ssh client, so it
# needs an explicit key/known_hosts file rather than picking up
# whatever plain `ssh` already trusts automatically.
local _SFTP_KEYFILE=""
[ -f /root/.ssh/id_ed25519 ] && _SFTP_KEYFILE=/root/.ssh/id_ed25519
[ -z "$_SFTP_KEYFILE" ] && [ -f /root/.ssh/id_rsa ] && _SFTP_KEYFILE=/root/.ssh/id_rsa
if [ -z "$_SFTP_KEYFILE" ]; then
log_warning " No SSH key found for root — this mirror needs one. Set one up (the"
log_warning " DISASTER-RECOVERY SPARE section above offers to generate one) and"
log_warning " re-run this installer to add the mirror."
elif ! ssh -o BatchMode=yes -o ConnectTimeout=5 "$_SFTP_DEST" true 2>/dev/null; then
log_warning " Couldn't SSH to $_SFTP_DEST without a password — not adding this"
log_warning " mirror until that works: ssh-copy-id $_SFTP_DEST"
else
log_info "Verifying SFTP mirror (dry-run sync against the 'default' repo)..."
local _sftp_err
if _sftp_err="$(env KOPIA_PASSWORD="${DEST_PASSWORDS[default]}" "$KOPIA_BIN" \
--config-file="${DEST_CONFIGS[default]}" repository sync-to sftp \
--host="$_SFTP_HOSTNAME" --username="$_SFTP_USER" --path="$_SFTP_PATH" \
--keyfile="$_SFTP_KEYFILE" --known-hosts=/root/.ssh/known_hosts \
--dry-run 2>&1)"; then
EXTRA_MIRROR_TYPE["$_MIRROR_NAME"]="sftp"
EXTRA_MIRROR_ARGS["$_MIRROR_NAME"]="--host=$_SFTP_HOSTNAME --username=$_SFTP_USER --path=$_SFTP_PATH --keyfile=$_SFTP_KEYFILE --known-hosts=/root/.ssh/known_hosts"
# Reusing an existing mirror name reconfigures it (the
# associative-array assignments above already do that)
# without duplicating it in the space-separated name list.
if [[ " $EXTRA_MIRROR_NAMES " != *" $_MIRROR_NAME "* ]]; then
if [ -z "$EXTRA_MIRROR_NAMES" ]; then
EXTRA_MIRROR_NAMES="$_MIRROR_NAME"
else
EXTRA_MIRROR_NAMES="$EXTRA_MIRROR_NAMES $_MIRROR_NAME"
fi
fi
log_success " SFTP mirror '$_MIRROR_NAME' verified — will run after every backup."
else
log_warning " SFTP dry-run failed — not adding this mirror:"
log_warning " $_sftp_err"
fi
fi
fi
fi
# ── 8. Write backup.conf ─────────────────────────────────────────────────
log_info "Writing $CONF_FILE ..."
{
@@ -775,6 +868,16 @@ install_backup() {
echo "REMOTE_TYPE=\"$REMOTE_TYPE\""
echo "REMOTE_ARGS=\"$REMOTE_ARGS\""
echo ""
echo "# ── Additional mirrors — run alongside REMOTE_TYPE above, not instead of it ──"
echo "# Space-separated list of names; each gets its own MIRROR_<name>_TYPE/_ARGS,"
echo "# same argument shape as REMOTE_ARGS. Every one of these runs after every"
echo "# backup too, in addition to the REMOTE_TYPE mirror above."
echo "EXTRA_MIRROR_NAMES=\"$EXTRA_MIRROR_NAMES\""
for _emn in $EXTRA_MIRROR_NAMES; do
echo "MIRROR_${_emn}_TYPE=\"${EXTRA_MIRROR_TYPE[$_emn]}\""
echo "MIRROR_${_emn}_ARGS=\"${EXTRA_MIRROR_ARGS[$_emn]}\""
done
echo ""
echo "# ── Notifications (ntfy) ─────────────────────────────────────────────────────"
echo "# Set NTFY_URL to receive backup success/failure alerts."
echo "# Leave blank to disable. NTFY_TOKEN is optional (for private topics)."