Add Gatus auto-sync from Caddyfile, promote ensure_yq to lib/common.sh

Adds one Gatus endpoint per Caddy site block automatically, tagged
group: caddy-sync — the sync only ever adds/removes entries in that
exact group, so anything added by hand (the default external checks,
a custom endpoint) is never touched regardless of what the Caddyfile
looks like. Offered at install time (syncs once immediately) and, if
systemd is available, scheduled via a timer every 15 minutes so a site
added or removed later gets picked up without re-running the installer
— matches the "schedule that checks the Caddyfile" shape asked for.

Domain extraction tracks actual brace depth (reusing the same approach
as remove_service's Caddy block removal) rather than a naive
line-by-line scan, so it correctly skips the global options block and
parenthesized snippet definitions like (authelia) without needing to
special-case them by name.

Verified end-to-end against a real Caddyfile/config.yaml fixture with
the actual mikefarah/yq binary: initial sync adds the right entries
and leaves the default "external" group alone, a second run with no
Caddyfile changes is a true no-op (0 added, 0 removed), and changing
the Caddyfile (removing one site, adding another) correctly adds the
new endpoint and removes only the stale one.

Also fixes a real gap surfaced while building this: ensure_yq (used by
both gatus.sh now and onlyoffice.sh already) checked `command -v yq`
alone, which a box can satisfy with a completely different, incompatible
yq — confirmed live in this environment, Debian/Ubuntu's own `yq`
apt package is kislyuk/yq (a Python jq-wrapper) which silently errors
on mikefarah/yq's `e '.path' file` syntax every caller here depends on.
Now checks the version string actually identifies as mikefarah's
before trusting it, installing to /usr/local/bin (which precedes
/usr/bin on Ubuntu's default PATH) if not. Promoted ensure_yq itself
from onlyoffice.sh (its only previous user) to lib/common.sh now that
gatus.sh needs the same thing, so both share one implementation.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01H4k6J1qXXyYxhGEgnJaMvn

lib/common.sh

@@ -244,6 +244,33 @@ ensure_caddy_network() {
         && log_info "Created Docker network ${_net} (needed by Caddy-fronted services)"
 }
 
+# Installs yq v4 (Go binary release, not the Python click-based yq some
+# distros package under the same name) if not already present. Shared by
+# any service that needs to patch YAML config robustly instead of
+# hand-rolling sed/awk text surgery — originally lived only in
+# services/onlyoffice.sh (FileBrowser config patching); promoted here once
+# services/gatus.sh needed the same thing (endpoint sync from Caddyfile),
+# so both use one implementation instead of two copies drifting apart.
+ensure_yq() {
+    # Not just `command -v yq` — confirmed live, a box can already have
+    # `yq` on PATH that's actually kislyuk/yq (the Python jq-wrapper apt
+    # packages under the same name on Debian/Ubuntu), which silently
+    # errors on mikefarah's `e '.path' file` syntax every caller here
+    # uses ("argument files: can't open '.path'"). Check the version
+    # string actually identifies as mikefarah's before trusting it.
+    yq --version 2>/dev/null | grep -q mikefarah && return 0
+    log_info "Installing yq..."
+    local _arch; _arch=$(uname -m)
+    local _binary="yq_linux_amd64"
+    [[ "$_arch" == "aarch64" || "$_arch" == "arm64" ]] && _binary="yq_linux_arm64"
+    # /usr/local/bin precedes /usr/bin on Ubuntu's default PATH, so this
+    # correctly shadows a wrong /usr/bin/yq without needing to touch or
+    # remove it (something else on the box may depend on the real one).
+    curl -fsSL "https://github.com/mikefarah/yq/releases/latest/download/${_binary}" \
+        -o /usr/local/bin/yq && chmod +x /usr/local/bin/yq \
+        && log_success "yq installed" || log_warning "yq install failed"
+}
+
 # Enables UFW if it isn't already active. Call this AFTER the caller has
 # already added its own `ufw allow` rules for whatever it needs — this only
 # flips UFW from inactive to active, it doesn't add rules for the calling

services/gatus.sh

@@ -207,6 +207,135 @@ fi
 
 register_service gatus utilities "Status & uptime monitoring page (Gatus)" 8086
 
+# ── Caddyfile -> Gatus endpoint sync ────────────────────────────────────────
+# Every auto-added endpoint is tagged group: "caddy-sync" — the sync script
+# only ever adds or removes entries in that exact group, so anything you add
+# by hand (the default external checks, a custom endpoint, different group
+# name) is never touched regardless of what the Caddyfile looks like.
+_gatus_write_sync_script() {
+    local gatus_dir="$1" caddy_file="$2"
+    cat > "$gatus_dir/sync-from-caddy.sh" << 'SYNC_SCRIPT'
+#!/bin/bash
+# Auto-generated by services/gatus.sh — do not edit by hand, re-run the
+# installer to regenerate. Syncs Gatus endpoints with the current
+# Caddyfile: adds a group: caddy-sync endpoint for every site block found,
+# removes caddy-sync endpoints for sites no longer there. Never touches
+# any other endpoint.
+set -uo pipefail
+
+CADDYFILE="$1"
+GATUS_CONFIG="$2"
+
+# Not just `command -v yq` — some distros package a completely different,
+# incompatible yq (kislyuk's Python jq-wrapper) under the same name; check
+# it actually identifies as mikefarah's before trusting its syntax.
+yq --version 2>/dev/null | grep -q mikefarah || { echo "yq unavailable or incompatible — skipping sync"; exit 0; }
+[ -f "$CADDYFILE" ] || { echo "No Caddyfile at $CADDYFILE — skipping sync"; exit 0; }
+[ -f "$GATUS_CONFIG" ] || { echo "No Gatus config at $GATUS_CONFIG — skipping sync"; exit 0; }
+
+# Site block headers at brace-depth 0, skipping the bare global-options
+# block (no address before its "{") and parenthesized snippet definitions
+# like (authelia) — same brace-tracking approach as the Caddy site-block
+# removal in lib/common.sh (remove_service), for the same reason: a naive
+# "next blank line" scan is exactly the class of bug that already bit this
+# repo once on a live Caddy/Samba config.
+CURRENT_DOMAINS="$(awk '
+    BEGIN { depth = 0 }
+    {
+        line = $0
+        opens = gsub(/\{/, "{", line)
+        closes = gsub(/\}/, "}", line)
+        if (depth == 0 && opens > 0) {
+            header = $0
+            sub(/[[:space:]]*\{.*$/, "", header)
+            gsub(/^[[:space:]]+|[[:space:]]+$/, "", header)
+            if (header != "" && header !~ /^\(/) print header
+            depth += opens - closes
+            next
+        }
+        if (depth > 0) { depth += opens - closes; next }
+    }
+' "$CADDYFILE")"
+
+EXISTING_SYNCED="$(yq e '.endpoints[] | select(.group == "caddy-sync") | .name' "$GATUS_CONFIG" 2>/dev/null)"
+
+ADDED=0
+REMOVED=0
+
+while IFS= read -r domain; do
+    [ -z "$domain" ] && continue
+    # Domains only — a defensive filter, not strictly needed since these
+    # come from our own Caddyfile, but cheap insurance against ever
+    # building a yq expression out of anything unexpected.
+    [[ "$domain" =~ ^[a-zA-Z0-9.-]+$ ]] || continue
+    if ! grep -qxF "$domain" <<< "$EXISTING_SYNCED"; then
+        yq e -i ".endpoints += [{\"name\": \"${domain}\", \"group\": \"caddy-sync\", \"url\": \"https://${domain}\", \"interval\": \"5m\", \"conditions\": [\"[STATUS] < 400\"]}]" "$GATUS_CONFIG" \
+            && ADDED=$((ADDED + 1))
+    fi
+done <<< "$CURRENT_DOMAINS"
+
+while IFS= read -r name; do
+    [ -z "$name" ] && continue
+    if ! grep -qxF "$name" <<< "$CURRENT_DOMAINS"; then
+        yq e -i "del(.endpoints[] | select(.group == \"caddy-sync\" and .name == \"${name}\"))" "$GATUS_CONFIG" \
+            && REMOVED=$((REMOVED + 1))
+    fi
+done <<< "$EXISTING_SYNCED"
+
+echo "Gatus sync: added $ADDED, removed $REMOVED endpoint(s) from $(basename "$CADDYFILE")"
+SYNC_SCRIPT
+    chmod 700 "$gatus_dir/sync-from-caddy.sh"
+    chown "$ACTUAL_USER:$ACTUAL_USER" "$gatus_dir/sync-from-caddy.sh" 2>/dev/null || true
+}
+
+_gatus_setup_caddy_sync() {
+    local gatus_dir="$1"
+    local caddy_file="$DOCKER_DIR/caddy/Caddyfile"
+
+    [ -f "$caddy_file" ] || return 0
+
+    echo ""
+    local ENABLE_SYNC=""
+    prompt_yn "Automatically keep Gatus endpoints in sync with the Caddyfile (syncs now, then every 15 minutes)? (y/n):" "y" ENABLE_SYNC
+    [[ "$ENABLE_SYNC" =~ ^[Yy]$ ]] || return 0
+
+    declare -F ensure_yq >/dev/null 2>&1 && ensure_yq
+    command -v yq >/dev/null 2>&1 || { log_warning "yq unavailable — skipping Caddy sync."; return 0; }
+
+    _gatus_write_sync_script "$gatus_dir" "$caddy_file"
+    "$gatus_dir/sync-from-caddy.sh" "$caddy_file" "$gatus_dir/gatus_config/config.yaml"
+
+    if command -v systemctl >/dev/null 2>&1; then
+        cat > /etc/systemd/system/gatus-caddy-sync.service << UNIT
+[Unit]
+Description=Sync Gatus endpoints from Caddyfile
+
+[Service]
+Type=oneshot
+ExecStart=${gatus_dir}/sync-from-caddy.sh ${caddy_file} ${gatus_dir}/gatus_config/config.yaml
+UNIT
+        cat > /etc/systemd/system/gatus-caddy-sync.timer << 'UNIT'
+[Unit]
+Description=Periodic Gatus/Caddyfile sync
+
+[Timer]
+OnBootSec=5min
+OnUnitActiveSec=15min
+
+[Install]
+WantedBy=timers.target
+UNIT
+        systemctl daemon-reload
+        if systemctl enable --now gatus-caddy-sync.timer >/dev/null 2>&1; then
+            log_success "Sync scheduled every 15 minutes (systemd timer: gatus-caddy-sync)"
+        else
+            log_warning "Couldn't enable the sync timer — re-run $gatus_dir/sync-from-caddy.sh manually after Caddyfile changes."
+        fi
+    else
+        log_warning "No systemd — synced once, but won't auto-repeat. Re-run $gatus_dir/sync-from-caddy.sh manually after Caddyfile changes."
+    fi
+}
+
 install_gatus() {
     require_docker || return 1
     log_info "Installing Gatus..."
@@ -218,6 +347,7 @@ install_gatus() {
         echo "[DRY-RUN] Would deploy twinproduction/gatus:latest"
         echo "[DRY-RUN] Port 8086 published (auto-scanned for a free host port),"
         echo "[DRY-RUN] config at gatus_config/config.yaml"
+        echo "[DRY-RUN] Would offer to auto-sync endpoints from the Caddyfile (adds/removes group: caddy-sync entries, every 15min via a systemd timer)"
         return 0
     fi
 
@@ -359,6 +489,28 @@ Key concepts:
 
 Full docs: https://github.com/TwiN/gatus
 
+## Caddyfile sync
+
+If you enabled it during install, \`sync-from-caddy.sh\` runs every 15
+minutes (systemd timer: \`gatus-caddy-sync\`) and keeps one endpoint per
+Caddy site block in sync automatically — added when a new site appears,
+removed when one goes away. Every auto-added endpoint is tagged
+\`group: caddy-sync\`; the sync only ever touches entries in that group,
+so anything you add or edit by hand (a different group, the default
+external checks) is left alone no matter what.
+
+Run it by hand any time instead of waiting for the timer:
+\`\`\`bash
+$GATUS_DIR/sync-from-caddy.sh $DOCKER_DIR/caddy/Caddyfile $GATUS_DIR/gatus_config/config.yaml
+\`\`\`
+
+Check/manage the timer:
+\`\`\`bash
+systemctl status gatus-caddy-sync.timer
+systemctl list-timers gatus-caddy-sync.timer
+sudo systemctl disable --now gatus-caddy-sync.timer   # turn it off
+\`\`\`
+
 ## Manage
 \`\`\`bash
 cd $GATUS_DIR
@@ -377,6 +529,8 @@ MD
             || log_warning "Failed to start — check: docker compose logs"
     fi
 
+    _gatus_setup_caddy_sync "$GATUS_DIR"
+
     echo "  Access at:  http://localhost:${WEB_PORT}"
     echo "  Config:     $GATUS_DIR/gatus_config/config.yaml (hot-reloaded)"
     echo ""

services/onlyoffice.sh

@@ -191,17 +191,8 @@ fi
 
 register_service onlyoffice utilities "Self-hosted OnlyOffice Document Server (Nextcloud/FileBrowser)" 8082
 
-# ── Helper: install yq v4 if absent ──────────────────────────────────────────
+# yq install helper moved to lib/common.sh (ensure_yq) — shared with
-_ensure_yq() {
+# services/gatus.sh now that it needs the same thing.
-    command -v yq &>/dev/null && return 0
-    log_info "Installing yq (required for FileBrowser config patching)..."
-    local _arch; _arch=$(uname -m)
-    local _binary="yq_linux_amd64"
-    [[ "$_arch" == "aarch64" || "$_arch" == "arm64" ]] && _binary="yq_linux_arm64"
-    curl -fsSL "https://github.com/mikefarah/yq/releases/latest/download/${_binary}" \
-        -o /usr/local/bin/yq && chmod +x /usr/local/bin/yq \
-        && log_success "yq installed" || log_warning "yq install failed — FileBrowser wiring skipped"
-}
 
 # ── Helper: wire OnlyOffice into Nextcloud (idempotent) ───────────────────────
 _wire_nextcloud() {
@@ -228,7 +219,7 @@ _wire_nextcloud() {
 _wire_filebrowser() {
     local _fbq_config="$DOCKER_DIR/filebrowser/config.yaml"
     [[ -f "$_fbq_config" ]] || { log_info "FileBrowser config not found — skipping"; return 0; }
-    command -v yq &>/dev/null || { log_info "yq not found — skipping FileBrowser wiring"; return 0; }
+    yq --version 2>/dev/null | grep -q mikefarah || { log_info "yq unavailable or incompatible — skipping FileBrowser wiring"; return 0; }
     log_info "Wiring OnlyOffice into FileBrowser Quantum..."
     yq e '.officeServer = "http://onlyoffice:80/"' -i "$_fbq_config" \
         && log_success "FileBrowser officeServer set" || log_warning "Could not patch FileBrowser config"
@@ -237,7 +228,7 @@ _wire_filebrowser() {
 
 install_onlyoffice() {
     require_docker || return 1
-    _ensure_yq
+    declare -F ensure_yq >/dev/null 2>&1 && ensure_yq
 
     log_info "Installing OnlyOffice Document Server..."
     local DIR="$DOCKER_DIR/onlyoffice"