Files
ubuntu-post-install/services/gatus.sh
T
Claude d1d234b4d2 Fix Gatus false-positive red on Authelia-protected and stale-synced sites
The auto-sync condition "[STATUS] < 400" reads as red for any site behind
Authelia's forward_auth: Gatus's probe is never logged in, so it correctly
gets a 401 back every time — the site is completely healthy, Authelia is
just doing its job, but that 401 fails the condition. Confirmed live: every
site the user actually logs into showed permanently red.

That single condition also had the opposite bug in reserve: on a genuine
outage (connection refused, DNS failure, TLS failure), Gatus reports
[STATUS] as 0, and 0 < 400 is true — a fully unreachable site would have
silently read as "up".

Fixed to two conditions together: "[CONNECTED] == true" (catches the actual
outage case) and "[STATUS] < 500" (accepts any real response, including
401/403/redirects from an auth gate, only failing on Caddy's own 502/503/504
when the backend itself is unreachable).

Also changed the sync loop to refresh conditions on already-synced
endpoints, not just add-missing-ones — the old add-if-missing-only logic
meant this fix would only apply to newly discovered domains, leaving every
already-synced site (which is most of them, on a live box) stuck on the
broken condition forever until removed and re-added by hand. Now every
sync run (every 15 minutes via the existing systemd timer, or the one that
happens immediately on a Gatus reinstall) self-heals all of them.

Verified end-to-end against the real mikefarah/yq binary: an existing
caddy-sync entry gets its conditions rewritten in place, an unrelated
manually-added endpoint is left untouched, and a newly-discovered domain
gets the corrected conditions from the start.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01H4k6J1qXXyYxhGEgnJaMvn
2026-08-11 13:39:02 +00:00

561 lines
22 KiB
Bash

#!/bin/bash
# services/gatus.sh — Gatus status/uptime monitoring page.
# Part of the modular post-install system (sourced by setup.sh).
#
# Can also be run standalone on any machine:
# sudo bash gatus.sh
# (Docker must already be installed when run standalone)
#
# Gatus polls endpoints (HTTP, TCP, DNS, ICMP) on a schedule and shows a
# clean status dashboard. Config is hot-reloaded from gatus_config/config.yaml.
# ── Standalone bootstrap ──────────────────────────────────────────────────────
# Detected when the script is executed directly rather than sourced by setup.sh.
# Sets up helpers and globals, then defers execution until after the function
# definition at the bottom of this file.
if [[ "${BASH_SOURCE[0]}" == "${0}" ]]; then
[[ "$(id -u)" == "0" ]] || { echo "Run with sudo: sudo bash $0"; exit 1; }
_SELF_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
_COMMON="$_SELF_DIR/../lib/common.sh"
if [[ -f "$_COMMON" ]]; then
# Full repo present — use the real helpers (picks up ~/docker/.config too)
# shellcheck source=../lib/common.sh
source "$_COMMON"
else
# One-off copy — inline minimal stubs so the script works without the repo
log_info() { echo -e "\033[0;34m[INFO]\033[0m $*"; }
log_success() { echo -e "\033[0;32m[OK]\033[0m $*"; }
log_warning() { echo -e "\033[1;33m[WARN]\033[0m $*"; }
log_error() { echo -e "\033[0;31m[ERROR]\033[0m $*" >&2; }
require_docker() {
command -v docker &>/dev/null || {
log_error "Docker not found. Install it first:"
log_error " curl -fsSL https://get.docker.com | sudo sh"
return 1
}
docker compose version &>/dev/null || {
log_error "Docker Compose plugin missing:"
log_error " sudo apt-get install -y docker-compose-plugin"
return 1
}
}
ensure_docker_dir_ownership() {
chown -R "$ACTUAL_USER:$ACTUAL_USER" "$@" 2>/dev/null || true
}
port_in_use() {
local _port="$1" _proto="${2:-tcp}"
local _flag="-tlnH"
[ "$_proto" = "udp" ] && _flag="-ulnH"
ss "$_flag" "sport = :${_port}" 2>/dev/null | grep -q .
}
find_free_port() {
local _varname="$1" _port="$2" _proto="${3:-tcp}"
while port_in_use "$_port" "$_proto"; do
_port=$((_port + 1))
done
eval "$_varname='$_port'"
}
# Match common.sh's eval-based pattern so local vars in install_* are set correctly
prompt_text() {
local _q="$1" _def="$2" _var="$3" _r
[[ "${UNATTENDED:-false}" == "true" ]] && { eval "$_var='$_def'"; return; }
read -r -p " $_q " _r
eval "$_var='${_r:-$_def}'"
}
prompt_yn() {
local _q="$1" _def="$2" _var="$3" _r
[[ "${UNATTENDED:-false}" == "true" ]] && { eval "$_var='$_def'"; return; }
read -r -p " $_q " _r
eval "$_var='${_r:-$_def}'"
}
configure_caddy_for_service() {
local _name="$1" _upstream="$2" _subdomain="$3" _extra="${4:-}"
local _caddy_dir="$DOCKER_DIR/caddy"
local _caddyfile="$_caddy_dir/Caddyfile"
local _display_port="${_upstream##*:}"
# Determine mode: local Caddy, remote Caddy, or none
local _mode="none"
[[ -d "$_caddy_dir" ]] && _mode="local"
[[ -n "${CADDY_REMOTE_HOST:-}" ]] && [[ "$_mode" != "local" ]] && _mode="remote"
[[ "$_mode" == "none" ]] && {
log_info "Access $_name directly on port $_display_port."
return 0
}
echo ""
local _do_caddy=""
if [[ "$_mode" == "remote" ]]; then
log_info "Remote Caddy configured (${CADDY_REMOTE_HOST})."
log_info "A snippet file will be saved to ~/docker/caddy-snippets/."
fi
read -r -p " Configure Caddy reverse proxy for $_name? [y/N]: " _do_caddy
[[ "${_do_caddy,,}" == "y" ]] || {
log_info "Skipping — access at: http://localhost:$_display_port"
return 0
}
# Domain prompt — pre-fill from SITE_DOMAIN when available
local _default_domain=""
if [[ -n "${SITE_DOMAIN:-}" ]] && [[ "$SITE_DOMAIN" != "example.com" ]]; then
_default_domain="${_subdomain}.${SITE_DOMAIN}"
log_info "Default: $_default_domain"
fi
local _domain=""
read -r -p " Domain [${_default_domain:-required}]: " _domain
_domain="${_domain:-$_default_domain}"
[[ -n "$_domain" ]] || { log_warning "No domain entered — skipping Caddy."; return 0; }
# Build upstream — remote Caddy uses host IP:port, not container name
local _block_upstream="$_upstream"
if [[ "$_mode" == "remote" ]]; then
_block_upstream="${CADDY_REMOTE_HOST}:${_display_port}"
fi
local _site_block
_site_block="$(cat << CBLOCK
# $_name
${_domain} {
reverse_proxy ${_block_upstream}
header {
Strict-Transport-Security "max-age=31536000; includeSubDomains; preload"
X-Content-Type-Options "nosniff"
X-Frame-Options "SAMEORIGIN"
Referrer-Policy "strict-origin-when-cross-origin"
}
log {
output file /var/log/caddy/${_domain}.log
format json
}
${_extra}
}
CBLOCK
)"
if [[ "$_mode" == "local" ]]; then
if [[ -f "$_caddyfile" ]]; then
local _bk="$_caddy_dir/Caddyfile.backup.$(date +%Y%m%d-%H%M%S)"
cp "$_caddyfile" "$_bk"
log_info "Backed up Caddyfile to $(basename "$_bk")"
else
touch "$_caddyfile"
fi
if grep -q "^${_domain}" "$_caddyfile" 2>/dev/null; then
log_warning "$_domain already in Caddyfile"
local _ow=""
read -r -p " Overwrite? [y/N]: " _ow
[[ "${_ow,,}" == "y" ]] || { log_info "Keeping existing entry."; return 0; }
sed -i "/^${_domain}/,/^}/d" "$_caddyfile"
fi
printf '%s\n' "$_site_block" >> "$_caddyfile"
log_success "Added $_domain to Caddyfile"
docker exec caddy caddy fmt --overwrite /etc/caddy/Caddyfile 2>/dev/null || true
if docker exec caddy caddy reload --config /etc/caddy/Caddyfile 2>/dev/null; then
log_success "$_name accessible at: https://$_domain"
else
log_warning "Reload failed — check: docker logs caddy"
log_info "Manual reload: docker exec caddy caddy reload --config /etc/caddy/Caddyfile"
fi
else
local _snippet_dir="$DOCKER_DIR/caddy-snippets"
local _snippet_file="$_snippet_dir/${_subdomain}.caddy"
mkdir -p "$_snippet_dir"
printf '%s\n' "$_site_block" > "$_snippet_file"
chown "$ACTUAL_USER:$ACTUAL_USER" "$_snippet_file" 2>/dev/null || true
log_success "Snippet saved: $_snippet_file"
log_info "Copy to Caddy machine:"
log_info " scp $_snippet_file caddy-host:~/caddy-snippets/"
log_info " rsync -av $_snippet_dir/ caddy-host:~/caddy-snippets/ (all at once)"
fi
}
write_readme() {
local _dir="$1"; shift
mkdir -p "$_dir"
cat > "$_dir/README.md"
}
fi
# Globals — ACTUAL_USER/ACTUAL_HOME must come before DOCKER_DIR
# ($HOME under sudo is /root, not the real user's home)
ACTUAL_USER="${ACTUAL_USER:-${SUDO_USER:-$USER}}"
ACTUAL_HOME="$(getent passwd "$ACTUAL_USER" 2>/dev/null | cut -d: -f6 || echo "${HOME:-/root}")"
DOCKER_DIR="${DOCKER_DIR:-$ACTUAL_HOME/docker}"
DRY_RUN="${DRY_RUN:-false}"
UNATTENDED="${UNATTENDED:-false}"
SITE_TZ="${SITE_TZ:-$(cat /etc/timezone 2>/dev/null || echo UTC)}"
SITE_DOMAIN="${SITE_DOMAIN:-example.com}"
SITE_CADDY_NET="${SITE_CADDY_NET:-caddy_net}"
register_service() { :; } # no-op — no wizard to register into
_RUN_STANDALONE=1
fi
# ─────────────────────────────────────────────────────────────────────────────
register_service gatus utilities "Status & uptime monitoring page (Gatus)" 8086
# ── Caddyfile -> Gatus endpoint sync ────────────────────────────────────────
# Every auto-added endpoint is tagged group: "caddy-sync" — the sync script
# only ever adds or removes entries in that exact group, so anything you add
# by hand (the default external checks, a custom endpoint, different group
# name) is never touched regardless of what the Caddyfile looks like.
_gatus_write_sync_script() {
local gatus_dir="$1" caddy_file="$2"
cat > "$gatus_dir/sync-from-caddy.sh" << 'SYNC_SCRIPT'
#!/bin/bash
# Auto-generated by services/gatus.sh — do not edit by hand, re-run the
# installer to regenerate. Syncs Gatus endpoints with the current
# Caddyfile: adds a group: caddy-sync endpoint for every site block found,
# removes caddy-sync endpoints for sites no longer there. Never touches
# any other endpoint.
set -uo pipefail
CADDYFILE="$1"
GATUS_CONFIG="$2"
# Not just `command -v yq` — some distros package a completely different,
# incompatible yq (kislyuk's Python jq-wrapper) under the same name; check
# it actually identifies as mikefarah's before trusting its syntax.
yq --version 2>/dev/null | grep -q mikefarah || { echo "yq unavailable or incompatible — skipping sync"; exit 0; }
[ -f "$CADDYFILE" ] || { echo "No Caddyfile at $CADDYFILE — skipping sync"; exit 0; }
[ -f "$GATUS_CONFIG" ] || { echo "No Gatus config at $GATUS_CONFIG — skipping sync"; exit 0; }
# Site block headers at brace-depth 0, skipping the bare global-options
# block (no address before its "{") and parenthesized snippet definitions
# like (authelia) — same brace-tracking approach as the Caddy site-block
# removal in lib/common.sh (remove_service), for the same reason: a naive
# "next blank line" scan is exactly the class of bug that already bit this
# repo once on a live Caddy/Samba config.
CURRENT_DOMAINS="$(awk '
BEGIN { depth = 0 }
{
line = $0
opens = gsub(/\{/, "{", line)
closes = gsub(/\}/, "}", line)
if (depth == 0 && opens > 0) {
header = $0
sub(/[[:space:]]*\{.*$/, "", header)
gsub(/^[[:space:]]+|[[:space:]]+$/, "", header)
if (header != "" && header !~ /^\(/) print header
depth += opens - closes
next
}
if (depth > 0) { depth += opens - closes; next }
}
' "$CADDYFILE")"
EXISTING_SYNCED="$(yq e '.endpoints[] | select(.group == "caddy-sync") | .name' "$GATUS_CONFIG" 2>/dev/null)"
ADDED=0
REMOVED=0
# [CONNECTED] == true catches an actual outage (refused/timed-out
# connection, DNS failure, TLS failure — [STATUS] reads 0 in all of those,
# which is < 400 too, so that condition ALONE would have silently reported
# a fully unreachable site as "up"). [STATUS] < 500 accepts anything the
# server actually answered with, including 401/403/redirects from an
# Authelia-protected site — Gatus's own probe is never logged in, so an
# Authelia-gated site correctly returns 401 to it every time, which is the
# site working exactly as designed, not an outage. Only Caddy's own
# 502/503/504 (backend unreachable) or a connection failure should ever
# read as down here. Confirmed live: with the old "[STATUS] < 400" alone,
# every Authelia-protected site the user actually logs into showed red
# permanently, and a would-be-real outage (STATUS=0) would have shown green.
CONDITIONS='["[CONNECTED] == true", "[STATUS] < 500"]'
while IFS= read -r domain; do
[ -z "$domain" ] && continue
# Domains only — a defensive filter, not strictly needed since these
# come from our own Caddyfile, but cheap insurance against ever
# building a yq expression out of anything unexpected.
[[ "$domain" =~ ^[a-zA-Z0-9.-]+$ ]] || continue
if grep -qxF "$domain" <<< "$EXISTING_SYNCED"; then
# Already synced — refresh its conditions too, not just add-if-
# missing, so a template fix like this one reaches every
# already-added endpoint on the very next sync instead of leaving
# them stuck on whatever conditions they were first created with.
yq e -i "(.endpoints[] | select(.group == \"caddy-sync\" and .name == \"${domain}\") | .conditions) = ${CONDITIONS}" "$GATUS_CONFIG"
else
yq e -i ".endpoints += [{\"name\": \"${domain}\", \"group\": \"caddy-sync\", \"url\": \"https://${domain}\", \"interval\": \"5m\", \"conditions\": ${CONDITIONS}}]" "$GATUS_CONFIG" \
&& ADDED=$((ADDED + 1))
fi
done <<< "$CURRENT_DOMAINS"
while IFS= read -r name; do
[ -z "$name" ] && continue
if ! grep -qxF "$name" <<< "$CURRENT_DOMAINS"; then
yq e -i "del(.endpoints[] | select(.group == \"caddy-sync\" and .name == \"${name}\"))" "$GATUS_CONFIG" \
&& REMOVED=$((REMOVED + 1))
fi
done <<< "$EXISTING_SYNCED"
echo "Gatus sync: added $ADDED, removed $REMOVED endpoint(s) from $(basename "$CADDYFILE")"
SYNC_SCRIPT
chmod 700 "$gatus_dir/sync-from-caddy.sh"
chown "$ACTUAL_USER:$ACTUAL_USER" "$gatus_dir/sync-from-caddy.sh" 2>/dev/null || true
}
_gatus_setup_caddy_sync() {
local gatus_dir="$1"
local caddy_file="$DOCKER_DIR/caddy/Caddyfile"
[ -f "$caddy_file" ] || return 0
echo ""
local ENABLE_SYNC=""
prompt_yn "Automatically keep Gatus endpoints in sync with the Caddyfile (syncs now, then every 15 minutes)? (y/n):" "y" ENABLE_SYNC
[[ "$ENABLE_SYNC" =~ ^[Yy]$ ]] || return 0
declare -F ensure_yq >/dev/null 2>&1 && ensure_yq
command -v yq >/dev/null 2>&1 || { log_warning "yq unavailable — skipping Caddy sync."; return 0; }
_gatus_write_sync_script "$gatus_dir" "$caddy_file"
"$gatus_dir/sync-from-caddy.sh" "$caddy_file" "$gatus_dir/gatus_config/config.yaml"
if command -v systemctl >/dev/null 2>&1; then
cat > /etc/systemd/system/gatus-caddy-sync.service << UNIT
[Unit]
Description=Sync Gatus endpoints from Caddyfile
[Service]
Type=oneshot
ExecStart=${gatus_dir}/sync-from-caddy.sh ${caddy_file} ${gatus_dir}/gatus_config/config.yaml
UNIT
cat > /etc/systemd/system/gatus-caddy-sync.timer << 'UNIT'
[Unit]
Description=Periodic Gatus/Caddyfile sync
[Timer]
OnBootSec=5min
OnUnitActiveSec=15min
[Install]
WantedBy=timers.target
UNIT
systemctl daemon-reload
if systemctl enable --now gatus-caddy-sync.timer >/dev/null 2>&1; then
log_success "Sync scheduled every 15 minutes (systemd timer: gatus-caddy-sync)"
else
log_warning "Couldn't enable the sync timer — re-run $gatus_dir/sync-from-caddy.sh manually after Caddyfile changes."
fi
else
log_warning "No systemd — synced once, but won't auto-repeat. Re-run $gatus_dir/sync-from-caddy.sh manually after Caddyfile changes."
fi
}
install_gatus() {
require_docker || return 1
log_info "Installing Gatus..."
local GATUS_DIR="$DOCKER_DIR/gatus"
local WEB_PORT="8086"
if [ "$DRY_RUN" = true ]; then
echo "[DRY-RUN] Would create $GATUS_DIR (gatus_config/, gatus_data/)"
echo "[DRY-RUN] Would deploy twinproduction/gatus:latest"
echo "[DRY-RUN] Port 8086 published (auto-scanned for a free host port),"
echo "[DRY-RUN] config at gatus_config/config.yaml"
echo "[DRY-RUN] Would offer to auto-sync endpoints from the Caddyfile (adds/removes group: caddy-sync entries, every 15min via a systemd timer)"
return 0
fi
# Scan for a free host port — a plain install shouldn't silently claim a
# port another already-running service holds. See CLAUDE.md's "Port
# collision avoidance" section.
find_free_port WEB_PORT "$WEB_PORT"
mkdir -p "$GATUS_DIR/gatus_config" "$GATUS_DIR/gatus_data"
ensure_docker_dir_ownership "$GATUS_DIR"
cd "$GATUS_DIR" || return 1
local TZ_VAL="${SITE_TZ:-$(cat /etc/timezone 2>/dev/null || echo UTC)}"
# Mirrors configure_caddy_for_service's own mode resolution (lib/common.sh):
# explicit CADDY_MODE from the site config wins, then a local ~/docker/caddy,
# then the legacy CADDY_REMOTE_HOST var. Only "local" joins caddy_net — a
# remote Caddy box can't resolve container names on this host's bridge
# network anyway; it reaches this service via the host's published port.
local _CADDY_MODE="${CADDY_MODE:-none}"
[ "$_CADDY_MODE" = "none" ] && [ -d "$DOCKER_DIR/caddy" ] && _CADDY_MODE="local"
[ "$_CADDY_MODE" = "none" ] && [ -n "${CADDY_REMOTE_HOST:-}" ] && _CADDY_MODE="remote"
local _CADDY_NET_BLOCK=""
local _CADDY_NET_SECTION=""
if [ "$_CADDY_MODE" = "local" ]; then
_CADDY_NET_BLOCK=" networks:
- caddy_net
"
_CADDY_NET_SECTION="
networks:
caddy_net:
external: true
name: ${SITE_CADDY_NET:-caddy_net}
"
fi
cat > docker-compose.yml << GATUS_COMPOSE
name: gatus
services:
gatus:
image: twinproduction/gatus:latest
container_name: gatus
hostname: gatus
restart: unless-stopped
env_file: .env
ports:
- "${WEB_PORT}:8080"
volumes:
- ./gatus_config:/config
- ./gatus_data:/data
${_CADDY_NET_BLOCK}${_CADDY_NET_SECTION}
GATUS_COMPOSE
cat > .env << GATUS_ENV
TZ=$TZ_VAL
CADDY_NET=$SITE_CADDY_NET
GATUS_ENV
# Write a sample config if none exists
if [ ! -f gatus_config/config.yaml ]; then
cat > gatus_config/config.yaml << 'GATUS_CFG'
# Gatus configuration — docs: https://github.com/TwiN/gatus
#
# Add or remove endpoints below. Config is hot-reloaded on changes.
# Alert types: ntfy, slack, discord, email, telegram, and more.
storage:
type: sqlite
path: /data/gatus.db
ui:
title: "Status"
header: "Services"
# ── Endpoints ─────────────────────────────────────────────────────────────────
endpoints:
- name: Google DNS
group: external
url: "8.8.8.8"
dns:
query-name: "google.com"
query-type: "A"
interval: 5m
conditions:
- "[DNS_RCODE] == NOERROR"
- name: Example HTTPS
group: external
url: "https://example.com"
interval: 5m
conditions:
- "[STATUS] == 200"
- "[RESPONSE_TIME] < 3000"
- "[CERTIFICATE_EXPIRATION] > 48h"
# ── Add your services below ────────────────────────────────────────────────
# - name: Mealie
# group: homelab
# url: "http://mealie:9000/api/app/about"
# interval: 1m
# conditions:
# - "[STATUS] == 200"
# - "[RESPONSE_TIME] < 500"
#
# - name: Portainer
# group: homelab
# url: "https://portainer:9443"
# interval: 1m
# conditions:
# - "[STATUS] == 200"
# client:
# insecure: true
GATUS_CFG
fi
chown -R "$ACTUAL_USER:$ACTUAL_USER" "$GATUS_DIR"
log_success "Gatus configured at $GATUS_DIR"
configure_caddy_for_service "Gatus" "gatus:8080" "status"
write_readme "$GATUS_DIR" << MD
# Gatus — status & uptime monitoring
Clean, self-hosted status page. Polls HTTP, TCP, DNS, and ICMP endpoints.
## Access
- URL: http://localhost:${WEB_PORT}
## Configuration
Edit \`gatus_config/config.yaml\` — changes are **hot-reloaded** without restarting.
Key concepts:
- \`endpoints:\` — what to check (HTTP, TCP, DNS, ICMP)
- \`interval:\` — how often (e.g. 1m, 5m)
- \`conditions:\` — pass/fail rules ([STATUS], [RESPONSE_TIME], etc.)
- \`alerts:\` — notify via ntfy, Slack, Discord, email, etc.
Full docs: https://github.com/TwiN/gatus
## Caddyfile sync
If you enabled it during install, \`sync-from-caddy.sh\` runs every 15
minutes (systemd timer: \`gatus-caddy-sync\`) and keeps one endpoint per
Caddy site block in sync automatically — added when a new site appears,
removed when one goes away. Every auto-added endpoint is tagged
\`group: caddy-sync\`; the sync only ever touches entries in that group,
so anything you add or edit by hand (a different group, the default
external checks) is left alone no matter what.
Run it by hand any time instead of waiting for the timer:
\`\`\`bash
$GATUS_DIR/sync-from-caddy.sh $DOCKER_DIR/caddy/Caddyfile $GATUS_DIR/gatus_config/config.yaml
\`\`\`
Check/manage the timer:
\`\`\`bash
systemctl status gatus-caddy-sync.timer
systemctl list-timers gatus-caddy-sync.timer
sudo systemctl disable --now gatus-caddy-sync.timer # turn it off
\`\`\`
## Manage
\`\`\`bash
cd $GATUS_DIR
docker compose up -d # start
docker compose down # stop
docker compose logs -f # logs (check config errors here)
docker compose pull && docker compose up -d # update
\`\`\`
MD
local START_GATUS=""
prompt_yn "Start Gatus now? (y/n):" "y" START_GATUS
if [ "$START_GATUS" = "y" ] || [ "$START_GATUS" = "Y" ]; then
docker compose up -d \
&& log_success "Gatus started" \
|| log_warning "Failed to start — check: docker compose logs"
fi
_gatus_setup_caddy_sync "$GATUS_DIR"
echo " Access at: http://localhost:${WEB_PORT}"
echo " Config: $GATUS_DIR/gatus_config/config.yaml (hot-reloaded)"
echo ""
}
# Run immediately when executed directly (deferred until after function definition)
[[ "${_RUN_STANDALONE:-0}" == 1 ]] && install_gatus