frigate_w_audio/HANDOFF-add-site-script.md

HANDOFF -- script to add a site behind Authelia

This doc is for a fresh session where you'll write a script that automates the "add a new site behind Authelia" workflow currently done by hand. Everything below is the context you need to write it correctly.

Goal

One command, e.g.

./add-site --subdomain doorbell --upstream 192.168.1.60:5555 --case 1 --policy two_factor

does all of:

1. Adds the matching rule to authelia/configuration.yml under access_control.rules:.
2. Adds (or modifies) the matching site block in the user's Caddyfile. If a site block already exists with basic_auth { ... }, removes that and inserts import authelia instead.
3. Validates Authelia config (authelia validate-config).
4. Validates Caddy config (caddy validate).
5. Restarts Authelia, then reloads Caddy. Order matters.
6. Optionally curl-tests the new URL and reports.

Idempotent: re-running with the same args is a no-op.

Stack context (what's already running)

• Authelia 4.39.19, file backend, SQLite local storage, in-memory sessions, filesystem notifier. Compose project lives at ~/docker/authelia/. Container name authelia.
• fail2ban as a sidecar in the same compose project. Watches ./authelia/authelia.log and /var/log/caddy/access.log.
• Caddy is in its own compose project at ~/docker/caddy/ (assumption -- script should accept the path as a parameter). Container name caddy. On the external docker network caddy_net.
• Both Caddy and Authelia are on caddy_net. Caddy reaches Authelia as authelia:9091.
• The portal is auth.{DOMAIN} with policy: bypass.
• default_policy: deny -- every gated domain MUST have a rule.
• DOMAIN substitution:
  • Authelia uses Go templates: '{{ env "DOMAIN" }}'. Requires X_AUTHELIA_CONFIG_FILTERS=template env var (already set in the docker-compose.yml).
  • Caddy uses {env.DOMAIN}. The Caddy compose passes DOMAIN through to the Caddy container.

Files the script touches

File	What lives there	Who edits
~/docker/authelia/authelia/configuration.yml	access_control.rules: list	the script
<user's Caddyfile> (path: parameter)	site blocks	the script
~/docker/authelia/.env	DOMAIN=..., TZ=..., version pins	read-only (script reads DOMAIN from here OR the environment)

The script does NOT touch:

• authelia/users_database.yml (user management is separate)
• authelia/secrets/* (manual one-time bootstrap)
• frigate_config/config.yml or any other app's own config (case 2a/2b require app-side edits the script can't safely automate -- it should print instructions instead)
• DNS, TLS, anything outside the local Caddy + Authelia configs

Manual workflow the script automates

For reference, here's what a human does today to add foo.example.com as a case-1 site:

# 1. Edit authelia/configuration.yml -- add under access_control.rules:
#    - domain: 'foo.{{ env "DOMAIN" }}'
#      policy: 'two_factor'

# 2. Edit your real Caddyfile -- add a new block (or modify existing):
#    foo.{env.DOMAIN} {
#        import accesslog
#        import authelia
#        reverse_proxy 192.168.1.60:5555
#    }

# 3. Validate before reloading
docker compose -f ~/docker/authelia/docker-compose.yml run --rm authelia \
    authelia validate-config --config /config/configuration.yml

docker compose -f ~/docker/caddy/docker-compose.yml exec caddy \
    caddy validate --config /etc/caddy/Caddyfile

# 4. Restart Authelia FIRST (so the rule is live before Caddy starts
#    forwarding to it -- otherwise default_policy: deny returns 403)
docker compose -f ~/docker/authelia/docker-compose.yml restart authelia

# 5. Reload Caddy (zero-downtime)
docker compose -f ~/docker/caddy/docker-compose.yml exec caddy \
    caddy reload --config /etc/caddy/Caddyfile

# 6. Test
curl -sI -o /dev/null -w "%{http_code}\n" https://foo.example.com
# expect 302 redirect to auth.example.com

Cases the script must handle

The full case taxonomy is in authelia/configuration.yml and caddy/snippets.caddyfile. Summary:

Case 1 -- no app auth, Authelia is the only gate

Authelia rule:

- domain: 'SUBDOMAIN.{{ env "DOMAIN" }}'
  policy: 'two_factor'    # or one_factor

Caddy block:

SUBDOMAIN.{env.DOMAIN} {
    import accesslog
    import authelia
    reverse_proxy UPSTREAM_IP:UPSTREAM_PORT
}

Most common case. The migration path from basic_auth -- the script should handle "block exists with basic_auth, swap it for import authelia".

Case 2a -- app supports trusted-header proxy auth

Same Caddy block as case 1. Same Authelia rule. Plus an app-side config change the script CANNOT safely automate (each app is different: Frigate's auth.enabled: False, Grafana's [auth.proxy] section, Gitea's ENABLE_REVERSE_PROXY_AUTHENTICATION, etc.). Script should print app-specific instructions from a lookup table and require --ack-app-config-done to proceed.

Case 2b -- app supports OIDC

Same Caddy block as case 1. Same Authelia rule. Plus an identity_providers.oidc.clients[] entry to add to configuration.yml. Each app needs its own client_id, client_secret, redirect_uris, etc.

This is more involved. Suggested: out of scope for v1 of the script; print a pointer to Authelia's OIDC docs and skip.

Case 3 -- app keeps its own auth, Authelia adds 2FA in front

Caddy block and Authelia rule are identical to case 1. The user just keeps logging into the app after Authelia. The script doesn't need to distinguish case 1 from case 3 mechanically -- the only difference is the user's mental model.

Case 4 -- no Authelia involvement

SUBDOMAIN.{env.DOMAIN} {
    import accesslog                # NO import authelia
    reverse_proxy UPSTREAM_IP:UPSTREAM_PORT
}

No Authelia rule. The script's job is just the Caddy edit + reload. Useful for things like Plex/Emby where Authelia's redirect breaks native clients.

Validation hooks the script must run

In order, before any reload:

1. Authelia config:

   docker compose -f ~/docker/authelia/docker-compose.yml run --rm authelia \
       authelia validate-config --config /config/configuration.yml
   

   Exit 0 = good. Anything else = abort, restore the file from backup.

2. Caddy config:

   docker compose -f ~/docker/caddy/docker-compose.yml exec caddy \
       caddy validate --config /etc/caddy/Caddyfile
   

   Exit 0 = good. Anything else = abort, restore Caddyfile from backup.

After reload:

3. HTTP probe:

   curl -sI -o /dev/null -w "%{http_code}\n" https://SUBDOMAIN.DOMAIN
   

   • Case 1 / 2a / 2b / 3: expect 302 (redirect to Authelia).
   • Case 4: expect 200 or whatever the upstream returns.
   • 403 means the Caddy block has import authelia but the Authelia rule isn't in place (or wasn't picked up). Most common script bug.

Reload semantics

Order: Authelia restart, THEN Caddy reload. Reverse order is briefly broken: Caddy starts forwarding to Authelia for a domain Authelia doesn't yet have a rule for, and default_policy: deny returns 403 to the user.

Authelia restart: full container restart. ~3-5 second outage on auth.{DOMAIN}. Acceptable for household use; if you want zero-downtime later, Authelia supports config reload via SIGHUP -- not used here.

Caddy reload: caddy reload is genuinely zero-downtime. It loads the new config, validates, and atomically swaps. If validation fails the old config keeps running.

Rollback: if anything fails, the script should revert the file edits from a .bak it took before mutating. Easiest pattern:

cp authelia/configuration.yml authelia/configuration.yml.bak
cp $CADDYFILE_PATH $CADDYFILE_PATH.bak
# ... edits ...
# if any validate fails:
mv authelia/configuration.yml.bak authelia/configuration.yml
mv $CADDYFILE_PATH.bak $CADDYFILE_PATH

Idempotency

The script must detect and short-circuit when the desired end state already exists:

• Authelia rule already present: parse access_control.rules:, look for an entry with matching domain:. If found and policy matches, skip the YAML edit. If found and policy differs, prompt or fail (don't silently overwrite).
• Caddy block already present with import authelia: parse the Caddyfile, look for SUBDOMAIN.{env.DOMAIN} { block. If found and it has import authelia, skip the Caddy edit.
• Caddy block exists with basic_auth: this is the migration case. Remove the basic_auth { ... } lines, add import authelia if missing. Preserve everything else (transport, header_up, etc).
• Caddy block exists without auth at all: just add import authelia.

After all checks: if neither file actually changed, skip both reloads (important -- restarting Authelia for a no-op kicks every active session).

Failure modes the script must handle

• DOMAIN env var not set / .env not present -> abort early with clear error.
• caddy_net docker network doesn't exist -> abort.
• Authelia container not running -> can still edit config and validate; reload step needs to be skipped or attempted with helpful error.
• Caddy container not running -> same.
• Caddyfile path doesn't exist or isn't writable -> abort.
• The Caddyfile doesn't have (authelia) and (accesslog) snippets defined -> the script could either inject them at the top of the file (fragile) or refuse and tell the user to run a one-time setup step first. Recommended: refuse, with a clear "run ./bootstrap-caddy first" message.
• Subdomain conflicts with an existing block that's NOT just a basic_auth migration target (e.g. an entirely different upstream) -> prompt, don't auto-overwrite.
• A site block exists with basic_auth AND something else complicated (custom matchers, multiple handle blocks) -> migration is hard. Recommended: detect the simple case (single basic_auth { ... } inside the block) and refuse the complex case.
• validate-config or caddy validate fails -> rollback both files, report the validator's stderr, exit non-zero.
• HTTP probe fails post-reload -> log the symptom but don't auto-revert; user may have DNS not pointing yet, etc.

YAML editing -- preserve comments

authelia/configuration.yml has substantial comments (the "ALSO PASTE INTO CADDYFILE" blocks, case explanations). A naive YAML round-trip will eat them. Use a comment-preserving library:

• Python: ruamel.yaml with YAML(typ='rt') (round-trip mode).
• Go: gopkg.in/yaml.v3 is comment-aware.
• yq (the Go-based one from mikefarah): preserves comments reasonably well for simple ops. Adding a list item:

  yq -i '.access_control.rules += [{"domain": "foo.{{ env \"DOMAIN\" }}", "policy": "two_factor"}]' \
      authelia/configuration.yml
  

  Note the escaping pain with {{ env "DOMAIN" }}. Test before committing.

The script should anchor inserts at a stable location. The least-bad anchor is the END of access_control.rules: -- always append, never splice in the middle.

Caddyfile editing -- there is no good parser

Caddyfile has its own grammar; standard YAML/JSON tools won't touch it. Options, in order of pragmatism:

1. Text-based pattern matching (recommended for v1). Anchors:

   • Find ^SUBDOMAIN\.\{env\.DOMAIN\} \{$ to detect existing block.
   • For the basic_auth migration: use a small state machine to find basic_auth { ... } inside the matched block and delete those lines, then ensure import authelia and import accesslog lines exist.
   • For new-block insertion: append at end of file, separated by a blank line.

2. caddy adapt: converts Caddyfile to JSON. You could edit the JSON, then... there's no Caddyfile emitter. Adapt is one-way. Skip.

3. caddy fmt: normalizes whitespace in a Caddyfile, doesn't semantically edit. Useful AFTER your edits to clean up.

The text-based approach is fragile for arbitrary Caddyfiles but predictable for the conventions this stack uses (one site block per subdomain, snippets imported at top, no exotic matchers in gated sites).

Suggested architecture

add-site
├── lib/
│   ├── env.sh              # find DOMAIN, container names, Caddyfile path
│   ├── yaml_edit.sh        # ruamel.yaml or yq wrapper for access_control
│   ├── caddyfile_edit.sh   # awk/sed-based site-block patcher
│   ├── validate.sh         # authelia + caddy validators
│   └── reload.sh           # restart authelia + reload caddy in order
├── cases/
│   ├── case-1.sh           # no-app-auth recipe
│   ├── case-2a.md          # printable app-side instructions table
│   ├── case-2b.md          # OIDC out-of-scope notice + pointer
│   └── case-4.sh           # no-Authelia recipe
├── add-site                # main entrypoint
└── README.md

Or in Python with ruamel.yaml and a small Caddyfile patcher class. Either is fine; the bash version has fewer install steps for an end-user.

Concrete test cases the script must pass

Use these as fixtures.

Test 1: fresh case-1 add

Pre-state:

• authelia/configuration.yml has only the auth.{DOMAIN} bypass rule.
• Caddyfile has (authelia) and (accesslog) snippets defined, no foo.{env.DOMAIN} block.

Invocation:

./add-site --subdomain foo --upstream 192.168.1.60:5555 --case 1

Post-state:

• access_control.rules: has new entry for foo.{{ env "DOMAIN" }}.
• Caddyfile has new foo.{env.DOMAIN} { ... } block with import authelia and import accesslog.
• validate-config and caddy validate both pass.
• curl -sI https://foo.example.com returns 302.

Test 2: idempotent re-run

Run Test 1's invocation twice. Second run: no file edits, no reloads, exit 0 with "already configured" message.

Test 3: basic_auth migration

Pre-state:

• Caddyfile has foo.{env.DOMAIN} { basic_auth { user $2a$... } reverse_proxy ... }.

Invocation:

./add-site --subdomain foo --case 1 --migrate-basic-auth

Post-state:

• The basic_auth { ... } lines are gone.
• import authelia and import accesslog are present.
• reverse_proxy line is unchanged.
• Authelia rule added.

Test 4: validation failure rollback

Pre-state: introduce a typo by hand into the YAML insert template (e.g. polciy: instead of policy:). Simulate by mocking the template.

Expected: validate-config fails, both files restored from .bak, non-zero exit, no reload attempted.

Test 5: case-4 (no Authelia)

Invocation:

./add-site --subdomain plex --upstream 192.168.1.5:32400 --case 4

Post-state:

• Caddyfile has new plex.{env.DOMAIN} block with import accesslog, NO import authelia.
• access_control.rules: is UNCHANGED.
• Caddy reloads, Authelia is NOT restarted.

Out of scope (tell the user, don't try to automate)

• DNS A-record creation. Caddy will fail to issue a cert for a domain that doesn't resolve. Print a "make sure DNS is pointing at this host" reminder when the script starts.
• TLS / Let's Encrypt failures. Caddy auto-provisions; if it fails, it's usually DNS or rate-limit. The script should not try to debug.
• App-side proxy-auth config (case 2a). Each app is different. Print the lookup-table snippet from caddy/snippets.caddyfile for that app and require --ack-app-config-done before running.
• OIDC client setup (case 2b). Big enough that it deserves its own tool. Out of scope.
• User management (users_database.yml).
• Secret rotation (authelia/secrets/*).
• TOTP enrollment.

Quick reference: existing files

If you want to read what's there to understand the conventions:

• docker-compose.yml -- the auth stack compose, including the X_AUTHELIA_CONFIG_FILTERS=template and DOMAIN=${DOMAIN} env vars.
• authelia/configuration.yml -- reference for the rule format, comment style, and the "ALSO PASTE INTO CADDYFILE" blocks under each case in access_control.rules:.
• caddy/snippets.caddyfile -- canonical examples of Caddy site blocks for every case, including app-specific notes the case-2a table can be extracted from.
• .env.example -- the shape of .env (DOMAIN, TZ, version pins).
• README.md -- the comprehensive bootstrap walkthrough; the "Adding a new protected site" section is what the script automates.

When in doubt

• For YAML edits, dry-run with docker compose run --rm authelia authelia config template --config /config/configuration.yml -- prints the rendered config so you can see exactly what Authelia will see.
• For Caddyfile edits, caddy fmt --overwrite Caddyfile normalizes whitespace and caddy validate catches syntax errors. Run BOTH before any reload.
• The single most useful debug command for "why is my site getting 403":

  docker compose -f ~/docker/authelia/docker-compose.yml exec authelia \
      tail -f /config/authelia.log
  

  Then hit the URL. The log line tells you exactly which rule (or default_policy) made the call.