frigate_w_audio/HANDOFF-add-site-script.md

# HANDOFF -- script to add a site behind Authelia

This doc is for a fresh session where you'll write a script that
automates the "add a new site behind Authelia" workflow currently done
by hand. Everything below is the context you need to write it
correctly.

## Goal

One command, e.g.

```
./add-site --subdomain doorbell --upstream 192.168.1.60:5555 --case 1 --policy two_factor
```

does all of:

1. Adds the matching rule to `authelia/configuration.yml` under
   `access_control.rules:`.
2. Adds (or modifies) the matching site block in the user's Caddyfile.
   If a site block already exists with `basic_auth { ... }`, removes
   that and inserts `import authelia` instead.
3. Validates Authelia config (`authelia validate-config`).
4. Validates Caddy config (`caddy validate`).
5. Restarts Authelia, then reloads Caddy. Order matters.
6. Optionally `curl`-tests the new URL and reports.

Idempotent: re-running with the same args is a no-op.

## Stack context (what's already running)

- **Authelia 4.39.19**, file backend, SQLite local storage, in-memory
  sessions, filesystem notifier. Compose project lives at
  `~/docker/authelia/`. Container name `authelia`.
- **fail2ban** as a sidecar in the same compose project. Watches
  `./authelia/authelia.log` and `/var/log/caddy/access.log`.
- **Caddy** is in its own compose project at `~/docker/caddy/`
  (assumption -- script should accept the path as a parameter).
  Container name `caddy`. On the external docker network `caddy_net`.
- Both Caddy and Authelia are on `caddy_net`. Caddy reaches Authelia
  as `authelia:9091`.
- The portal is `auth.{DOMAIN}` with `policy: bypass`.
- `default_policy: deny` -- every gated domain MUST have a rule.
- DOMAIN substitution:
  - Authelia uses Go templates: `'{{ env "DOMAIN" }}'`. Requires
    `X_AUTHELIA_CONFIG_FILTERS=template` env var (already set in the
    docker-compose.yml).
  - Caddy uses `{env.DOMAIN}`. The Caddy compose passes `DOMAIN`
    through to the Caddy container.

## Files the script touches

| File | What lives there | Who edits |
|------|------------------|-----------|
| `~/docker/authelia/authelia/configuration.yml` | `access_control.rules:` list | the script |
| `<user's Caddyfile>` (path: parameter) | site blocks | the script |
| `~/docker/authelia/.env` | `DOMAIN=...`, `TZ=...`, version pins | read-only (script reads DOMAIN from here OR the environment) |

The script does NOT touch:
- `authelia/users_database.yml` (user management is separate)
- `authelia/secrets/*` (manual one-time bootstrap)
- `frigate_config/config.yml` or any other app's own config
  (case 2a/2b require app-side edits the script can't safely automate
  -- it should print instructions instead)
- DNS, TLS, anything outside the local Caddy + Authelia configs

## Manual workflow the script automates

For reference, here's what a human does today to add `foo.example.com`
as a case-1 site:

```bash
# 1. Edit authelia/configuration.yml -- add under access_control.rules:
#    - domain: 'foo.{{ env "DOMAIN" }}'
#      policy: 'two_factor'

# 2. Edit your real Caddyfile -- add a new block (or modify existing):
#    foo.{env.DOMAIN} {
#        import accesslog
#        import authelia
#        reverse_proxy 192.168.1.60:5555
#    }

# 3. Validate before reloading
docker compose -f ~/docker/authelia/docker-compose.yml run --rm authelia \
    authelia validate-config --config /config/configuration.yml

docker compose -f ~/docker/caddy/docker-compose.yml exec caddy \
    caddy validate --config /etc/caddy/Caddyfile

# 4. Restart Authelia FIRST (so the rule is live before Caddy starts
#    forwarding to it -- otherwise default_policy: deny returns 403)
docker compose -f ~/docker/authelia/docker-compose.yml restart authelia

# 5. Reload Caddy (zero-downtime)
docker compose -f ~/docker/caddy/docker-compose.yml exec caddy \
    caddy reload --config /etc/caddy/Caddyfile

# 6. Test
curl -sI -o /dev/null -w "%{http_code}\n" https://foo.example.com
# expect 302 redirect to auth.example.com
```

## Cases the script must handle

The full case taxonomy is in `authelia/configuration.yml` and
`caddy/snippets.caddyfile`. Summary:

### Case 1 -- no app auth, Authelia is the only gate

**Authelia rule:**
```yaml
- domain: 'SUBDOMAIN.{{ env "DOMAIN" }}'
  policy: 'two_factor'    # or one_factor
```

**Caddy block:**
```caddyfile
SUBDOMAIN.{env.DOMAIN} {
    import accesslog
    import authelia
    reverse_proxy UPSTREAM_IP:UPSTREAM_PORT
}
```

Most common case. The migration path from `basic_auth` -- the script
should handle "block exists with basic_auth, swap it for import authelia".

### Case 2a -- app supports trusted-header proxy auth

Same Caddy block as case 1. Same Authelia rule. **Plus** an app-side
config change the script CANNOT safely automate (each app is different:
Frigate's `auth.enabled: False`, Grafana's `[auth.proxy]` section,
Gitea's `ENABLE_REVERSE_PROXY_AUTHENTICATION`, etc.). Script should
print app-specific instructions from a lookup table and require
`--ack-app-config-done` to proceed.

### Case 2b -- app supports OIDC

Same Caddy block as case 1. Same Authelia rule. **Plus** an
`identity_providers.oidc.clients[]` entry to add to
`configuration.yml`. Each app needs its own `client_id`,
`client_secret`, `redirect_uris`, etc.

This is more involved. **Suggested**: out of scope for v1 of the script;
print a pointer to Authelia's OIDC docs and skip.

### Case 3 -- app keeps its own auth, Authelia adds 2FA in front

Caddy block and Authelia rule are identical to case 1. The user just
keeps logging into the app after Authelia. The script doesn't need to
distinguish case 1 from case 3 mechanically -- the only difference is
the user's mental model.

### Case 4 -- no Authelia involvement

```caddyfile
SUBDOMAIN.{env.DOMAIN} {
    import accesslog                # NO import authelia
    reverse_proxy UPSTREAM_IP:UPSTREAM_PORT
}
```

**No Authelia rule.** The script's job is just the Caddy edit + reload.
Useful for things like Plex/Emby where Authelia's redirect breaks
native clients.

## Validation hooks the script must run

In order, before any reload:

1. **Authelia config:**
   ```bash
   docker compose -f ~/docker/authelia/docker-compose.yml run --rm authelia \
       authelia validate-config --config /config/configuration.yml
   ```
   Exit 0 = good. Anything else = abort, restore the file from backup.

2. **Caddy config:**
   ```bash
   docker compose -f ~/docker/caddy/docker-compose.yml exec caddy \
       caddy validate --config /etc/caddy/Caddyfile
   ```
   Exit 0 = good. Anything else = abort, restore Caddyfile from backup.

After reload:

3. **HTTP probe:**
   ```bash
   curl -sI -o /dev/null -w "%{http_code}\n" https://SUBDOMAIN.DOMAIN
   ```
   - Case 1 / 2a / 2b / 3: expect `302` (redirect to Authelia).
   - Case 4: expect `200` or whatever the upstream returns.
   - `403` means the Caddy block has `import authelia` but the Authelia
     rule isn't in place (or wasn't picked up). Most common script bug.

## Reload semantics

**Order**: Authelia restart, THEN Caddy reload. Reverse order is briefly
broken: Caddy starts forwarding to Authelia for a domain Authelia
doesn't yet have a rule for, and `default_policy: deny` returns 403 to
the user.

**Authelia restart**: full container restart. ~3-5 second outage on
auth.{DOMAIN}. Acceptable for household use; if you want zero-downtime
later, Authelia supports config reload via SIGHUP -- not used here.

**Caddy reload**: `caddy reload` is genuinely zero-downtime. It loads
the new config, validates, and atomically swaps. If validation fails
the old config keeps running.

**Rollback**: if anything fails, the script should revert the file
edits from a `.bak` it took before mutating. Easiest pattern:

```bash
cp authelia/configuration.yml authelia/configuration.yml.bak
cp $CADDYFILE_PATH $CADDYFILE_PATH.bak
# ... edits ...
# if any validate fails:
mv authelia/configuration.yml.bak authelia/configuration.yml
mv $CADDYFILE_PATH.bak $CADDYFILE_PATH
```

## Idempotency

The script must detect and short-circuit when the desired end state
already exists:

- **Authelia rule already present**: parse `access_control.rules:`, look
  for an entry with matching `domain:`. If found and policy matches, skip
  the YAML edit. If found and policy differs, prompt or fail (don't
  silently overwrite).
- **Caddy block already present with `import authelia`**: parse the
  Caddyfile, look for `SUBDOMAIN.{env.DOMAIN} {` block. If found and
  it has `import authelia`, skip the Caddy edit.
- **Caddy block exists with `basic_auth`**: this is the migration case.
  Remove the `basic_auth { ... }` lines, add `import authelia` if
  missing. Preserve everything else (transport, header_up, etc).
- **Caddy block exists without auth at all**: just add `import authelia`.

After all checks: if neither file actually changed, skip both reloads
(important -- restarting Authelia for a no-op kicks every active
session).

## Failure modes the script must handle

- `DOMAIN` env var not set / `.env` not present -> abort early with
  clear error.
- `caddy_net` docker network doesn't exist -> abort.
- Authelia container not running -> can still edit config and validate;
  reload step needs to be skipped or attempted with helpful error.
- Caddy container not running -> same.
- Caddyfile path doesn't exist or isn't writable -> abort.
- The Caddyfile doesn't have `(authelia)` and `(accesslog)` snippets
  defined -> the script could either inject them at the top of the file
  (fragile) or refuse and tell the user to run a one-time setup step
  first. **Recommended**: refuse, with a clear "run `./bootstrap-caddy`
  first" message.
- Subdomain conflicts with an existing block that's NOT just a
  `basic_auth` migration target (e.g. an entirely different upstream)
  -> prompt, don't auto-overwrite.
- A site block exists with `basic_auth` AND something else complicated
  (custom matchers, multiple `handle` blocks) -> migration is hard.
  Recommended: detect the simple case (single `basic_auth { ... }`
  inside the block) and refuse the complex case.
- `validate-config` or `caddy validate` fails -> rollback both files,
  report the validator's stderr, exit non-zero.
- HTTP probe fails post-reload -> log the symptom but don't auto-revert;
  user may have DNS not pointing yet, etc.

## YAML editing -- preserve comments

`authelia/configuration.yml` has substantial comments (the
"ALSO PASTE INTO CADDYFILE" blocks, case explanations). A naive YAML
round-trip will eat them. Use a comment-preserving library:

- **Python**: `ruamel.yaml` with `YAML(typ='rt')` (round-trip mode).
- **Go**: `gopkg.in/yaml.v3` is comment-aware.
- **`yq`** (the Go-based one from mikefarah): preserves comments
  reasonably well for simple ops. Adding a list item:
  ```bash
  yq -i '.access_control.rules += [{"domain": "foo.{{ env \"DOMAIN\" }}", "policy": "two_factor"}]' \
      authelia/configuration.yml
  ```
  Note the escaping pain with `{{ env "DOMAIN" }}`. Test before
  committing.

The script should anchor inserts at a stable location. The least-bad
anchor is the END of `access_control.rules:` -- always append, never
splice in the middle.

## Caddyfile editing -- there is no good parser

Caddyfile has its own grammar; standard YAML/JSON tools won't touch it.
Options, in order of pragmatism:

1. **Text-based pattern matching** (recommended for v1). Anchors:
   - Find `^SUBDOMAIN\.\{env\.DOMAIN\} \{$` to detect existing block.
   - For the basic_auth migration: use a small state machine to find
     `basic_auth {` ... `}` inside the matched block and delete those
     lines, then ensure `import authelia` and `import accesslog` lines
     exist.
   - For new-block insertion: append at end of file, separated by a
     blank line.

2. **`caddy adapt`**: converts Caddyfile to JSON. You could edit the
   JSON, then... there's no Caddyfile emitter. Adapt is one-way. Skip.

3. **`caddy fmt`**: normalizes whitespace in a Caddyfile, doesn't
   semantically edit. Useful AFTER your edits to clean up.

The text-based approach is fragile for arbitrary Caddyfiles but
predictable for the conventions this stack uses (one site block per
subdomain, snippets imported at top, no exotic matchers in gated
sites).

## Suggested architecture

```
add-site
├── lib/
│   ├── env.sh              # find DOMAIN, container names, Caddyfile path
│   ├── yaml_edit.sh        # ruamel.yaml or yq wrapper for access_control
│   ├── caddyfile_edit.sh   # awk/sed-based site-block patcher
│   ├── validate.sh         # authelia + caddy validators
│   └── reload.sh           # restart authelia + reload caddy in order
├── cases/
│   ├── case-1.sh           # no-app-auth recipe
│   ├── case-2a.md          # printable app-side instructions table
│   ├── case-2b.md          # OIDC out-of-scope notice + pointer
│   └── case-4.sh           # no-Authelia recipe
├── add-site                # main entrypoint
└── README.md
```

Or in Python with `ruamel.yaml` and a small Caddyfile patcher class.
Either is fine; the bash version has fewer install steps for an
end-user.

## Concrete test cases the script must pass

Use these as fixtures.

### Test 1: fresh case-1 add

Pre-state:
- `authelia/configuration.yml` has only the `auth.{DOMAIN}` bypass rule.
- Caddyfile has `(authelia)` and `(accesslog)` snippets defined, no
  `foo.{env.DOMAIN}` block.

Invocation:
```
./add-site --subdomain foo --upstream 192.168.1.60:5555 --case 1
```

Post-state:
- `access_control.rules:` has new entry for `foo.{{ env "DOMAIN" }}`.
- Caddyfile has new `foo.{env.DOMAIN} { ... }` block with `import
  authelia` and `import accesslog`.
- `validate-config` and `caddy validate` both pass.
- `curl -sI https://foo.example.com` returns 302.

### Test 2: idempotent re-run

Run Test 1's invocation twice. Second run: no file edits, no reloads,
exit 0 with "already configured" message.

### Test 3: basic_auth migration

Pre-state:
- Caddyfile has `foo.{env.DOMAIN} { basic_auth { user $2a$... }
  reverse_proxy ... }`.

Invocation:
```
./add-site --subdomain foo --case 1 --migrate-basic-auth
```

Post-state:
- The `basic_auth { ... }` lines are gone.
- `import authelia` and `import accesslog` are present.
- `reverse_proxy` line is unchanged.
- Authelia rule added.

### Test 4: validation failure rollback

Pre-state: introduce a typo by hand into the YAML insert template
(e.g. `polciy:` instead of `policy:`). Simulate by mocking the
template.

Expected: `validate-config` fails, both files restored from `.bak`,
non-zero exit, no reload attempted.

### Test 5: case-4 (no Authelia)

Invocation:
```
./add-site --subdomain plex --upstream 192.168.1.5:32400 --case 4
```

Post-state:
- Caddyfile has new `plex.{env.DOMAIN}` block with `import accesslog`,
  NO `import authelia`.
- `access_control.rules:` is UNCHANGED.
- Caddy reloads, Authelia is NOT restarted.

## Out of scope (tell the user, don't try to automate)

- DNS A-record creation. Caddy will fail to issue a cert for a domain
  that doesn't resolve. Print a "make sure DNS is pointing at this
  host" reminder when the script starts.
- TLS / Let's Encrypt failures. Caddy auto-provisions; if it fails,
  it's usually DNS or rate-limit. The script should not try to debug.
- App-side proxy-auth config (case 2a). Each app is different. Print
  the lookup-table snippet from `caddy/snippets.caddyfile` for that
  app and require `--ack-app-config-done` before running.
- OIDC client setup (case 2b). Big enough that it deserves its own
  tool. Out of scope.
- User management (`users_database.yml`).
- Secret rotation (`authelia/secrets/*`).
- TOTP enrollment.

## Quick reference: existing files

If you want to read what's there to understand the conventions:

- `docker-compose.yml` -- the auth stack compose, including the
  `X_AUTHELIA_CONFIG_FILTERS=template` and `DOMAIN=${DOMAIN}` env vars.
- `authelia/configuration.yml` -- reference for the rule format,
  comment style, and the "ALSO PASTE INTO CADDYFILE" blocks under
  each case in `access_control.rules:`.
- `caddy/snippets.caddyfile` -- canonical examples of Caddy site
  blocks for every case, including app-specific notes the case-2a
  table can be extracted from.
- `.env.example` -- the shape of `.env` (DOMAIN, TZ, version pins).
- `README.md` -- the comprehensive bootstrap walkthrough; the
  "Adding a new protected site" section is what the script
  automates.

## When in doubt

- For YAML edits, dry-run with
  `docker compose run --rm authelia authelia config template --config
  /config/configuration.yml` -- prints the rendered config so you can
  see exactly what Authelia will see.
- For Caddyfile edits, `caddy fmt --overwrite Caddyfile` normalizes
  whitespace and `caddy validate` catches syntax errors. Run BOTH
  before any reload.
- The single most useful debug command for "why is my site getting
  403":
  ```bash
  docker compose -f ~/docker/authelia/docker-compose.yml exec authelia \
      tail -f /config/authelia.log
  ```
  Then hit the URL. The log line tells you exactly which rule (or
  default_policy) made the call.