The import-photos.sh helper script now handles the entire setup flow
without requiring any manual web UI interaction:
1. Checks if Immich is initialized via /api/server/config
2. If uninitialized: creates admin account via /api/auth/admin-sign-up
(prompts for email, password, name)
3. If already initialized: prompts for existing credentials
4. Logs in via /api/auth/login to get bearer token
5. Creates API key via /api/api-keys automatically
6. Configures storage template via /api/system-config (PUT)
7. Installs immich-cli (checks immich → npx → npm → offers apt install)
8. Runs recursive upload with EXIF date preservation
Also accepts an API key as argument to skip account setup for re-runs.
Fixes:
- Removed -f flag from curl calls that suppressed error details
- Fixed $? check after login (now uses if ! command pattern)
- Variable naming collision (INSTALL_NODE → INSTALL_NODE_YN)
- Post-setup instructions now just say "run the import script"
instead of listing manual steps
https://claude.ai/code/session_01NAtxAkC5t6YVb3gcP1VcX8
When Strategy 1 (unified library) is chosen with existing photos, the
setup now generates an import-photos.sh script that automates the entire
import process:
- Verifies Immich is running (API health check)
- Prompts for API key with validation
- Configures the storage template via API automatically (uses python3
for JSON manipulation, falls back to manual instructions)
- Installs immich-cli via npx if Node.js available, offers to install
Node.js if not
- Runs the upload with --recursive from the baked-in source path
- Shows photo count and progress
Also:
- Ask where existing photos currently live (separate from library path)
- Use that path for both the external library mount (Strategy 2) and
the import script source (Strategy 1)
- Post-setup instructions reduced to 3 steps: create account, get API
key, run import script
- .env comments reference the import script instead of raw CLI commands
https://claude.ai/code/session_01NAtxAkC5t6YVb3gcP1VcX8
Replace the confusing two-prompt (upload/external) configuration with a
cohesive strategy-based flow:
- One question for photo library path instead of two separate prompts
- Ask if user has existing photos, then present two clear strategies:
[1] Import everything into Immich (unified library, recommended)
[2] Keep existing photos in place (external library, read-only)
- Generate strategy-specific docker-compose.yml (no unused external
mount when not needed)
- Strategy-specific .env files with relevant instructions only
- Fix nesting issue: external library uploads now go to sibling dir
instead of subfolder (prevents duplicate scan)
- Unified post-setup instructions that match chosen strategy
https://claude.ai/code/session_01NAtxAkC5t6YVb3gcP1VcX8
- Check for existing .env file instead of extracting from docker-compose
- Replace KC_PROXY with KC_PROXY_HEADERS in .env
- Add KC_PROXY_HEADERS if missing
- Remove KC_PROXY from docker-compose.yml if present
- Show current configuration before restart
- Handles both migration scenarios:
1. Old config with KC_PROXY in .env
2. Existing .env without proxy settings
- Replace deprecated KC_PROXY=edge with KC_PROXY_HEADERS=xforwarded
- Fixes 'Hostname v1 options [proxy] are still in use' warning
- Convert docker-compose-keycloak.yml to use .env file
- Remove hardcoded passwords from docker-compose.yml
- Add comprehensive .env template in comments
- Update deployment instructions and production checklist
- Resolves CORS and secure context warnings
- All credentials now in .env with proper security
This comprehensive update addresses multiple security and usability issues:
## Docker Directory Ownership
- Added ensure_docker_dir_ownership() helper function
- Applied to ALL 25+ services (Immich, Keycloak, ActualBudget, Jellyfin,
Emby, ARM, FileBrowser, MagicMirror, Lyrion, Mealie, Minecraft, Frigate,
ntfy, Uptime Kuma, wg-easy, Traccar, Portainer, MeshCentral, FindMyDevice,
Frigate-Notify, Watchtower, Kopia, Caddy)
- Fixed disaster recovery path (line 309) to set ownership
- Docker folders now owned by sudo user, not root
- Users can run docker commands without sudo
## Keycloak Security Improvements
- Implemented password validation with retry loop
- Password requirements: 12+ chars, alphanumeric only (no special chars)
- Auto-generate secure passwords by pressing ENTER
- Moved all credentials to .env file (no passwords in docker-compose.yml)
- Added production vs development mode selection
- Production mode uses 'start' command with hostname configuration
- Development mode uses 'start-dev' for testing only
- Proper KC_HOSTNAME configuration for public deployments
- Interactive prompts with clear security warnings
## Environment Variable Management
- ActualBudget now uses .env file for configuration
- Keycloak uses .env for admin and database passwords
- Consistent .env pattern across services
- Passwords no longer visible in docker-compose files
- Easier credential management and rotation
## Helper Functions
- ensure_docker_dir_ownership(): Fix ownership recursively
- generate_password(): Generate secure alphanumeric passwords
- validate_password(): Validate Keycloak-compatible passwords
## Documentation
- Added SECURITY-IMPROVEMENTS.md with comprehensive guide
- Password requirements and best practices
- Keycloak setup guide for ActualBudget on Pikapods
- Migration guide for existing services
- Troubleshooting section
- Verification checklist
## Integration Status
- Caddy2 reverse proxy: Already integrated via configure_caddy_for_service()
- fail2ban monitoring: Already configured with labels on all services
- HTTPS and security headers: Already implemented
- JSON logging for fail2ban: Already configured
All services now follow consistent patterns for ownership, credentials,
and security configuration. Script tested with bash -n for syntax errors.
MAJOR IMPROVEMENT: Drive detection now happens ONCE at startup and
is reused by all services, instead of each service detecting separately.
1. **New detect_drives() Function:**
- Runs once before service selection menu
- Scans ~/drives directory for all mounted drives
- Shows drive name, path, size, used space, available space
- Sets global variables for all services to use
2. **Global Variables Set:**
- PRIMARY_DRIVE: name of first drive (e.g., "storage1")
- PRIMARY_DRIVE_PATH: full path to first drive
- DRIVES_DETECTED: true/false
- DRIVES_DIR: base drives directory
- AVAILABLE_DRIVES_COUNT: number of drives
3. **Display Example:**
```
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
DETECTED DRIVES
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
✓ storage1
Path: /home/user/drives/storage1
Size: 2.0T (Used: 800G, Available: 1.2T)
✓ backup
Path: /home/user/drives/backup
Size: 4.0T (Used: 1.5T, Available: 2.5T)
Using 'storage1' as primary drive for default paths
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
```
4. **Updated Services to Use Global Detection:**
- Immich: photos/immich-uploads
- AudioBookshelf: audiobooks
- Emby: media
- Jellyfin: media
- ARM: ripped
- FileBrowser: (root browse)
- LMS: music
5. **Benefits:**
- Detect drives only once (faster)
- Consistent paths across all services
- Shows actual drive information (size, usage)
- Falls back to $HOME if no ~/drives exists
- All services use PRIMARY_DRIVE_PATH variable
This fixes user complaint:
"There is no ~/drives.... it should detect which drives are already
in ~/drives and offer to use one of those"
- Added check_service_exists() to ARM (Automatic Ripping Machine)
- Added check_service_exists() to FileBrowser
- Both now show Skip/Reconfigure/Restart options
- Prevents accidental overwrite of existing configurations
Progress: 5/24 services complete (Immich, AudioBookshelf, Emby, ARM, FileBrowser)
1. **Created check_service_exists() Helper Function:**
- Reusable function to check if service exists
- Shows options: Skip / Reconfigure / Restart
- Handles backup automatically on reconfigure
- Returns boolean flag to control installation flow
- Reduces code duplication across services
2. **Applied Reconfigure Fix to Services:**
- Immich (with drive detection)
- AudioBookshelf
- Emby
3. **How It Works:**
```bash
check_service_exists "ServiceName" "$SERVICE_DIR" RECONFIGURE_FLAG
if [ "$RECONFIGURE_FLAG" = "true" ]; then
# Only run configuration if true
fi
```
4. **Benefits:**
- Consistent behavior across all services
- Automatic backup before any changes
- Safe default (Skip) protects existing configs
- Easy to apply to remaining services
Next: Apply to remaining 20+ services (ActualBudget, Mealie, Jellyfin, etc.)
CRITICAL FIX: Script was reinstalling and reconfiguring services from scratch
when they were already installed, potentially breaking existing configurations.
1. **Detect Existing Installations:**
- Check for existing docker-compose.yml before configuration
- Show options: Skip / Reconfigure / Restart only
- Default to Skip to preserve existing configs
2. **Skip/Reconfigure/Restart Options:**
- Option 1: Skip (keep existing configuration)
- Option 2: Reconfigure (backup existing, then reconfigure)
- Option 3: Restart containers only (no reconfiguration)
3. **Automatic Backup on Reconfigure:**
- Creates timestamped backup before any changes
- Backup location: ~/docker/backups/YYYYMMDD-HHMMSS-servicename/
- Full service directory backed up
4. **Fixed Drive Detection:**
- Detects existing ~/drives directory
- Lists all available drives to user
- Uses detected drives in default paths
- Changed from $HOME_DIR to $ACTUAL_HOME for correct paths
- No more missing "/drives/primary/..." paths
5. **Drive Path Examples:**
- Detects: ~/drives/storage1, ~/drives/backup, ~/drives/media
- Shows: "Detected drives: storage1, backup, media"
- Default becomes: ~/drives/storage1/photos/immich-uploads
Applied to: Immich (template for other services)
This fixes user issues:
- "I selected keep the items that were already installed, then it lead
me to reinstall configure from scratch all those services"
- "It did not detect if the service was already in the Caddyfile"
- "it might have messed up what I already had"
- "There is no ~/drives...."
- "it should detect which drives are already in ~/drives and offer to
use one of those"
Implemented automatic Caddy reverse proxy configuration that runs
BEFORE each service is started, with backup, reload, and formatting.
1. **New configure_caddy_for_service() Function:**
- Detects if Caddy is installed (skips if not)
- Prompts user if they want to configure reverse proxy
- Asks for domain/subdomain (e.g., photos.example.com)
- Backs up Caddyfile with timestamp
- Checks for existing configuration and offers to overwrite
- Adds service block with security headers and fail2ban logging
- Reloads Caddy configuration
- Formats Caddyfile with `caddy fmt --overwrite`
- Final reload after formatting
- Shows final access URL (https://...)
2. **Configuration Sequence:**
- Service docker-compose.yml created
- Service .env configured
- Caddy configuration added (if Caddy installed)
- Caddy reloaded and formatted
- Service containers started
- All happens before `docker compose up -d`
3. **Integrated into Services:**
- Immich (photos.example.com)
- AudioBookshelf (audiobooks.example.com)
- Emby (emby.example.com)
- ActualBudget (budget.example.com)
- Mealie (recipes.example.com)
- Jellyfin (jellyfin.example.com)
- Uptime Kuma (uptime.example.com)
4. **Caddy Configuration Includes:**
- Automatic HTTPS via Let's Encrypt
- Security headers (HSTS, X-Content-Type-Options, etc.)
- JSON logging for fail2ban
- Proper reverse_proxy to localhost:PORT
5. **Backup & Safety:**
- Caddyfile backed up to: Caddyfile.backup.YYYYMMDD-HHMMSS
- Existing configs detected and user can choose to overwrite
- Reload errors show backup file path for restoration
6. **Commands Used:**
- `docker exec caddy caddy reload --config /etc/caddy/Caddyfile`
- `docker exec caddy caddy fmt --overwrite /etc/caddy/Caddyfile`
This addresses user request:
"new/reconfigured services are add to the Caddyfile as part of the
install/reconfigure before they are attempt to be brought up, as a
part of the install/reconfigure of the service? (With backup of
Caddyfile and docker exec -w /etc/caddy caddy caddy reload &&
docker exec -w /etc/caddy caddy caddy fmt --overwrite after each
addition to Caddyfile?)"
Added comprehensive uninstall functionality to whiptail menu:
1. **Install/Uninstall Menu Choice:**
- Added action menu: Install new services, Uninstall existing, or Cancel
- Automatically detects existing services in ~/docker/
2. **Smart Service Detection:**
- Scans for existing Docker services and marks them in the install menu
- Install menu shows [*] for already-installed services
- Helps users identify what's already running on their server
3. **Uninstall Functionality:**
- Uninstall menu only shows services that are currently installed
- All services selected by default for quick removal
- Automatic backup before uninstall to ~/docker/backups/
- Stops containers, removes directories, backs up data
- Special handling for system packages (fail2ban)
4. **Improved Re-run Support:**
- Script detects existing installations on startup
- Can rerun on configured servers to add new services
- Uninstall option allows cleanup of unwanted services
- Backups ensure data safety during removals
Services Supported:
- All 24 Docker services (Immich, Keycloak, Caddy, etc.)
- System packages (fail2ban)
- Backups created with timestamp: YYYYMMDD-HHMMSS-servicename
Usage Examples:
- Fresh install: Select services to install
- Add services: Rerun script, existing services auto-selected
- Remove services: Choose "Uninstall", select what to remove
- Cleanup: All data backed up automatically
This addresses user request for:
- Ability to reinstall on current server
- Option to remove services from whiptail menu
- Better handling of existing installations
Added planning documents for future script reorganization:
NEW-SCRIPT-STRUCTURE.md:
- Outlines two-phase structure (Essential Setup vs Service Selection)
- Documents plan to move optional services to whiptail menu
- Plans for uninstall functionality
- Re-run detection improvements
SCRIPT-FLOW-INTRO.txt:
- Proposed intro text for users
- Explains Phase 1 (required) and Phase 2 (optional)
- Documents re-running behavior
These are planning documents for future enhancements.
Current commit only implements the duplicate prompt fix.
Applied WHIPTAIL_USED flag check to all 25 service prompts that appear
after the whiptail menu to prevent services from being prompted
individually when they weren't selected in the checkbox menu.
This fixes the issue where services like AudioBookshelf were being
prompted even when not selected in whiptail.
Services fixed:
- AudioBookshelf, Emby, ARM, FileBrowser, Magic Mirror
- ActualBudget, Keycloak, Caddy, fail2ban
- Lyrion Music Server, Mealie, Minecraft, Jellyfin, Frigate
- ddclient, ntfy, Uptime Kuma, wg-easy
- Traccar, Portainer, MeshCentral Server
- FindMyDevice, Frigate-Notify, Watchtower
Pattern applied:
if [ "$WHIPTAIL_USED" != true ] && [ -z "$INSTALL_SERVICE" ]; then
# Show prompt
fi
This ensures prompts only appear when whiptail was not used OR
the service variable is not yet set.
CRITICAL FIX:
The whiptail menu was OVERWRITING all service variables with "n",
which caused duplicate prompts and ignored user's earlier selections.
BEFORE (broken):
- User answers y/n prompts
- Whiptail menu appears
- Whiptail sets INSTALL_IMMICH="n" (overwrites previous "y")
- Individual prompt appears again (because variable check fails)
- User gets prompted twice for same service!
AFTER (fixed):
- Variables only set to "n" if not already set
- Uses bash parameter expansion: : ${VAR:="default"}
- Preserves any earlier choices
- Whiptail menu updates to "y" if selected
- Individual prompts skip if variable already set
- No duplicate prompts!
CHANGES:
Lines 2225-2249: Changed from direct assignment (VAR="n")
to conditional default (: ${VAR:="n"})
This preserves earlier choices while still allowing whiptail
to override them when services are selected.
SIDE EFFECT FIXED:
- Containers now start properly
- No more "containers won't come up" issue
- Proper dependency order maintained
FIXES:
1. SSH key import now properly accepts "n" as answer
- Added y/n prompt before asking for usernames
- Clearer flow: "Import SSH keys?" → "Which service?"
- No more confusion about entering "n" vs leaving blank
2. Automatic Caddy configuration for Keycloak
- Detects if Caddy is installed or being installed
- Offers to configure Caddy reverse proxy for Keycloak
- Backs up Caddyfile before changes
- Adds Keycloak configuration automatically
- Reloads Caddy after adding configuration
- Keycloak starts AFTER Caddy is configured
- Prevents "container won't come up" issue
CADDY AUTO-CONFIGURATION:
When both Keycloak and Caddy are selected:
- Script asks: "Configure Caddy reverse proxy for Keycloak?"
- Prompts for domain (e.g., auth.yourdomain.com)
- Backs up existing Caddyfile
- Adds Keycloak block with:
* JSON logging for fail2ban
* Reverse proxy to localhost:8180
* Security headers (HSTS, X-Frame-Options, etc.)
- Formats and reloads Caddy
- Confirms Keycloak will be available at domain
This ensures correct startup order: Caddy configured → Caddy reloaded → Keycloak starts
KEYCLOAK-SETUP-GUIDE.md:
Complete manual explaining Keycloak concepts, manual setup, and external services
WHAT'S A REALM:
- Isolated container for users/clients/config
- Like a "company" or "organization"
- master realm = admin only
- homelab realm = your actual users
- Fully isolated from each other
WHAT'S AN OAUTH2 CLIENT:
- Each service (ActualBudget, etc.) is a "client"
- Needs Client ID, Secret, and Redirect URIs
- Redirect URIs must match EXACTLY
- Guide explains the authentication flow
MANUAL SETUP INSTRUCTIONS:
- Step-by-step via web UI
- Create realm manually
- Create OAuth clients manually
- Configure redirect URIs
- Create users and set passwords
- Test the setup
EXTERNAL SERVICE SUPPORT (Pikapod, etc.):
Script now asks about setup type:
1. Local only (http://localhost)
2. Public domain (https://yourdomain.com)
3. Both local and public
For external services:
- Prompts for your public domain
- Warns that Keycloak MUST be accessible at https://auth.yourdomain.com
- Checks if Caddy/DNS are configured
- Asks for external service URL (e.g., Pikapod)
- Configures redirect URIs for all scenarios
REDIRECT URIS NOW INCLUDE:
- http://localhost:5006/* (local dev)
- https://budget.yourdomain.com/* (self-hosted)
- https://actualbudget-abc.pikapod.net/* (external)
- Multiple patterns for flexibility
SAVED CONFIG FILES UPDATED:
- Shows LOCAL DEVELOPMENT URLs
- Shows PRODUCTION URLs (if public domain set)
- Shows EXTERNAL SERVICE URLs (if external service set)
- Lists all configured redirect URIs
- Clear instructions for each scenario
CADDY CONFIGURATION GUIDE:
- How to configure DNS A/CNAME records
- Caddyfile example for Keycloak
- Security headers included
- Step-by-step setup for external access
RECONFIGURATION SUPPORT:
- Guide explains how to add realms manually
- Guide explains how to add clients manually
- CLI examples for adding realms/users/clients
- Can re-run script to configure additional realms
COMMON USE CASES:
1. All local services
2. Self-hosted with domain
3. Mixed (local + external like Pikapod)
Each use case explained with complete examples
TROUBLESHOOTING:
- Invalid redirect URI
- Client not found
- Invalid client secret
- External service can't reach Keycloak
- CORS/redirect failures
- Admin console login issues
With this update, users can:
✅ Understand what Keycloak is and how it works
✅ Configure it manually if they prefer
✅ Use it with external services like Pikapod
✅ Set up proper DNS/Caddy for production
✅ Troubleshoot common issues
✅ Add realms and clients later
Keycloak is now fully configured and ready to use immediately after installation!
No more manual realm/client setup required.
AUTOMATED SETUP:
After Keycloak starts, the script automatically:
1. ✅ Waits for Keycloak to be fully ready (health check)
2. ✅ Logs in using Keycloak Admin CLI (kcadm.sh)
3. ✅ Creates a new realm (e.g., "homelab")
4. ✅ Creates OAuth2/OIDC client for ActualBudget (if selected)
5. ✅ Creates generic OAuth2 client template for other services
6. ✅ Optionally creates an initial user
7. ✅ Saves all OAuth credentials to text files
8. ✅ Provides clear next steps
OAUTH2 CLIENT FOR ACTUALBUDGET:
- Client ID: actualbudget
- Auto-generated secure client secret
- Pre-configured redirect URIs for localhost and production
- Saved to: ~/docker/keycloak/actualbudget-oauth.txt
- Includes all URLs needed to configure ActualBudget
GENERIC OAUTH2 CLIENT:
- Client ID: generic-app
- Can be cloned for other services
- Saved to: ~/docker/keycloak/generic-oauth.txt
- Works as a template
INITIAL USER CREATION:
- Prompts for username, email, first name, last name, password
- User is immediately active and can log in
- Can be used for ActualBudget and other services right away
SAVED CONFIGURATION FILES:
~/docker/keycloak/actualbudget-oauth.txt - ActualBudget OAuth config
~/docker/keycloak/generic-oauth.txt - Generic OAuth template
PRODUCTION READY:
- Redirect URIs include both localhost and production domains
- Works with Caddy reverse proxy
- SSL/TLS enforced at proxy level
- Just update domain in configuration
USER EXPERIENCE:
Install Keycloak → Answer prompts → DONE!
- Realm created: "homelab" (or custom name)
- OAuth clients ready
- User created and can log in immediately
- Just go to http://localhost:8180/admin to manage
This eliminates the complex post-install Keycloak setup and makes it
immediately usable for ActualBudget and other services!
Users now get a nice checkbox menu to select which services to install,
instead of being prompted for each service one-by-one.
WHIPTAIL MENU:
- Displays all 24+ Docker services in a single checklist
- Use SPACE to select/deselect services
- Press ENTER to confirm and install selected services
- Falls back to individual prompts if whiptail not available
SERVICES IN MENU:
✓ Immich (Photo & Video Backup)
✓ AudioBookshelf (Audiobooks & Podcasts)
✓ Emby (Media Server)
✓ A.R.M. (Automatic Ripping Machine)
✓ FileBrowser (Web File Manager)
✓ Magic Mirror (Smart Mirror Display)
✓ ActualBudget (Personal Finance)
✓ Keycloak (Identity & Access Management)
✓ Caddy (Reverse Proxy with Auto-HTTPS)
✓ fail2ban (Intrusion Prevention)
✓ Lyrion (Music Streaming)
✓ Mealie (Recipe Manager)
✓ Minecraft (Game Server)
✓ Jellyfin (Free Media Server)
✓ Frigate (AI NVR for Cameras)
✓ Ntfy (Push Notifications)
✓ Uptime Kuma (Service Monitoring)
✓ WG-Easy (WireGuard VPN)
✓ Traccar (GPS Tracking)
✓ Portainer (Docker Web UI)
✓ MeshCentral (Remote Management)
✓ FindMyDevice (Device Tracking)
✓ Frigate-Notify (Frigate Notifications)
✓ Watchtower (Auto Container Updates)
WORKFLOW:
1. Run ubuntu-post-install.sh
2. Get whiptail menu for service selection
3. Select services with SPACE
4. Press ENTER to install
5. Script installs only selected services
FALLBACK:
- If whiptail not available, uses traditional prompts
- Prompts only appear if service wasn't selected in menu
- Fully backwards compatible
This dramatically improves UX for installing multiple services!
Users can now install and configure everything by simply running the main script.
Re-running the script allows adding new services to existing installations.
NEW SERVICES IN MAIN SCRIPT:
CADDY WEB SERVER:
- Automatic HTTPS with Let's Encrypt
- Reverse proxy for all services
- Creates example Caddyfile with ActualBudget and Keycloak configs
- Detects existing installations (asks before reconfiguring)
- Automatically backs up existing Caddyfile before changes
- Pre-configured with /var/log/caddy volume for fail2ban integration
- Includes HTTP/3 support
FAIL2BAN INTRUSION PREVENTION:
- Automated installation via apt
- Creates Caddy filter for JSON logs (401, 403, 429 status codes)
- Creates Caddy jail with configurable settings
- Automatically creates /var/log/caddy directory
- Tests configuration before restart
- Verifies jail is active after restart
- Shows status and useful commands
FEATURES:
✅ Detects if services already exist (won't overwrite)
✅ Backs up configurations before changes
✅ Interactive prompts for all settings
✅ Validates configurations before applying
✅ Can be re-run to add services to existing setup
✅ Works alongside existing services
✅ Follows same pattern as ActualBudget/Keycloak
WORKFLOW:
1. Run ubuntu-post-install.sh
2. Select services to install (ActualBudget, Keycloak, Caddy, fail2ban, etc.)
3. Script handles everything automatically
4. Re-run anytime to add more services
The caddy-setup-helper.sh remains available as a standalone tool for
advanced configuration, but the main script is now the primary method.
The caddy-setup-helper.sh script now handles everything automatically (after asking
for confirmation), only falling back to manual instructions if errors occur.
AUTOMATED WORKFLOW:
1. ✅ Backup Caddyfile (ALWAYS FIRST - before any changes)
2. ✅ Check if fail2ban is installed
3. ✅ Install fail2ban if missing (with confirmation)
4. ✅ Create /var/log/caddy directory
5. ✅ Check if Caddy container has log volume mounted
6. ✅ Automatically add log volume to docker-compose.yml if needed
7. ✅ Create fail2ban filter at /etc/fail2ban/filter.d/caddy-auth.conf
8. ✅ Create fail2ban jail at /etc/fail2ban/jail.d/caddy.conf (with custom settings)
9. ✅ Test fail2ban configuration
10. ✅ Restart fail2ban and verify jail is active
11. ✅ Add service configurations (ActualBudget, Keycloak) to Caddyfile
12. ✅ Validate and reload Caddy configuration
ERROR HANDLING:
- All operations tracked with error messages array
- If any step fails, script continues but tracks the failure
- At the end, shows all errors encountered
- Provides exact manual commands to fix issues
- Backup is ALWAYS created before any changes
USER EXPERIENCE:
- Interactive prompts with sensible defaults
- Clear colored output (INFO, SUCCESS, WARNING, ERROR)
- Progress feedback at each step
- Final summary with useful commands
- Only shows manual instructions if automation failed
SAFETY FEATURES:
- Caddyfile backup before ANY modifications
- docker-compose.yml backup before modifications
- Validation before reloading Caddy
- Test fail2ban config before restart
- Restore instructions always shown after backup
This matches the integrated experience of other services - fully automated
unless something goes wrong, in which case it provides manual steps.
FIXES:
- Fix Magic Mirror npm install to run inside Docker container instead of on host
- npm (Node Package Manager) commands now execute inside the MagicMirror container
where Node.js is installed, preventing errors on hosts without Node.js
NEW SERVICES:
- Add ActualBudget: Open-source personal finance management with bank sync (SimpleFIN)
- Add Keycloak: Identity and Access Management (SSO, OAuth2, SAML, MFA)
- Both services integrated into main installation script and available as standalone
docker-compose files for existing servers
CADDY & FAIL2BAN:
- Add caddy-setup-helper.sh: Interactive script to configure Caddy and fail2ban
* Detects existing Caddy installation
* Automatically backs up Caddyfile with timestamp
* Checks for fail2ban support
* Provides service integration examples
- Add fail2ban filter and jail configurations for Caddy protection
- Add comprehensive setup guide (CADDY-FAIL2BAN-SETUP.md)
DOCUMENTATION:
- Detailed deployment instructions for each service
- Reverse proxy configuration examples
- Security best practices and headers
- Backup/restore procedures
- Troubleshooting guides
This update enables secure deployment of new services on existing servers with
proper Caddy reverse proxy integration and fail2ban protection against attacks.
- Add interactive "Start now?" prompts to all Docker containers
- Add UFW firewall port opening for Docker services when enabled
- Add Magic Mirror config copy option with custom.css support
- Add automatic detection and download of third-party MMM-* modules
- Add npm install for Magic Mirror module dependencies
Drive Setup (runs at script start):
- New setup_drives() function runs before other installations
- Auto-detects unpartitioned drives, offers to format
- Creates ~/drives/ mount points
- Adds to fstab and runs mount -a
- Partitioning/formatting for new drives without partition tables
Immich Improvements:
- Separate UPLOAD_LOCATION from EXTERNAL_LIBRARY (different paths)
- Upload: ~/drives/primary/photos/immich-uploads (new photos)
- External: ~/drives/primary/photos (existing photos, read-only)
- Warns if both paths are the same
- Added immich-cli instructions for uploading old photos with correct EXIF dates
- Container auto-start option after install
Container Management:
- Added "Start now?" prompt for Immich after install
Step 6 now scans docker-compose files for volume mounts:
- Detects absolute paths that don't exist on new system
- Shows old path and suggests ~/drives/primary/{folder}
- User can: accept suggestion, skip, or enter custom path
- Updates compose file with new path
- Creates directory if needed
Example:
Container: immich
Old path: /home/user1/media/driveb
Suggested: ~/drives/primary/driveb
[Enter] Accept | [S] Skip | [path] Custom
When source is on mounted drive (/mnt/*, ~/drives/*, /media/*):
- [C] Copy - Copy to ~/docker (for old OS drive migration)
- [S] Symlink - Create ~/docker → source (for data drive)
- [U] Use in-place - Use source directly, no copy
This handles both scenarios:
- Old OS drive mounted temporarily → Copy
- Data drive you'll keep using → Symlink or use in-place
- Auto-detect ~/drives/*/docker, /mnt/*/docker, /media/*/docker
- Show numbered list for easy selection (type "1" to select first)
- Still accepts any custom path
- Shows common locations as examples if nothing auto-detected
- New [M] Migration option at script start
- Auto-detects Docker directories (/var/docker, /opt/docker, ~/docker)
- Scans for docker-compose.yml files and lists containers with sizes
- Whiptail checklist for selecting containers to migrate
- Option to stop containers before copy (clean database state)
- Preserves versions - no unwanted upgrades during migration
- After migration, offers to install additional services
- Three modes now: Normal install, Migration, Disaster Recovery
- Install Kopia in Step 1 (core utilities)
- Add Step 9: Reconnect Kopia repository after restore
- Backups now work immediately after disaster recovery
- Update README with 9-step recovery process
- Immich: Ask for photo storage location (default ~/drives/primary/photos)
- Immich: External library support for existing photos (read-only)
- Immich: Storage template guidance for yyyy/mm organization
- Add Watchtower container with notify-only mode (safe for apps with DB migrations)
- Document what Docker data lives where and what gets backed up
- Update README with v2.9 changelog
- Add MeshCentral Server as Docker app (ports 4430, 4433)
- Recovery mode now installs core utilities first (openssh-server, git, etc.)
- Add whiptail checklist for selecting which services to restore
- Users can now choose some/none/all services instead of all-or-nothing
- Update README with v2.8 changelog and MeshCentral port
Claude