CRITICAL FIX:
The whiptail menu was OVERWRITING all service variables with "n",
which caused duplicate prompts and ignored user's earlier selections.
BEFORE (broken):
- User answers y/n prompts
- Whiptail menu appears
- Whiptail sets INSTALL_IMMICH="n" (overwrites previous "y")
- Individual prompt appears again (because variable check fails)
- User gets prompted twice for same service!
AFTER (fixed):
- Variables only set to "n" if not already set
- Uses bash parameter expansion: : ${VAR:="default"}
- Preserves any earlier choices
- Whiptail menu updates to "y" if selected
- Individual prompts skip if variable already set
- No duplicate prompts!
CHANGES:
Lines 2225-2249: Changed from direct assignment (VAR="n")
to conditional default (: ${VAR:="n"})
This preserves earlier choices while still allowing whiptail
to override them when services are selected.
SIDE EFFECT FIXED:
- Containers now start properly
- No more "containers won't come up" issue
- Proper dependency order maintained
FIXES:
1. SSH key import now properly accepts "n" as answer
- Added y/n prompt before asking for usernames
- Clearer flow: "Import SSH keys?" → "Which service?"
- No more confusion about entering "n" vs leaving blank
2. Automatic Caddy configuration for Keycloak
- Detects if Caddy is installed or being installed
- Offers to configure Caddy reverse proxy for Keycloak
- Backs up Caddyfile before changes
- Adds Keycloak configuration automatically
- Reloads Caddy after adding configuration
- Keycloak starts AFTER Caddy is configured
- Prevents "container won't come up" issue
CADDY AUTO-CONFIGURATION:
When both Keycloak and Caddy are selected:
- Script asks: "Configure Caddy reverse proxy for Keycloak?"
- Prompts for domain (e.g., auth.yourdomain.com)
- Backs up existing Caddyfile
- Adds Keycloak block with:
* JSON logging for fail2ban
* Reverse proxy to localhost:8180
* Security headers (HSTS, X-Frame-Options, etc.)
- Formats and reloads Caddy
- Confirms Keycloak will be available at domain
This ensures correct startup order: Caddy configured → Caddy reloaded → Keycloak starts
KEYCLOAK-SETUP-GUIDE.md:
Complete manual explaining Keycloak concepts, manual setup, and external services
WHAT'S A REALM:
- Isolated container for users/clients/config
- Like a "company" or "organization"
- master realm = admin only
- homelab realm = your actual users
- Fully isolated from each other
WHAT'S AN OAUTH2 CLIENT:
- Each service (ActualBudget, etc.) is a "client"
- Needs Client ID, Secret, and Redirect URIs
- Redirect URIs must match EXACTLY
- Guide explains the authentication flow
MANUAL SETUP INSTRUCTIONS:
- Step-by-step via web UI
- Create realm manually
- Create OAuth clients manually
- Configure redirect URIs
- Create users and set passwords
- Test the setup
EXTERNAL SERVICE SUPPORT (Pikapod, etc.):
Script now asks about setup type:
1. Local only (http://localhost)
2. Public domain (https://yourdomain.com)
3. Both local and public
For external services:
- Prompts for your public domain
- Warns that Keycloak MUST be accessible at https://auth.yourdomain.com
- Checks if Caddy/DNS are configured
- Asks for external service URL (e.g., Pikapod)
- Configures redirect URIs for all scenarios
REDIRECT URIS NOW INCLUDE:
- http://localhost:5006/* (local dev)
- https://budget.yourdomain.com/* (self-hosted)
- https://actualbudget-abc.pikapod.net/* (external)
- Multiple patterns for flexibility
SAVED CONFIG FILES UPDATED:
- Shows LOCAL DEVELOPMENT URLs
- Shows PRODUCTION URLs (if public domain set)
- Shows EXTERNAL SERVICE URLs (if external service set)
- Lists all configured redirect URIs
- Clear instructions for each scenario
CADDY CONFIGURATION GUIDE:
- How to configure DNS A/CNAME records
- Caddyfile example for Keycloak
- Security headers included
- Step-by-step setup for external access
RECONFIGURATION SUPPORT:
- Guide explains how to add realms manually
- Guide explains how to add clients manually
- CLI examples for adding realms/users/clients
- Can re-run script to configure additional realms
COMMON USE CASES:
1. All local services
2. Self-hosted with domain
3. Mixed (local + external like Pikapod)
Each use case explained with complete examples
TROUBLESHOOTING:
- Invalid redirect URI
- Client not found
- Invalid client secret
- External service can't reach Keycloak
- CORS/redirect failures
- Admin console login issues
With this update, users can:
✅ Understand what Keycloak is and how it works
✅ Configure it manually if they prefer
✅ Use it with external services like Pikapod
✅ Set up proper DNS/Caddy for production
✅ Troubleshoot common issues
✅ Add realms and clients later
Keycloak is now fully configured and ready to use immediately after installation!
No more manual realm/client setup required.
AUTOMATED SETUP:
After Keycloak starts, the script automatically:
1. ✅ Waits for Keycloak to be fully ready (health check)
2. ✅ Logs in using Keycloak Admin CLI (kcadm.sh)
3. ✅ Creates a new realm (e.g., "homelab")
4. ✅ Creates OAuth2/OIDC client for ActualBudget (if selected)
5. ✅ Creates generic OAuth2 client template for other services
6. ✅ Optionally creates an initial user
7. ✅ Saves all OAuth credentials to text files
8. ✅ Provides clear next steps
OAUTH2 CLIENT FOR ACTUALBUDGET:
- Client ID: actualbudget
- Auto-generated secure client secret
- Pre-configured redirect URIs for localhost and production
- Saved to: ~/docker/keycloak/actualbudget-oauth.txt
- Includes all URLs needed to configure ActualBudget
GENERIC OAUTH2 CLIENT:
- Client ID: generic-app
- Can be cloned for other services
- Saved to: ~/docker/keycloak/generic-oauth.txt
- Works as a template
INITIAL USER CREATION:
- Prompts for username, email, first name, last name, password
- User is immediately active and can log in
- Can be used for ActualBudget and other services right away
SAVED CONFIGURATION FILES:
~/docker/keycloak/actualbudget-oauth.txt - ActualBudget OAuth config
~/docker/keycloak/generic-oauth.txt - Generic OAuth template
PRODUCTION READY:
- Redirect URIs include both localhost and production domains
- Works with Caddy reverse proxy
- SSL/TLS enforced at proxy level
- Just update domain in configuration
USER EXPERIENCE:
Install Keycloak → Answer prompts → DONE!
- Realm created: "homelab" (or custom name)
- OAuth clients ready
- User created and can log in immediately
- Just go to http://localhost:8180/admin to manage
This eliminates the complex post-install Keycloak setup and makes it
immediately usable for ActualBudget and other services!
Users now get a nice checkbox menu to select which services to install,
instead of being prompted for each service one-by-one.
WHIPTAIL MENU:
- Displays all 24+ Docker services in a single checklist
- Use SPACE to select/deselect services
- Press ENTER to confirm and install selected services
- Falls back to individual prompts if whiptail not available
SERVICES IN MENU:
✓ Immich (Photo & Video Backup)
✓ AudioBookshelf (Audiobooks & Podcasts)
✓ Emby (Media Server)
✓ A.R.M. (Automatic Ripping Machine)
✓ FileBrowser (Web File Manager)
✓ Magic Mirror (Smart Mirror Display)
✓ ActualBudget (Personal Finance)
✓ Keycloak (Identity & Access Management)
✓ Caddy (Reverse Proxy with Auto-HTTPS)
✓ fail2ban (Intrusion Prevention)
✓ Lyrion (Music Streaming)
✓ Mealie (Recipe Manager)
✓ Minecraft (Game Server)
✓ Jellyfin (Free Media Server)
✓ Frigate (AI NVR for Cameras)
✓ Ntfy (Push Notifications)
✓ Uptime Kuma (Service Monitoring)
✓ WG-Easy (WireGuard VPN)
✓ Traccar (GPS Tracking)
✓ Portainer (Docker Web UI)
✓ MeshCentral (Remote Management)
✓ FindMyDevice (Device Tracking)
✓ Frigate-Notify (Frigate Notifications)
✓ Watchtower (Auto Container Updates)
WORKFLOW:
1. Run ubuntu-post-install.sh
2. Get whiptail menu for service selection
3. Select services with SPACE
4. Press ENTER to install
5. Script installs only selected services
FALLBACK:
- If whiptail not available, uses traditional prompts
- Prompts only appear if service wasn't selected in menu
- Fully backwards compatible
This dramatically improves UX for installing multiple services!
Users can now install and configure everything by simply running the main script.
Re-running the script allows adding new services to existing installations.
NEW SERVICES IN MAIN SCRIPT:
CADDY WEB SERVER:
- Automatic HTTPS with Let's Encrypt
- Reverse proxy for all services
- Creates example Caddyfile with ActualBudget and Keycloak configs
- Detects existing installations (asks before reconfiguring)
- Automatically backs up existing Caddyfile before changes
- Pre-configured with /var/log/caddy volume for fail2ban integration
- Includes HTTP/3 support
FAIL2BAN INTRUSION PREVENTION:
- Automated installation via apt
- Creates Caddy filter for JSON logs (401, 403, 429 status codes)
- Creates Caddy jail with configurable settings
- Automatically creates /var/log/caddy directory
- Tests configuration before restart
- Verifies jail is active after restart
- Shows status and useful commands
FEATURES:
✅ Detects if services already exist (won't overwrite)
✅ Backs up configurations before changes
✅ Interactive prompts for all settings
✅ Validates configurations before applying
✅ Can be re-run to add services to existing setup
✅ Works alongside existing services
✅ Follows same pattern as ActualBudget/Keycloak
WORKFLOW:
1. Run ubuntu-post-install.sh
2. Select services to install (ActualBudget, Keycloak, Caddy, fail2ban, etc.)
3. Script handles everything automatically
4. Re-run anytime to add more services
The caddy-setup-helper.sh remains available as a standalone tool for
advanced configuration, but the main script is now the primary method.
The caddy-setup-helper.sh script now handles everything automatically (after asking
for confirmation), only falling back to manual instructions if errors occur.
AUTOMATED WORKFLOW:
1. ✅ Backup Caddyfile (ALWAYS FIRST - before any changes)
2. ✅ Check if fail2ban is installed
3. ✅ Install fail2ban if missing (with confirmation)
4. ✅ Create /var/log/caddy directory
5. ✅ Check if Caddy container has log volume mounted
6. ✅ Automatically add log volume to docker-compose.yml if needed
7. ✅ Create fail2ban filter at /etc/fail2ban/filter.d/caddy-auth.conf
8. ✅ Create fail2ban jail at /etc/fail2ban/jail.d/caddy.conf (with custom settings)
9. ✅ Test fail2ban configuration
10. ✅ Restart fail2ban and verify jail is active
11. ✅ Add service configurations (ActualBudget, Keycloak) to Caddyfile
12. ✅ Validate and reload Caddy configuration
ERROR HANDLING:
- All operations tracked with error messages array
- If any step fails, script continues but tracks the failure
- At the end, shows all errors encountered
- Provides exact manual commands to fix issues
- Backup is ALWAYS created before any changes
USER EXPERIENCE:
- Interactive prompts with sensible defaults
- Clear colored output (INFO, SUCCESS, WARNING, ERROR)
- Progress feedback at each step
- Final summary with useful commands
- Only shows manual instructions if automation failed
SAFETY FEATURES:
- Caddyfile backup before ANY modifications
- docker-compose.yml backup before modifications
- Validation before reloading Caddy
- Test fail2ban config before restart
- Restore instructions always shown after backup
This matches the integrated experience of other services - fully automated
unless something goes wrong, in which case it provides manual steps.
FIXES:
- Fix Magic Mirror npm install to run inside Docker container instead of on host
- npm (Node Package Manager) commands now execute inside the MagicMirror container
where Node.js is installed, preventing errors on hosts without Node.js
NEW SERVICES:
- Add ActualBudget: Open-source personal finance management with bank sync (SimpleFIN)
- Add Keycloak: Identity and Access Management (SSO, OAuth2, SAML, MFA)
- Both services integrated into main installation script and available as standalone
docker-compose files for existing servers
CADDY & FAIL2BAN:
- Add caddy-setup-helper.sh: Interactive script to configure Caddy and fail2ban
* Detects existing Caddy installation
* Automatically backs up Caddyfile with timestamp
* Checks for fail2ban support
* Provides service integration examples
- Add fail2ban filter and jail configurations for Caddy protection
- Add comprehensive setup guide (CADDY-FAIL2BAN-SETUP.md)
DOCUMENTATION:
- Detailed deployment instructions for each service
- Reverse proxy configuration examples
- Security best practices and headers
- Backup/restore procedures
- Troubleshooting guides
This update enables secure deployment of new services on existing servers with
proper Caddy reverse proxy integration and fail2ban protection against attacks.
- Add interactive "Start now?" prompts to all Docker containers
- Add UFW firewall port opening for Docker services when enabled
- Add Magic Mirror config copy option with custom.css support
- Add automatic detection and download of third-party MMM-* modules
- Add npm install for Magic Mirror module dependencies
Drive Setup (runs at script start):
- New setup_drives() function runs before other installations
- Auto-detects unpartitioned drives, offers to format
- Creates ~/drives/ mount points
- Adds to fstab and runs mount -a
- Partitioning/formatting for new drives without partition tables
Immich Improvements:
- Separate UPLOAD_LOCATION from EXTERNAL_LIBRARY (different paths)
- Upload: ~/drives/primary/photos/immich-uploads (new photos)
- External: ~/drives/primary/photos (existing photos, read-only)
- Warns if both paths are the same
- Added immich-cli instructions for uploading old photos with correct EXIF dates
- Container auto-start option after install
Container Management:
- Added "Start now?" prompt for Immich after install
Step 6 now scans docker-compose files for volume mounts:
- Detects absolute paths that don't exist on new system
- Shows old path and suggests ~/drives/primary/{folder}
- User can: accept suggestion, skip, or enter custom path
- Updates compose file with new path
- Creates directory if needed
Example:
Container: immich
Old path: /home/user1/media/driveb
Suggested: ~/drives/primary/driveb
[Enter] Accept | [S] Skip | [path] Custom
When source is on mounted drive (/mnt/*, ~/drives/*, /media/*):
- [C] Copy - Copy to ~/docker (for old OS drive migration)
- [S] Symlink - Create ~/docker → source (for data drive)
- [U] Use in-place - Use source directly, no copy
This handles both scenarios:
- Old OS drive mounted temporarily → Copy
- Data drive you'll keep using → Symlink or use in-place
- Auto-detect ~/drives/*/docker, /mnt/*/docker, /media/*/docker
- Show numbered list for easy selection (type "1" to select first)
- Still accepts any custom path
- Shows common locations as examples if nothing auto-detected
- New [M] Migration option at script start
- Auto-detects Docker directories (/var/docker, /opt/docker, ~/docker)
- Scans for docker-compose.yml files and lists containers with sizes
- Whiptail checklist for selecting containers to migrate
- Option to stop containers before copy (clean database state)
- Preserves versions - no unwanted upgrades during migration
- After migration, offers to install additional services
- Three modes now: Normal install, Migration, Disaster Recovery
- Install Kopia in Step 1 (core utilities)
- Add Step 9: Reconnect Kopia repository after restore
- Backups now work immediately after disaster recovery
- Update README with 9-step recovery process
- Immich: Ask for photo storage location (default ~/drives/primary/photos)
- Immich: External library support for existing photos (read-only)
- Immich: Storage template guidance for yyyy/mm organization
- Add Watchtower container with notify-only mode (safe for apps with DB migrations)
- Document what Docker data lives where and what gets backed up
- Update README with v2.9 changelog
- Add MeshCentral Server as Docker app (ports 4430, 4433)
- Recovery mode now installs core utilities first (openssh-server, git, etc.)
- Add whiptail checklist for selecting which services to restore
- Users can now choose some/none/all services instead of all-or-nothing
- Update README with v2.8 changelog and MeshCentral port
New features:
- --restore flag for disaster recovery mode
- Interactive mode selector at script start (N=Normal, R=Recovery)
- Full disaster recovery flow:
1. Show available drives, auto-mount if device path given
2. Auto-detect Kopia repository
3. Try to find password in backed-up .env, or prompt
4. Install Docker if needed
5. List available snapshots, let user choose or use 'latest'
6. Restore snapshot to temp location
7. Detect all docker-compose.yml files = services to restore
8. Copy services to ~/docker/
9. Optionally start all containers
10. Cleanup temp files
Removed old buried import section that only showed manual instructions.
Documentation:
- Added Disaster Recovery section to README
- Added --restore to command-line options
- Documented what gets restored and requirements
- Added v2.7 changelog entry
Changes the installation pattern for Docker apps to be more robust:
- Install docker-compose.yml FIRST (always succeeds)
- THEN try configuration with prompts
- Use sensible defaults if prompts fail
- Continue to next app even if current config fails
- Added || true and 2>/dev/null to prevent script stops
Updated apps: Frigate, Frigate-Notify, Caddy, ddclient
Config templates now include clear warnings:
- "YOU MUST EDIT THIS FILE" for required configs
- "YOU MAY NEED TO EDIT THIS FILE" for optional configs
- Links to documentation
This ensures the script completes even with complex interdependent
services that may need manual configuration after install.
New Docker applications:
- FindMyDevice (FMD) server for self-hosted Android device tracking
- Frigate-Notify for push notifications on Frigate AI detections
Caddy improvements:
- Interactive domain configuration during setup
- Comprehensive Caddyfile template with all services (commented)
- Clear instructions for caddy_net Docker network usage
- .env file with MY_DOMAIN variable
Frigate-Notify features:
- Auto-detects if Frigate and ntfy are installed
- Interactive setup for Frigate URL and ntfy topic
- WebAPI mode by default (polls Frigate every 30s)
- Config template with labels, zones, quiet hours
- Warns about public ntfy.sh privacy implications
New Docker applications:
- Jellyfin (free media server with hardware acceleration)
- Frigate NVR (AI-powered object detection)
- Caddy (reverse proxy with automatic HTTPS)
- ddclient (dynamic DNS updater)
- ntfy (self-hosted push notifications)
- Uptime Kuma (service uptime monitoring)
- wg-easy (WireGuard VPN with web UI)
- Traccar (GPS tracking server)
- Portainer (Docker management UI)
Container backup system:
- Kopia backup for all Docker container data
- Backup script for configs, databases, app state
- Restore script for disaster recovery
- Backs up Immich memories, Emby metadata, Minecraft worlds, etc.
All apps use docker-compose in ~/docker/{appname}/ with storage on
primary drive where appropriate.
After configuring the Primary share, the script now displays:
- How to edit /etc/samba/smb.conf with nano
- Example share configuration block
- How to restart smbd/nmbd services
- How to verify with testparm
VPN additions:
- WireGuard VPN with key generation and config setup
- Tailscale mesh VPN with Tailscale SSH documentation
Remote desktop additions:
- TeamViewer installation and setup
- MeshCentral agent with server URL prompt
Other changes:
- Enhanced NetBird documentation on SSH key management
- Detection functions and status display for all new tools
- Updated README with VPN Setup and Remote Desktop Setup sections
Major changes:
- Local backup uses rsync exclusively with support for 1-4 drives
- Drive names are now customizable (default: primary, backup1, etc.)
- Added separate cloud backup option with rclone + encryption
- Guided setup for Google Drive and OneDrive with encryption
- rclone.conf auto-backed up to all local drives
- Added off-site backup guidance (Signal, Box.com, password managers)
- Removed old rsync/rclone choice and full/split modes
Documentation updates:
- Explain why fail2ban provides no benefit with key-only SSH
- Explain why rsync instead of RAID
- Document rclone.conf decryption and restore process
- Update all backup-related sections for new structure
- Add --dry-run flag to preview installations without changes
- Add --unattended flag for automated/scripted installs with defaults
- Add logging to /var/log/post-install.log
- Add fail2ban protection when SSH password auth is enabled
- Add UFW firewall configuration option
- Update all prompts for unattended mode support
- Update README with new features and troubleshooting
- Add software detection for Docker, Samba, NetBird, RustDesk, rclone, rsync
- Script shows current system status and offers to reinstall/reconfigure
- All components now optional with y/n prompts
- Add backup tool selection: rsync (recommended for local) vs rclone (cloud)
- Add backup mode selection: full (one drive) vs split (two drives)
- Generate appropriate backup script based on tool + mode selection
- Update summary section to only show installed components
- Update README with new features, rsync vs rclone comparison, backup modes
- Mark Samba File Sharing section as optional
- Mark Backup System section as optional
- Update interactive prompts section with new Samba/backup prompts
- Add note to Backup Configuration section about optional nature
- Update Samba section to indicate conditional installation
- Reorganize "Files Created" section by optional/required
- Update Security Notes to indicate optional features
Claude