install-local-gpu.sh and bring-up-local-gpu.sh now call
prefetch-models.sh --sdxl (host-side download, all models including
SDXL/inpaint, ~13GB) automatically rather than treating it as a manual
DNS-troubleshooting step. install-local-gpu.sh drops back to the
invoking user via sudo -u "$SUDO_USER" -H before running it, since
that script itself runs as root and would otherwise leave downloaded
files root-owned. Both call sites are non-fatal: a prefetch failure
logs a warning and the container still starts, falling back to its
own in-container download attempt.
Updated README to match.
- Serve /dist and /src with Cache-Control: no-cache so browsers always
revalidate the webpack bundle (it has a fixed filename with no content
hash, so a stale browser cache was hiding the new Remove Background
model dropdown after the rebuild).
- Enable BEN2's refine_foreground pass by default - it's the model's own
documented option for preserving fine/semi-transparent edge detail
(text borders, light rays) instead of a harder cutout, at a small
speed cost.
- Make install-local-gpu.sh and bring-up-local-gpu.sh pre-create ./data
as the invoking non-root user before Docker ever runs, so its daemon
never gets a chance to auto-create those bind-mount dirs as root.
- Make prefetch-models.sh self-heal a root-owned ./data automatically
(sudo chown) instead of erroring out with a manual fix-it command.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Ro4PwQKvSc3CH19LSN21Ht
The mkdir -p preflight check only verified that data/models and
data/hf_cache could be created, which succeeds vacuously on a directory
that already exists — exactly the case when Docker created ./data as
root on a prior run. Confirmed in practice: U2Net's download reached the
network fine but failed writing the file with Permission denied. Now
test actual writability and point at the one-time fix (chown ./data back
to the calling user) instead of suggesting sudo re-runs that would leave
the ownership problem in place permanently.
Also handle pip's PEP 668 "externally-managed-environment" error
(Debian/Ubuntu 12+) when installing huggingface_hub, retrying with
--break-system-packages — safe here since --user already keeps the
install out of apt-managed system site-packages.
download_sam_model.py crashed with an uncaught PermissionError when
data/models/ is root-owned (common after a prior Docker run) and a
non-root host user tries to recreate the convenience sam_model.pth
symlink — observed in the wild, and it aborted the whole prefetch run
before U2Net/BEN2/BiRefNet-HR were ever attempted. Worse, the same
unguarded symlink call sat inside the post-download try/except, so a
successful download could get deleted just because the symlink step
failed afterward. Wrapped symlink creation in a shared helper that
warns and continues instead of raising — the real model file already
satisfies entrypoint.sh's checks regardless of the symlink.
prefetch-models.sh now treats SAM, U2Net, and the HuggingFace models as
independent steps (one failing no longer aborts the rest) and prints a
summary of which steps failed, so a single run gives full diagnostic
signal instead of stopping at the first error.
Lets SAM, U2Net, BEN2, and BiRefNet-HR (optionally SDXL via --sdxl) be
downloaded outside Docker into ./data/, which is already bind-mounted
into the GPU container — so a blocked container network no longer blocks
first-run setup. Reuses the existing dual-mode download_sam_model.py and
download_u2net_model.py as-is. For the HuggingFace Hub models, sets
HF_HOME (rather than --cache-dir) so the host-side cache layout matches
the container's default ~/.cache/huggingface resolution exactly, avoiding
a path-nesting mismatch between the two.
Wired into install-local-gpu.sh's completion banner and
bring-up-local-gpu.sh's header, and referenced from the relevant README
troubleshooting sections and the hf_cache bind-mount comment in
docker-compose.gpu.yml.
BEN2 becomes the new default local backend (clean cutouts, strong on
hair/fur edges), with BiRefNet-HR available as a high-res/print
alternate and U2Net kept as the lightweight fallback. Both are
MIT-licensed and download weights from HuggingFace on first use
(cached via the existing hf_cache bind mount), unlike U2Net/SAM which
need an explicit download script.
- config: new BG_REMOVAL_MODEL setting (default "ben2")
- tools.py: remove-background-base64 now tries local backends in
order (request.model override > BG_REMOVAL_MODEL > ben2/u2net),
falling back to rembg's birefnet-general session as a last resort
- requirements.gpu.txt / Dockerfile.gpu: add ben2 + transformers deps
needed for the new backends, with a build-time smoke test for ben2
- frontend: model dropdown in the Remove Background dialog, threaded
through api.js to the new request field
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Ro4PwQKvSc3CH19LSN21Ht
Commit 5c85987 appended a second copy of the toolkit-install/DNS-fix/
GPU-verify steps instead of inserting the new prerequisite-checks
section once. Removed the duplicate, keeping the more informative
GPU-check failure message from the second copy.
U2Net only ever downloaded lazily on the first Remove Background click,
unlike SAM which retries on every container start. If that one attempt
failed (DNS/firewall) the model was never fetched again, surfacing as
"No background removal method available. Install u2net or rembg."
Mirrors the existing SAM auto-download/AUTO_DOWNLOAD_SAM pattern for
U2Net, and documents manual host-side recovery in the README.
Also deletes backend/app/services/u2net_model.py (hand-written
U2NET/U2NETP PyTorch classes) — unused since tools.py switched to
cv2.dnn.readNetFromONNX for background removal.
- Backend: POST /api/image/replace-subject
Uses rembg to extract the subject from a source photo, scales it to fit
the selection mask bounding box (or canvas centre when no selection is
active), applies a partial LAB color transfer (blend=0.45) so the
subject's lighting matches the background, then composites the result.
Also adds POST /api/image/extract-subject for standalone subject extraction.
- Frontend: "Replace with subject from file" button in the selection panel
(selection_actions.js) — opens a native file picker so no clipboard API
or HTTPS is required. Calls the new endpoint with the current SAM mask.
- Frontend: Image > Replace Subject (AI)... menu entry backed by
modules/image/replace_subject.js — a full dialog with file picker,
thumbnail preview, and "match background lighting" toggle. Works with
or without a prior Smart Select; if a selection exists it confines the
subject to that region.
https://claude.ai/code/session_01UtrvbisMp1yGu6PqeLmrFr
Checks Docker installed + daemon running, NVIDIA driver (nvidia-smi),
and curl. Prints specific install commands for each missing dependency
and exits early with a clear summary rather than failing partway through.
https://claude.ai/code/session_01WVDg7amsy1TTtxvpku7bcM
The iptables DOCKER-USER rule is lost on reboot; the script re-applies
it each run, checks for duplicates, and is silently skipped on macOS/WSL.
Default behaviour (no args): docker compose up -d --build.
All docker compose subcommands can be passed as args (logs, down, etc.).
README Quick Start and Updates sections now reference ./start-gpu.sh.
https://claude.ai/code/session_01WVDg7amsy1TTtxvpku7bcM
iptables fix is simpler (one command, no 13 GB download) and does not
affect container isolation — adds clarifying note so users understand
it only restores Docker's default outbound DNS behaviour.
Docker helper container kept as fallback option.
https://claude.ai/code/session_01WVDg7amsy1TTtxvpku7bcM
Replaces 'pip install huggingface-hub' (breaks on PEP 668 / Debian 12+)
with a docker run --rm python:3.11-slim one-liner that downloads directly
into ./data/hf_cache without touching host Python packages.
https://claude.ai/code/session_01WVDg7amsy1TTtxvpku7bcM
Build / pip layer fixes:
- Add BUILDID ARG to Dockerfile.gpu; pass from docker-compose.gpu.yml build args
so pip layers can be force-busted without --no-cache:
BUILDID=$(date +%s) docker compose -f docker-compose.gpu.yml up --build
Model download (DNS-blocked environments):
- Change HF model cache from named volume to ./data/hf_cache bind mount
so models can be pre-downloaded on the host (no rebuild needed)
- Remove now-unused hf_model_cache named volume
- README: add iptables fix + huggingface-cli offline download instructions
Error handling improvements:
- ai_edit_region: catch ConnectError/Errno-3 → return 503 with exact fix commands
- _require_remote: give actionable message when local_gpu provider fails to load
- _build_provider: catch AttributeError (torch.xpu from wrong diffusers) not just ImportError
- local_diffusion.py: fix docstring to reflect <0.29.0 pin
https://claude.ai/code/session_01WVDg7amsy1TTtxvpku7bcM
- Pin diffusers to >=0.28.0,<0.29.0 to avoid AttributeError on torch.xpu
(diffusers 0.29+ requires PyTorch 2.4 but base image ships 2.1.2)
- Pin transformers to <4.40.0 to match
- README: add SAM offline download troubleshooting for DNS-blocked containers
- SelectionActions panel: 'Selection ready' title + subtitle makes clear
nothing has fired yet; AI actions show inline description (not just tooltip);
section labels 'AI Actions' / 'Classic Tools'; scale hint updates live;
Enter key submits AI edit prompt; _actionCard hover border for clickability
https://claude.ai/code/session_01WVDg7amsy1TTtxvpku7bcM
1.1.1.1 uses a different anycast route than 8.8.8.8 so may reach the
container even when Google's servers don't. Falls back to Google if
Cloudflare is also unreachable. If all fail (Errno -3), port 53 UDP is
blocked at the Docker bridge level — comment has the iptables fix.
https://claude.ai/code/session_01WVDg7amsy1TTtxvpku7bcM
ai_tools.py: add 'from io import BytesIO' — scale_selection and paste
endpoints used BytesIO directly but it was only imported locally in one
unrelated function, causing NameError on every scale call.
provider-badge.js: redesign badge to fit the narrow (~40px) left toolbar.
Was rendering 'GPU · sdxl_offload · Quadro RTX 3000' inline which wrapped
into multiple lines covering tool icons. Now shows a status dot + short
label (SDXL / OAI / Rep…) with all details moved to the hover tooltip.
https://claude.ai/code/session_01WVDg7amsy1TTtxvpku7bcM
selection_actions.js uses named exports only (no default export).
load_plugins() was calling new classObj.default(ctx) on every file in tools/,
causing TypeError which aborted render_main_tools() -> render_main_gui() ->
leaving menu and toolbar blank. Added null guard and per-file try/catch,
matching the defensive pattern already added to load_modules().
https://claude.ai/code/session_01WVDg7amsy1TTtxvpku7bcM
Without this, any constructor throw in any module stops load_modules() mid-loop
and render_main_gui() (which builds the menu) never runs — resulting in a blank
page with no menu. The failing module is now logged to the browser console
instead of silently aborting startup.
https://claude.ai/code/session_01WVDg7amsy1TTtxvpku7bcM
main.py referenced info.vram_gb but the field is info.vram_total_gb.
This crashed the FastAPI lifespan hook on every startup when
AI_PROVIDER=local_gpu, causing a restart loop.
https://claude.ai/code/session_01WVDg7amsy1TTtxvpku7bcM
Old README described the original React/Fabric.js UI and listed
"No local GPU inference" as a non-goal. Updated to reflect:
- miniPaint-based editor with SAM brush selection
- GPU quick-start (nvidia-container-toolkit prereqs, docker-compose.gpu.yml)
- Cloud API quick-start
- GPU tier auto-selection table (FLUX/SDXL/SD by VRAM)
- Full feature list (selection actions, print tools, progress bars)
- Correct clone URL and update commands
- Troubleshooting section
https://claude.ai/code/session_01WVDg7amsy1TTtxvpku7bcM
Backend:
- local_diffusion.py: add _make_step_cb() that writes step/total_steps/
progress into _states on every diffusers callback_on_step_end; wired into
txt2img, inpaint, img2img with TypeError fallback for older diffusers
- ai_tools.py: GET /api/generate/progress SSE endpoint — streams _states
as JSON array every 200ms so clients get live denoising step counts
Frontend:
- progress_overlay.js: add connectProgressSSE(pipeType, baseUrl) /
disconnectProgressSSE() — opens EventSource, maps step/total_steps
to bar percentage (0→85% during denoising, 85→100 for decode/place)
- text_to_image.js: connect SSE before POST, disconnect on done/error
- selection_actions.js: connect SSE for AI edit / asymmetry operations
Result: for local GPU, progress bar shows "Step 12 / 30" with exact fill;
for remote providers and upscale (no step callbacks), shimmer animates.
https://claude.ai/code/session_01WVDg7amsy1TTtxvpku7bcM
- New progress_overlay.js: animated fullscreen overlay with shimmer bar,
fake progress creep, Esc-to-cancel, used by all slow AI operations
- text_to_image.js: allow local_gpu provider (was incorrectly blocked);
show provider/model/VRAM info in dialog; show estimated generation time;
use progress overlay during generation
- upscale.js: replace alertify.message with progress overlay (90s estimate
for AI upscale, 10s for Lanczos)
- frame_fit.js: progress overlay for extend mode (AI outpaint ~45s)
- print_prepare.js: progress overlay for full upscale+frame chain (~2 min)
- selection_actions.js: progress overlay for all AI region edits
https://claude.ai/code/session_01WVDg7amsy1TTtxvpku7bcM
- Add 18x24" to FRAME_SIZES in backend and frontend (frame_fit.js)
- Add 200 DPI option to frame_fit dialog (adequate for large-format prints)
- Add 18x24 portrait/landscape at 200 and 300 DPI to Canvas Size presets (size.js)
- New /api/print/prepare endpoint: chains AI upscale to target DPI then frame-fit
in one server-side call (avoids round-tripping a large upscaled image)
- New print_prepare.js module: "Prepare for Print" dialog with per-frame quality
assessment (current effective DPI, needed upscale factor, AI vs Lanczos note)
- Add "Prepare for Print..." to Image menu above "Fit to Frame..."
https://claude.ai/code/session_01WVDg7amsy1TTtxvpku7bcM
Backend (3 new endpoints under /api/image/):
- POST /api/image/scale-selection — scale selected object by any % in-place;
LaMa/OpenCV fills the exposed gap so the scene looks natural
- POST /api/image/ai-edit-region — AI redraws the masked region via the
configured inpaint provider (local_gpu / InvokeAI / ComfyUI / OpenAI)
- POST /api/image/paste-into-selection — scales clipboard image to fit the
selection bounding box, masks it to the selection shape, composites result
Frontend (selection_actions.js + tool integration):
- New SelectionActions panel: fixed bottom-center HUD that appears
automatically after every SAM selection (click or paint)
- Panel actions: Scale by % (default 3%), Make less symmetrical (AI),
custom AI Edit prompt, Replace with clipboard, Copy/Cut to layer, Erase
- Both smart_select.js and brush_select.js updated to show the panel,
add updateLayerWithResult(), and hide panel on clearSelection/on_leave
- brush_select: offerFloatSelection() replaced with richer action panel
Real-world workflows now supported in one click after painting over object:
"Make this 3% bigger" → scale-selection (LaMa fills gap)
"Make this less symmetrical" → ai-edit-region with asymmetry prompt
"Replace this with what I copied" → paste-into-selection
https://claude.ai/code/session_01WVDg7amsy1TTtxvpku7bcM
Model selection:
- Add sdxl_offload tier (eff_vram ≥ 4.0 GB) for GTX 1060 6GB and Quadro
6GB cards that were falling through to SD 2.1 despite SDXL fitting with
model_cpu_offload. Cards with 5.3 GB effective VRAM now get SDXL quality.
- Update _tier_label(), _caps(), _build_warnings() for new tier.
Frontend GPU display:
- api.js: add getGpuStatus() fetching /api/gpu/status
- capabilities.js: add getGpuStatus() export with own LRU cache;
refreshCapabilities() now also resets GPU status cache
- provider-badge.js: when AI_PROVIDER=local_gpu show green badge with
GPU name, tier, VRAM, CC, feature flags, and capabilities in tooltip.
Strip "NVIDIA GeForce" prefix so "GTX 1060 6GB" fits in badge.
- ai_provider_settings.js: add local_gpu to all provider dropdowns;
show GPU info panel (device, VRAM, CC, features, tier, model table per
operation) in the settings dialog when a GPU is detected.
https://claude.ai/code/session_01WVDg7amsy1TTtxvpku7bcM
The fitted image was processed and inserted correctly but the canvas
(config.WIDTH/HEIGHT) was never updated, so the result was clipped to
the original canvas size. Now wraps both new-layer and replace-layer
paths in Prepare_canvas_action + Update_config_action so the canvas
expands (or shrinks) to the target frame size automatically.
https://claude.ai/code/session_017wupXfpjuoSdAqVyak4fJf
Resize dialog:
- Units selector (pixels / inches) at the top; switching updates the
Width/Height placeholders and clears any partially entered values
- New "Crop to fill" checkbox: when both Width and Height are given,
scales the image with cover-fit (fills target without letterboxing)
then center-crops — subject looks the same at 5x7, 8x10, 11x14
- resize_layer/resize_gui now honour params.units instead of only
reading the global default_units setting
Global units persistence:
- Switching units in either Resize or Canvas Size saves the choice to
default_units so both dialogs open with the same unit next time
https://claude.ai/code/session_017wupXfpjuoSdAqVyak4fJf
- Units selector (pixels/inches) directly in the dialog; switching live-converts
the width/height fields so you can type e.g. 5 / 7 without mental math
- Six print-size presets at 300 DPI added to the Resolution dropdown:
5x7, 8x10, and 11x14 in both Portrait and Landscape orientations
- New "Resize & crop image" checkbox: when enabled, image layers are scaled
with cover-fit (fills the target canvas, no letterboxing) and center-cropped
so the subject stays proportionally the same across all three print sizes
https://claude.ai/code/session_017wupXfpjuoSdAqVyak4fJf
ai_edit.js:
- Extend Base_tools_class and add load() + default_events() so mouse
events actually wire up (was the root cause of tool not working)
- Use get_mouse_info() for coordinate mapping instead of manual
clientX/Y math — consistent with all other miniPaint tools
- Fix _mouseToImage() to use mouse.x/y (already in image coords)
and correct display-scale for overlay brush rendering
- mousedown/mousemove/mouseup now guard on config.TOOL.name
ai_edit.svg: new icon (brush + sparkle star) for the left toolbar
layout.css: add .ai_edit:after CSS rule for the icon
main.js:
- Collapse right-panel Colors section by default (respects saved cookie
so user preference persists)
- Mount compact foreground/background color swatches at the bottom of
the left toolbar; click either square to toggle the full color picker
open/closed; syncs live with config.COLOR every 250ms
https://claude.ai/code/session_01B58MaJCU1R6KwBDJCp8AfN
sklearn was not installed in the container, causing ModuleNotFoundError on
import of ai_tools.py and preventing the server from starting.
Replaced with a self-contained numpy k-means++ implementation:
- k-means++ seeding for better initial centers
- 20-iteration Lloyd's algorithm
- Same output: hex colors sorted by cluster frequency
No new dependencies required.
https://claude.ai/code/session_01B58MaJCU1R6KwBDJCp8AfN
pick_color.js: on hover shows a floating tooltip with:
- 40px color swatch of the sampled pixel
- Hex (#RRGGBB), RGB, and HSL values
- Nearest Pantone name and swatch from the database
- ΔE value (CIE76 color distance in LAB space)
- Quality badge: Excellent (<2) / Good (2-5) / Fair (5-10) / Poor (>10)
- Warning note when ΔE > 10 (color cannot be faithfully printed as Pantone)
Click copies hex to clipboard and sets as active color (existing behavior).
color_utils.js: hexToRgb, rgbToHsl, sRGB→XYZ→CIE LAB conversion (D65),
CIE76 deltaE, nearestPantone() (scans all ~350 entries, returns best match).
Pantone database is pre-converted to LAB at module load for fast scanning.
pantone_colors.js: ~350 representative PMS colors across reds, oranges,
yellows, greens, teals, blues, purples, magentas, browns, grays, black/white.
Hex approximations from open-source Pantone datasets (not official Pantone data).
https://claude.ai/code/session_01B58MaJCU1R6KwBDJCp8AfN
Auto-Enhance (Image menu):
POST /api/enhance — gray-world white balance, CLAHE contrast on L channel,
saturation boost ×1.15 in HSV, unsharp mask; all blended by strength slider
Frontend: strength selector (25/50/75/100%), keep-original option
Extract Color Palette (Image menu):
POST /api/extract-colors — k-means on 150×150 thumbnail, returns N dominant
colors sorted by cluster size. Frontend: floating swatch panel, click=copy
hex, shift+click=set as active color, toggle on/off.
History Panel (Edit menu, Ctrl+H):
Pure frontend — reads app.State.action_history and action_history_index,
renders clickable list of past actions (newest first), click any step to
undo/redo to that point. Auto-refreshes every 800ms while open.
Align to Canvas (Layer menu):
Floating toolbar with 7 alignment buttons: center H, center V, center both,
align left/right/top/bottom edges. Uses Update_layer_action for undo support.
Add Text (Generate menu):
6 styled presets (Heading, Subheading, Body, Caption, Quote, Bold Label)
shown as live-rendered previews in the dialog. Click a preset to insert a
text layer with the correct font/size/weight/color pre-applied.
https://claude.ai/code/session_01B58MaJCU1R6KwBDJCp8AfN
- tiff-writer.js: pure-JS multipage TIFF encoder with RGBA and CMYK
color modes; RGB→CMYK conversion; one IFD per layer for multilayer
- pdf-writer.js: pure-JS PDF-1.4 writer; RGB pages use browser JPEG
encoding (DCTDecode); CMYK pages embed raw DeviceCMYK pixel data;
multipage support (one page per visible layer)
- save.js: adds TIFF (CMYK), TIFF (Multilayer), PDF (RGB), PDF (CMYK)
to the export dialog; CMYK types bypass the separated-layers loop and
composite with a white background before color conversion
Backend:
- sam_service.py: auto-downloads SAM ViT-B (~375 MB) on first use with
progress tracking; loads model to CUDA/MPS/CPU; predict_points() takes
multi-point prompts (include/exclude labels) and returns best mask
- POST /api/segment/point: SAM point-prompt endpoint; returns mask PNG
- GET /api/segment/install-status: poll download progress
- POST /api/segment/install: explicit trigger (also auto on first click)
- main.py: pre-download SAM on startup alongside NCNN
Frontend (ai_edit.js):
- Click mode (default): click object → SAM generates mask instantly
Alt+click → subtract (deselect over-selected area)
Multiple clicks accumulate for multi-object or refinement
- Brush + / Brush − modes: paint to add or erase from SAM mask by hand
- If SAM model is still downloading on first click: inline progress bar,
user retries the click when done
- Unified action bar: Erase | Replace (inline prompt) | Upscale | Expand | Clear
- All modes share the same mask canvas; SAM and brush are fully composited
https://claude.ai/code/session_01B58MaJCU1R6KwBDJCp8AfN
Before: 4 separate tools (ai_inpaint, ai_lama_erase, ai_smart_inpaint,
ai_replace_selection) with no clear relationship, buried in toolbar.
After: one "AI Edit" brush tool. Paint mask → floating action bar appears:
Erase — LaMa removes masked content, no prompt needed
Replace — inline prompt reveals in-place ("make her smile" / "replace
with a wolf") → calls /api/inpaint/remote
Upscale — delegates to existing upscale module
Expand — delegates to existing outpaint module
Clear — wipe mask and start over
Brush size controlled by tool attributes (5–200px).
Red overlay gives immediate visual feedback while painting.
Floating panel appears as soon as first brush stroke is made.
https://claude.ai/code/session_01B58MaJCU1R6KwBDJCp8AfN
- Add _vulkan_available(): checks /dev/dri/renderD* on Linux, assumes
true on macOS/Windows; set REALESRGAN_NCNN=force to override
- Add _test_ncnn_binary(): test-runs the binary after install and checks
stderr for "no vulkan" — marks skipped if Vulkan init fails at runtime
- ensure_ncnn_installed() now returns early with state=skipped when no
Vulkan detected, avoiding a wasted ~30MB download on CPU-only servers
- Recommend PyTorch CPU when available on headless (AI quality, slow but
works); Lanczos as final fallback
- Frontend: handle state=skipped immediately (no polling needed), show
brief informational toast; show "Headless server" note in dialog
https://claude.ai/code/session_01B58MaJCU1R6KwBDJCp8AfN
- upscale.py: add InstallStatus dataclass + ensure_ncnn_installed() async
function that downloads and extracts the NCNN binary for the current
platform (Linux/macOS/Windows), tracks progress (0-100%), and busts the
caps cache when done
- main.py: trigger ensure_ncnn_installed() as a background task on app
startup when no AI upscaler is detected
- print_tools.py: /upscale/available triggers install task when no AI
upscaler found; new GET /upscale/install-status endpoint for polling
- upscale.js: if no AI upscaler on open, poll install-status showing a
progress bar notification, then refresh caps and proceed when done
https://claude.ai/code/session_01B58MaJCU1R6KwBDJCp8AfN
Detection priority (probed once, cached):
1. Real-ESRGAN PyTorch + CUDA GPU → fastest, best quality
2. Real-ESRGAN PyTorch + Apple MPS → fast on Apple Silicon
3. Real-ESRGAN NCNN Vulkan binary → fast on any GPU via Vulkan (no CUDA needed)
4. Real-ESRGAN PyTorch CPU → works, slow (warned in UI)
5. Lanczos → always available, instant fallback
Backend:
- services/upscale.py: full capability probe (probe_upscale_capabilities),
implementations for PyTorch (CUDA/MPS/CPU auto-device) and NCNN binary,
upscale_sync() resolves method with fallback chain,
async upscale_image() runs in thread pool
- print_tools.py: /api/print/upscale uses new service; method="auto" by default;
GET /api/print/upscale/available returns full capability map with device info
and recommended_label; POST /api/print/upscale/refresh-caps busts cache
without restart (useful after installing NCNN binary into container)
Frontend:
- upscale.js: fetches capability map on first open; builds method selector showing
only available options; labels recommended method with ★; shows device info
(CUDA/MPS/CPU/NCNN) in dialog; maps display label back to method key on submit;
shows actual method used in success toast and undo history entry
Scripts:
- scripts/download_realesrgan.py: downloads NCNN Vulkan binary for current platform
(Linux/macOS/Windows) to /app/data/models/realesrgan/; makes executable;
run inside container or locally
https://claude.ai/code/session_01B58MaJCU1R6KwBDJCp8AfN
Backend — new /api/print/* router:
- POST /api/print/frame-fit: fit image to 4x6/5x7/8x10/11x14/16x20/20x24/24x36
and square sizes (4x4/8x8/12x12) at configurable DPI.
Three modes:
crop — center-crop to aspect ratio, Lanczos scale to print res (no AI)
extend — scale to fill one dimension, AI-inpaint the gap; mirror-fill fallback
smart — auto: extend if gap < 15% of frame dimension, else crop
Auto-detects orientation from image shape; respects explicit portrait/landscape.
- POST /api/print/upscale: Lanczos scale (always) or Real-ESRGAN (if installed)
- GET /api/print/frame-sizes: frame catalogue with pixel dimensions at 300dpi
- GET /api/print/upscale/available: reports whether Real-ESRGAN is installed
Frontend:
- modules/image/frame_fit.js: dialog with frame size, orientation, mode, DPI,
optional extend prompt; shows current image size; result as new layer option
- modules/image/upscale.js: dialog with scale factor (1.5–4×), method selector
(auto-hides AI option if Real-ESRGAN not available); result as new layer option
- config-menu.js: Fit to Frame... and Upscale... added under Image menu
https://claude.ai/code/session_01B58MaJCU1R6KwBDJCp8AfN
Each operation (inpaint, txt2img, img2img, outpaint) can now use a different
provider. Resolution order: per-op override → global AI_PROVIDER default.
Example: txt2img→openai, inpaint→invokeai, everything else→invokeai default.
Backend:
- config.py: add AI_PROVIDER_INPAINT / TXT2IMG / IMG2IMG / OUTPAINT settings
- remote_provider.py: get_remote_provider(operation) resolves override then default;
_build_provider() extracted as shared factory; _OP_FIELD maps op→setting name
- ai_tools.py: each endpoint passes its operation to _require_remote();
GET /api/config runs per-op health checks concurrently, returns operations map
and overrides; POST /api/config accepts and applies per-op override fields
Frontend:
- ai_provider_settings.js: four new selects (inpaint/txt2img/img2img/outpaint);
persists to localStorage and sends per-op fields to POST /api/config
- provider-badge.js: shows override summary (e.g. "invokeai · txt2img→openai")
and per-op health in tooltip
- .env.example: document per-op override env vars with examples
https://claude.ai/code/session_01B58MaJCU1R6KwBDJCp8AfN
Frontend:
- tools/ai_replace_selection.js: use any selection → remote inpaint with prompt
- modules/generate/text_to_image.js: Text → Image dialog (new layer or replace canvas)
- modules/generate/outpaint.js: Expand Canvas in any direction via remote provider
- modules/tools/ai_provider_settings.js: in-app provider config (OpenAI / InvokeAI /
ComfyUI / Replicate); persists to localStorage, pushes to POST /api/config at runtime
- config.js: register ai_replace_selection tool
- config-menu.js: add Generate menu (Text→Image, Outpaint); AI Provider Settings under Tools
- modules/help/about.js: updated credits (LaMa, rembg, SAM, InvokeAI, ComfyUI, OpenAI)
- api/capabilities.js: add refreshCapabilities() for post-save cache invalidation
Backend:
- routers/ai_tools.py: POST /api/config — apply provider settings at runtime
without restart (session-scoped, non-persistent; .env for permanence)
https://claude.ai/code/session_01B58MaJCU1R6KwBDJCp8AfN
- Switch U2Net from onnxruntime to OpenCV DNN to avoid
"cannot enable executable stack" error in Docker
- Add alertify dialog styling to fix white text on white
background issue in popups
- Add offerFloatSelection() to AI Paint that prompts user
after selection to move/scale it (Canva-like workflow)
- Auto-switch to Select tool after floating selection
https://claude.ai/code/session_01CLedz6CanT9t46KBvng3vz
- Fix brush_select isProcessing flag not resetting after first use
(reset in on_leave() when switching tools)
- Add onnxruntime dependency for U2net background removal
- Use full U2net model (176MB) instead of lightweight for better quality
https://claude.ai/code/session_01CLedz6CanT9t46KBvng3vz
- Fix brush_select and smart_select mask scaling to match layer dimensions
- Add "Float Selection" feature to Select tool - when switching to Select
with an active AI selection, offers to copy it to a movable layer
- Add aspect ratio lock toggle (🔗 button) in layer details panel
- When locked, changing width auto-updates height and vice versa
- Add U2net model auto-download - will download lightweight u2netp.onnx (~4MB)
automatically if no model found
- Improve error messages for background removal
- Register on_activate for select tool in config
Workflow: Select object with AI tool → Click Select tool → "Float" selection
→ Move/scale the floated layer freely
https://claude.ai/code/session_01CLedz6CanT9t46KBvng3vz
- Fix My Library: Add CSS styling for library browser, items now visible
- Integrate My Library into Shapes tool with tabbed interface
- Improve AI Inpaint: Add transform mode for scaling/sizing selections
- Add helpful guidance explaining inpaint vs transform modes
- Add U2net as alternative background removal (avoids rembg issues)
- Create U2net model definition and download script
- Improve Caddyfile with multiple options and troubleshooting guide
Note: Brush Select (AI Paint) tool was already implemented and working.
https://claude.ai/code/session_01CLedz6CanT9t46KBvng3vz
scikit-image 0.26+ (required by rembg) needs numpy 2.x compatible
packages. Updated torch from 2.1.2 to 2.4+ and torchvision from
0.16.2 to 0.19+ which officially support numpy 2.x.
https://claude.ai/code/session_01MYpjNQXD1fZE4gCweGU4QQ
rembg>=2.0.70 requires scikit-image>=0.26.0, which conflicted with
the pinned scikit-image==0.22.0. Instead of pinning specific versions,
let pip resolve compatible versions automatically based on rembg's
requirements.
Also removed numpy pin as it may conflict with torch/rembg dependencies
- pip will select a compatible version.
https://claude.ai/code/session_01MYpjNQXD1fZE4gCweGU4QQ
rembg 2.0.70+ requires Pillow>=12.1.0,<13.0.0 which conflicted with
the pinned Pillow==10.2.0. Updated to use the version range that
satisfies rembg while remaining compatible with scikit-image and
torchvision (both have no upper bound on Pillow).
https://claude.ai/code/session_01MYpjNQXD1fZE4gCweGU4QQ
- Use onnxruntime>=1.17.0 (fixed executable stack issues, no execstack needed)
- Use rembg>=2.0.70 with BiRefNet model support
- Update all remove-background endpoints to use birefnet-general model
- BiRefNet provides better edge detection and matting quality than u2net
- Falls back to default model if BiRefNet unavailable
- Use onnxruntime 1.14.1 (older version without executable stack requirement)
- Make rembg pre-download optional (won't fail build if onnxruntime has issues)
- Remove Background will be disabled if rembg can't load
- Save layer or selection to personal library
- Browse library with category grouping and thumbnails
- Insert assets as new layers (double-click or Insert button)
- Delete unwanted assets
- Export library to JSON for backup
- Import library from JSON backup
- Assets stored in browser IndexedDB (persistent)
- Categories: General, Shapes, Borders, Icons, Templates, etc.
- New Image > Selection Effects submenu
- Invert Selection: Invert colors only within selected area
- Adjust Selection: Brightness/contrast/gamma on selection only
- Greyscale Selection: Convert just the selected area to greyscale
- All effects work with any selection tool (Smart Select, Brush Select, etc.)
- Useful for CNC depth maps where specific objects need different treatments
The onnxruntime library requires an executable stack which is blocked
by security restrictions in Docker containers. This adds execstack tool
and uses it to clear the executable stack flag after pip install.
https://claude.ai/code/session_01AGoPJaXqdJnnuxtmSv6NLR
- New tool: Paint/brush over objects to select them
- AI (SAM) detects actual object boundaries from brush strokes
- Collects sample points along brush path, sends to SAM
- Combines multiple masks for multi-object selection
- Shift+brush to add to existing selection
- Full cut/copy/delete support with mask-shaped results
- Visual feedback: brush stroke preview, sample points, marching ants
- Fix cut/copy to preserve mask shape with transparency
- Add missing CSS icons for magic_wand, lasso, ellipse_select tools
- Fix toast messages styling (visible on dark theme)
- Update AI Inpaint to work with all selection tools
- Change tab title to +miniPaint
- Add DNS config to docker-compose for external API access
Frontend:
- Added "Remove Background (AI)" to Image menu
- Created remove_background.js module with dialog options
- Added removeBackground method to API service
Backend:
- Added /tools/remove-background-base64 endpoint for miniPaint frontend
- Uses rembg library for AI-powered background removal
Features:
- Automatically detects main subject and removes background
- Option to create as new layer or replace current
- Enables transparency mode after removal
- Works with any image layer
New Selection Tools:
- Magic Wand: Click to select by color similarity (like GIMP)
- Configurable tolerance (0-100%)
- Contiguous or global mode
- Shift+Click to add to selection
- Lasso: Freehand selection by drawing
- Draw around area to select
- Shift+Draw to add to selection
- Ellipse Select: Draw elliptical/circular selections
- Drag to create ellipse
- Shift+Drag for perfect circle
- Alt+Drag to draw from center
Smart Select Improvements:
- Fixed copy/cut to layer errors
- Added Shift+Click for multi-select (additive selection)
- All operations use proper layer action system
All selection tools support:
- Ctrl+C: Copy selection to new layer
- Ctrl+X: Cut selection to new layer
- Delete: Delete selected area
- Escape: Clear selection
- Marching ants animation on selection edge
- SAM selection now shows actual mask contour instead of bounding box
- Added marching ants animation on the actual mask edge
- Added keyboard shortcuts:
- Ctrl+C: Copy selection to new layer
- Ctrl+X: Cut selection to new layer (removes from original)
- Delete: Delete selected area
- Escape: Clear selection
- Removed eye catalog download from startup (was failing with 429)
Frontend:
- Fix smart_select.js to properly render mask overlay
- Add marching ants border around selection
- Calculate selection bounds from mask
- Trigger re-render after mask is loaded
Backend:
- Fix inpaint endpoint to call edit_image() instead of inpaint()
- The AI providers use edit_image() method, not inpaint()
- Update main Dockerfile to copy all miniPaint static files:
- index.html
- dist/ (webpack bundle)
- images/ (icons and assets)
- src/css/ (stylesheets)
- Update backend main.py to conditionally mount static directories
- Checks if each directory exists before mounting
- Supports both React (assets/) and miniPaint (dist/, images/, src/) structures
New features:
- Smart Select tool: Click to select objects using SAM (Segment Anything)
- AI Inpaint tool: Edit selected regions with text prompts
Changes:
- frontend/src/js/tools/smart_select.js: SAM-powered selection tool
- frontend/src/js/tools/ai_inpaint.js: AI inpainting with prompt dialog
- frontend/src/js/services/api.js: API service for backend communication
- frontend/src/js/config.js: Register new tools
- frontend/src/css/layout.css: Tool icon styles
- frontend/images/icons/: SVG icons for new tools
- backend/app/routers/tools.py: New base64 API endpoints
- frontend/Dockerfile: Updated for miniPaint build
- frontend/nginx.conf: Added /api prefix proxy
ROOT CAUSE: ImageCanvas referenced props that don't exist:
- activeTool, onSmartSelect, onColorSelect were used but never declared
- selectionMode checked for 'smart'/'color' but App passes advancedToolMode with 'smart-select'/'color-select'
Changes:
- Update mode checking to use advancedToolMode instead of selectionMode for smart/color select
- Replace onSmartSelect/onColorSelect calls with onAdvancedToolClick
- Fix useEffect dependency array to reference actual props
- Add missing state/refs: currentZoom, imageRef, baseScaleRef, isDrawingRef
- Add zoom control buttons to canvas UI
- Fix tool mode indicator to check advancedToolMode
- Add zoom support via prop (scales image around center)
- Fix selection interaction: clicking on selection transforms, clicking elsewhere creates new
- Use refs for state accessed in event handlers (fixes stale closure issue)
- Add smart select mode (clicks call onSmartSelect with image coordinates)
- Add color select mode placeholder
- Add pan mode with cursor feedback
- Add move mode for manipulating selections
- Use dashed blue selection style (more visible)
- forwardRef to expose canvas methods
- Redesign layout: left toolbar, center canvas, right sidebar
- Add vertical tool strip with keyboard shortcuts (V, R, E, F, W, B, G, Z, H)
- Collapsible sidebar panels: Tool Options, AI Edit, Quick Actions, Eyes, Layers, History
- Canvas now fills available space with checkerboard background
- Add zoom controls in status bar
- Add menu bar with New, Save, Reset
- Remove old multi-column layout
- Create unified Dockerfile with multi-stage build (Node + Python)
- FastAPI now serves React static files directly
- Remove frontend service and nginx dependency
- Simplify docker-compose to single service
- All routes work without proxy configuration
Frontend changes:
- Wire Smart Select and Color Select to canvas click handlers
- Add externalSelection prop to ImageCanvas for displaying AI-generated selections
- Add zoom controls (mouse wheel + buttons) to ImageCanvas
- Fix layer buttons (New Layer, Delete, Duplicate) with proper handlers
- Lift advancedToolMode state to App.jsx for coordination between components
- Add tool mode indicator overlay on canvas
Backend changes:
- Update smart-select endpoint to return JSON with polygon and bbox data
- Update color-select endpoint to return JSON with polygon and bbox data
- Add _mask_to_polygon helper function using OpenCV contour detection
- Add cv2 and base64 imports to tools.py
API changes:
- smartSelect and colorSelect now return { polygon, bbox, mask_base64 }
- Fix database path mismatch: download_sample_eyes.py now uses
ai_photo_edit.db instead of photoedit.db
- Add init_database.py script to initialize DB before eye import
- Add AUTO_DOWNLOAD_SAM=true environment variable (default: enabled)
- Update entrypoint.sh to:
1. Initialize database first
2. Auto-download SAM model (~375MB) on first startup
3. Then import eyes (now works since DB exists)
- Update path detection to work in both Docker and local environments
- Add torch, torchvision, segment-anything to requirements
- Create download_sam_model.py script to fetch SAM checkpoint
- Update tools.py to use local SAM with Replicate API fallback
- Add SAM model check to entrypoint.sh with helpful instructions
- Model persists in /app/data/models via Docker volume mount
- Create entrypoint.sh that auto-downloads sample eyes on first run
- Update Dockerfile to use entrypoint script
- Rewrite .env.example with step-by-step setup instructions
- Add detailed troubleshooting section
- Clarify which models work for inpainting vs text-to-image
- Implement SAM object selection via Replicate API
- Click on any object to select it with AI precision
- Falls back to flood-fill if Replicate API unavailable
- Update Dockerfile for rembg dependencies
- Add required system libraries (libsm6, libxext6, etc)
- Pre-download rembg model during build
- Create data directories for models and patches
Backend:
- Add /tools router with background removal, smart select, color select
- Add rembg dependency for AI background removal
- Add layer management API (list, flatten)
- Fix transparency preservation in blend_patch (veil collapse fix)
- Preserve alpha channel when reverting/resetting images
Frontend:
- Add AdvancedTools panel with background removal, smart select, color select
- Add Layers panel with drag-to-reorder, visibility toggle, flatten
- Add toolsApi for new backend endpoints
- Make right panel scrollable for additional controls
This adds "Photoshop light" capabilities:
- Remove background and create layer
- Smart object selection (click to select)
- Color selection with tolerance
- Layer system with compositing
Creates scripts/download_sample_eyes.py that:
- Downloads public domain classical sculpture images from Wikimedia
- Creates patch records in database
- Generates thumbnails
- Populates the Eye Catalog with starter content
Fixes:
- Remove scale limit to allow image to fill canvas
- Fix Lasso tool by using Polygon instead of Polyline
- Remove duplicate Clear Selection button
- Fix Undo/Redo dependencies with useCallback
New Features:
- Eye Catalog UI for browsing and applying saved eyes
- Upload new eyes to catalog
- Apply eyes to selected areas with feathering
- Delete eyes from catalog
- Make canvas use full viewport height for larger image display
- Add download button to export edited images
- Implement Undo/Redo with Ctrl+Z / Ctrl+Y keyboard shortcuts
- Add transform controls for selections (move, resize, rotate)
- Add warn on reload to prevent data loss
- Make project naming optional, auto-generate from filename
Added complete system for populating eye catalog with classical carved eyes:
Features:
- Eye import script (import_eyes.py) for batch/single eye imports
- Public domain source guide (museums: Met, Smithsonian, Getty, etc.)
- Seed catalog system for pre-populating database
- Organized by emotion (serene, fierce, wise, peaceful, etc.)
- Organized by style (Greek, Roman, Egyptian, Renaissance)
- Organized by side (left, right, both)
- Auto-generates thumbnails and metadata
- CNC-ready tagging system
Workflow:
1. Download classical sculpture photos from public domain museums
2. Crop eyes in any image editor
3. Run import script with metadata
4. Eyes saved to catalog with proper tags
5. Apply to colored photos (pure image compositing, no AI regeneration)
6. Convert result to grayscale for CNC carving
Documentation:
- PUBLIC_DOMAIN_EYE_SOURCES.md: Where to find carved eyes
- scripts/README.md: How to import eyes
- Includes recommended starting collection (10 essential eyes)
Benefits for wood carving:
- Build library from master sculptors (2000+ years of proven designs)
- Reusable across all projects
- Consistent emotional weight in carvings
- No AI regeneration - just intelligent copy/paste/blend
- Perfect for CNC workflow (colored preview → grayscale → carve)
The patch library system uses PIL/OpenCV for image compositing,
NOT AI regeneration, so it preserves exact carved geometry.
- Changed backend port from 8000 to 8101 (avoiding port conflict)
- Added REPLICATE_API_KEY to docker-compose environment variables
- Simplified Caddyfile - only need to reverse proxy frontend
- Frontend nginx internally handles routing API calls to backend
Claude