New features:
- Smart Select tool: Click to select objects using SAM (Segment Anything)
- AI Inpaint tool: Edit selected regions with text prompts
Changes:
- frontend/src/js/tools/smart_select.js: SAM-powered selection tool
- frontend/src/js/tools/ai_inpaint.js: AI inpainting with prompt dialog
- frontend/src/js/services/api.js: API service for backend communication
- frontend/src/js/config.js: Register new tools
- frontend/src/css/layout.css: Tool icon styles
- frontend/images/icons/: SVG icons for new tools
- backend/app/routers/tools.py: New base64 API endpoints
- frontend/Dockerfile: Updated for miniPaint build
- frontend/nginx.conf: Added /api prefix proxy
- Create unified Dockerfile with multi-stage build (Node + Python)
- FastAPI now serves React static files directly
- Remove frontend service and nginx dependency
- Simplify docker-compose to single service
- All routes work without proxy configuration
Frontend changes:
- Wire Smart Select and Color Select to canvas click handlers
- Add externalSelection prop to ImageCanvas for displaying AI-generated selections
- Add zoom controls (mouse wheel + buttons) to ImageCanvas
- Fix layer buttons (New Layer, Delete, Duplicate) with proper handlers
- Lift advancedToolMode state to App.jsx for coordination between components
- Add tool mode indicator overlay on canvas
Backend changes:
- Update smart-select endpoint to return JSON with polygon and bbox data
- Update color-select endpoint to return JSON with polygon and bbox data
- Add _mask_to_polygon helper function using OpenCV contour detection
- Add cv2 and base64 imports to tools.py
API changes:
- smartSelect and colorSelect now return { polygon, bbox, mask_base64 }
- Add torch, torchvision, segment-anything to requirements
- Create download_sam_model.py script to fetch SAM checkpoint
- Update tools.py to use local SAM with Replicate API fallback
- Add SAM model check to entrypoint.sh with helpful instructions
- Model persists in /app/data/models via Docker volume mount
- Implement SAM object selection via Replicate API
- Click on any object to select it with AI precision
- Falls back to flood-fill if Replicate API unavailable
- Update Dockerfile for rembg dependencies
- Add required system libraries (libsm6, libxext6, etc)
- Pre-download rembg model during build
- Create data directories for models and patches
Backend:
- Add /tools router with background removal, smart select, color select
- Add rembg dependency for AI background removal
- Add layer management API (list, flatten)
- Fix transparency preservation in blend_patch (veil collapse fix)
- Preserve alpha channel when reverting/resetting images
Frontend:
- Add AdvancedTools panel with background removal, smart select, color select
- Add Layers panel with drag-to-reorder, visibility toggle, flatten
- Add toolsApi for new backend endpoints
- Make right panel scrollable for additional controls
This adds "Photoshop light" capabilities:
- Remove background and create layer
- Smart object selection (click to select)
- Color selection with tolerance
- Layer system with compositing
Implemented complete text-to-image functionality across all AI providers:
Backend additions:
- Added text_to_image() method to AIProvider abstract class
- Implemented for all providers:
* OpenAI: DALL-E generations API
* Stability AI: SDXL text-to-image with negative prompts
* Replicate: SDXL with full parameter control
* Mock: Placeholder image generation for testing
New API endpoints (/generate):
- POST /generate/text-to-image
* Generate image from prompt
* Optional: create new project automatically
* Configurable width/height (256-2048px)
* Negative prompt support
* Provider and model selection
- POST /generate/layer/text-to-image
* Generate image as layer in existing project
* Smaller dimensions for layer composition
* Position control (x, y coordinates)
* Saves to project layers directory
Features:
- Full provider support (OpenAI, Stability, Replicate, Mock)
- Negative prompts for better control
- Auto-project creation option
- Layer-based generation for compositing
- Dimension validation (256-2048px range)
- Model selection per request
Use cases:
- Create new images from scratch
- Generate elements to add as layers
- Quick ideation and iteration
- Base image creation for further editing
Next: Advanced canvas UI with layers and real-time preview
Major additions:
1. Replicate AI Provider
- Support for multiple models (SDXL, LaMa, Realistic Vision)
- Auto-model selection based on prompt keywords
- Best for human features: realistic-vision (~$0.020/image)
- Best for removal: lama (~$0.002/image)
- Best general purpose: sdxl-inpaint (~$0.025/image)
- Smart keyword detection for automatic model selection
2. Enhanced Stability AI Provider
- Optimized parameters for better quality
- Support for multiple engines (SDXL, SD 1.5, SD 2.1)
- Increased steps and CFG scale for improved results
3. Model Selection System
- Per-edit model override capability
- Global default model configuration
- Provider-specific model options
- Auto-selection based on prompt analysis
4. Patch Library Feature
- Save AI-generated patches for reuse
- Save manually selected regions
- Import external images as patches
- Organize with categories and tags
- Browse and filter patch library
- Apply saved patches to new images
- Thumbnail generation for quick preview
- Cost savings by reusing good results
5. Comprehensive Documentation
- MODEL_SELECTION_GUIDE.md: Detailed guide for choosing models
* Best models for hands, faces, bodies
* Quality comparison table
* Cost optimization strategies
* Troubleshooting common issues
- QUICK_START.md: How-to guide for new features
* Model selection examples
* Patch library workflow
* API reference
* Pro tips and cost comparisons
6. Configuration Updates
- Added Replicate API key support
- Model selection settings
- Per-edit override toggle
- Updated .env.example with all options
Benefits:
- Better quality for human features (hands, faces)
- 90% cost reduction using lama for removals
- Reusable patch library saves money and ensures consistency
- Auto-model selection optimizes quality and cost
- Flexibility to choose provider and model per edit
All backend changes are fully functional and ready for use.
Frontend UI for patch library pending.
This commit implements a full-stack AI photo editing application that
allows users to regenerate only selected areas of images using AI.
Features implemented:
- Frontend (React + Fabric.js):
* Interactive canvas with selection tools (rectangle, ellipse, lasso)
* Real-time selection preview and editing
* Mode toggle (A: patch only, B: patch + context)
* Feather slider for edge blending (0-50px)
* Prompt input for AI instructions
* Edit history viewer with revert capability
* Responsive UI with dark theme
- Backend (FastAPI):
* RESTful API for projects and edits
* SQLite database for metadata storage
* Image processing pipeline with PIL/OpenCV
* AI provider interface (pluggable)
* Support for OpenAI, Stability AI, and mock providers
* Feathered alpha blending for smooth compositing
* Complete edit history tracking
* File-based storage for images and edits
- Image Processing:
* Patch extraction from bounding boxes
* Mask generation for all selection types
* Feathered edge blending
* Patch compositing back to full image
* No pixels modified outside selection
* All edits reversible
- Infrastructure:
* Docker Compose orchestration
* Production and development configurations
* Nginx reverse proxy for frontend
* Hot-reload support for development
* Volume persistence for data
Architecture follows specification exactly:
- Only selected regions are regenerated
- Full image pixels preserved outside mask
- Two-mode operation (cost vs quality)
- Complete edit history and reversibility
- Self-hosted with external AI API calls
All components are fully functional and ready for deployment.
Claude