stable-diffusion_skill

Overview

This skill provides state-of-the-art text-to-image generation using Stable Diffusion models via the HuggingFace Diffusers library. It covers text-to-image, image-to-image, inpainting, ControlNet conditioning, LoRA adapters, and memory optimizations for GPU workflows. The content is practical and focused on getting reproducible, high-quality images with configurable pipelines and schedulers.

How this skill works

The skill orchestrates Stable Diffusion pipelines that convert text prompts into embeddings, run a denoising loop with a scheduler and a UNet/transformer noise predictor, and decode latents with a VAE into final images. It exposes utilities for image conditioning (image-to-image, masks, ControlNet) and adapter loading (LoRA) plus memory/precision settings to fit models on different hardware. You can swap schedulers, set guidance and negative prompts, and use torch generators for reproducible outputs.

When to use it

• Generate photorealistic or stylized images from text prompts.
• Transform or stylize existing images (image-to-image) or create variations.
• Perform inpainting or outpainting to fill or extend image regions.
• Apply spatial control using ControlNet (edges, pose, depth).
• Prototype fast, low-step generation or produce high-quality outputs with SDXL.

Best practices

• Use FP16/BF16 on compatible GPUs and enable model CPU offload when OOM occurs.
• Tune guidance_scale (7–12 typical) and num_inference_steps for tradeoff between fidelity and speed.
• Use negative_prompt to remove common artifacts and unwanted styles.
• Swap to DPMSolverMultistep or UniPC for faster, high-quality convergence when reducing steps.
• Enable attention/vae slicing and xFormers (if available) for large images or limited memory.

Example use cases

• Create marketing visuals: generate multiple variations of a product scene for A/B testing.
• Photo retouching: inpaint or replace objects in photos using masks.
• Style transfer and character design: apply LoRA adapters to inject artistic styles.
• Pose-to-image: use ControlNet openpose to generate consistent human poses.
• Rapid prototyping: use LCM/LCM-scheduler and fused LoRA for 4–8 step preview renders.