📋 Table of Contents
- ▸How diffusion models learn to create realistic images
- ▸
- ▸
- ▸
- ▸
- ▸
- ▸
- ▸
How diffusion models learn to create realistic images
When you look at an image generated by a model like Stable Diffusion, you’re seeing the end of a long chain of learned denoising. It starts with pure Gaussian noise and, step by step, a neural network removes the noise until a coherent scene emerges. The magic isn’t in inventing pixels from scratch—it’s in reversing a controlled corruption process the model learned from real data.
How diffusion models work step by step
The process begins with the forward diffusion process. You take a real image x₀ and gradually add small amounts of Gaussian noise over T timesteps. At each step t, the noisy image xₜ is generated by sampling from a normal distribution centered on xₜ₋₁ with variance βₜ:
xₜ = √(1 − βₜ) · xₜ₋₁ + √βₜ · ε, ε ~ N(0, I)
After enough steps, the image becomes indistinguishable from pure noise. The key insight is that this corruption schedule can be reversed. Instead of adding noise, you train a neural network ε_θ(xₜ, t) to predict the noise ε that was added at step t, then subtract it.
This is why the reverse process feels like magic: the model doesn’t build an image pixel by pixel. It learns to undo the damage it was trained to recognize. When you run inference, you start from random noise and iteratively denoise using the predicted noise. In practice, most implementations use around 1,000 steps, but recent methods can reduce this to 50–200 steps without significant quality loss.
The math behind the noise schedule
The noise schedule defines how βₜ changes over time. A linear schedule from β₁=10⁻⁴ to β_T=0.02 is common in early work, but cosine schedules often perform better. In the DDPM paper, a cosine schedule improved FID scores by about 10–15% compared to linear schedules on CIFAR-10. The cosine schedule uses:
ᾱₜ = cos(πt/2T)² / cos(π(t−1)/2T)²
This smooths the variance drop-off, reducing the risk of underfitting early in training and overfitting late. Cosine schedules particularly shine when generating high-resolution images where fine details matter. The schedule also affects training stability: too aggressive a schedule leads to gradient explosion, while too slow a schedule makes learning inefficient.
Neural network architectures that power diffusion
Most diffusion models rely on a U-Net backbone for its skip connections that preserve spatial information during downsampling and upsampling. The U-Net in Stable Diffusion v1.5 has 860M parameters and uses 4 downsampling blocks with attention at 16×16 resolution.
Attention mechanisms are crucial for modeling long-range dependencies. In latent diffusion models, spatial attention is applied at resolutions 16×16 and 8×8, with grouped-query attention introduced in SDXL to reduce memory usage. Without attention, models struggle with coherent textures and object relationships.
For conditioning, text-to-image systems like Stable Diffusion use cross-attention layers where text embeddings from a CLIP ViT-L/14 encoder are projected and added to the U-Net’s intermediate layers. This allows the model to respond to prompts like “a cyberpunk city at night” without explicit architectural changes.
Training strategies that make diffusion models practical
Almost all modern diffusion models predict noise ε rather than the clean image x₀. This choice simplifies the loss function and stabilizes training. The standard loss is mean squared error:
L = Eₜ,x₀,ε [ ||ε − ε_θ(xₜ, t)||² ]
For higher perceptual quality, some researchers replace MSE with LPIPS (Learned Perceptual Image Patch Similarity), which matches human perception better. In experiments, switching to LPIPS improved FID by about 8% on FFHQ at 256×256.
Training on high-resolution images requires tricks like gradient checkpointing and mixed-precision (FP16). For 1024×1024 images, Stable Diffusion uses a batch size of 8–16 across 8 A100 GPUs with gradient accumulation. Without these optimizations, training would take weeks instead of days.
Sampling techniques that control image generation
The original DDPM sampler runs the reverse process with 1,000 steps, but it’s slow. DDIM (Denoising Diffusion Implicit Models) introduced deterministic sampling with fewer steps by approximating the reverse process as an ODE. On MS-COCO 256×256, DDIM with 50 steps matches DDPM’s FID while being 20× faster.
For stronger prompt adherence, classifier-free guidance is now standard. Instead of conditioning on text alone, the model jointly trains on unconditional and conditional objectives, then combines predictions:
ε̃_θ(xₜ, t, y) = ε_θ(xₜ, t, ∅) + s · (ε_θ(xₜ, t, y) − ε_θ(xₜ, t, ∅))
With s=7.5, this improves CLIP score by about 15% on DrawBench prompts. The tradeoff is increased inference time due to two forward passes per step.
Evaluating diffusion model quality and performance
The most common metric is FID (Fréchet Inception Distance), which compares feature distributions between real and generated images. On ImageNet 256×256, Stable Diffusion 2.1 achieves FID=12.6, while Imagen reaches 7.3 with a larger U-Net and higher resolution training.
Inception Score (IS) measures both image quality and diversity, but it’s biased toward low-diversity models. CLIP score evaluates text-image alignment by computing cosine similarity between image and text embeddings. For human evaluation, studies show that CLIP score correlates better with human preference than FID, especially for text-to-image tasks.
Observations reveal that models trained with perceptual loss and larger batch sizes tend to score higher on human evaluations even when FID is similar. This suggests FID alone doesn’t capture all aspects of quality.
Real-world applications and limitations
Text-to-image systems like Stable Diffusion XL use latent diffusion to reduce memory usage. They compress images to 64×64 latent space using a VAE, then run diffusion in this compressed space. This allows inference on a single RTX 3060 with 8GB VRAM.
But limitations remain. Artifacts often appear in fine details like hands or text. Prompt sensitivity is high—minor wording changes can flip the output. Training requires massive datasets and compute: Stable Diffusion XL used 10M–20M images and 16 A100 GPUs for weeks.
Emerging directions include 3D diffusion models like DreamFusion, which distill 2D diffusion into NeRF representations, and video diffusion where temporal consistency is enforced with additional 3D convolutions. The field moves fast—experiments with video diffusion using 1-second clips at 128×256 resolution show inconsistent quality but rapid improvement.