Spectral Normalization
Spectral Normalization constrains the Lipschitz constant of network layers by normalizing with the largest singular value – standard stabilization in GANs.
Spectral Normalization constrains the Lipschitz constant via the largest singular value – the standard stabilization for GANs.
Explanation
W_SN = W / σ(W), where σ(W) is the largest singular value (efficiently computed via power iteration). Limits the "aggressiveness" of the discriminator in GANs and stabilizes training without gradient penalty.
Marketing Relevance
Standard stabilization technique in GANs (SNGAN, BigGAN, StyleGAN) and increasingly in diffusion models.
Origin & History
Miyato et al. (2018) introduced Spectral Normalization for GANs (SNGAN). BigGAN (2018) and StyleGAN (2019) adopted the technique. Today also common in diffusion models (e.g., Stable Diffusion U-Net).
Comparisons & Differences
Spectral Normalization vs. Gradient Penalty (WGAN-GP)
Gradient Penalty computes gradient norm per batch (expensive); Spectral Norm normalizes weights directly (faster, simpler).