Lion (Evolved Sign Momentum)
Optimizer discovered by Google Brain through AutoML search that only uses the sign of gradients – simpler than Adam, often comparable results.
Lion is an optimizer discovered through AutoML that only uses gradient signs – saves memory vs. Adam and shows that optimizers can be automatically discovered.
Explanation
Lion uses sign(momentum + gradient) as update direction with uniform step size. Less memory than Adam (no 2nd moment), but requires careful LR and weight decay tuning.
Marketing Relevance
Lion shows that AutoML can discover not only models but also optimizers. Saves ~15-30% memory vs. Adam.
Common Pitfalls
Requires different hyperparameters than Adam (lower LR, higher weight decay). Not as extensively tested as AdamW. Performance advantages not consistent.
Origin & History
Chen et al. (Google Brain, 2023) used evolutionary search over millions of optimizer candidates and found Lion. It achieved state-of-the-art on vision and language benchmarks with less memory than AdamW.
Comparisons & Differences
Lion (Evolved Sign Momentum) vs. AdamW
AdamW stores 2 moment buffers (memory-intensive); Lion uses only 1 (sign-based). Lion needs different hyperparameter ranges.
Lion (Evolved Sign Momentum) vs. SGD
SGD is manually designed, Lion discovered through AutoML. Both are simpler than Adam, but Lion adapts better to different tasks.