StableLion: Robust Sign-Based Optimization Through Layerwise Adaptation

View PDF

Abstract:We present StableLion, a stabilized sign-based optimizer combining layerwise learning rate adaptation with gradient norm clipping for language model pretraining. While sign-based methods like Lion offer memory efficiency, they often suffer from training instability. StableLion addresses this through three mechanisms: (1) parameter-specific trust ratios bounding update magnitudes, (2) layerwise learning rate adaptation inspired by LAMB, and (3) gradient norm stabilization. On a 134M parameter Qwen model trained on FineWeb, StableLion achieves 4.931 validation loss, outperforming Lion (6.114) and approaching AdamW (4.927) while using 30\% less memory than adaptive methods. We provide ablation studies and implementation details to support these findings.