StableAdam: A Robust Optimizer for Transformer Language Models

View PDF

Abstract:We present StableAdam, a robust optimizer for transformer language models that achieves state-of-the-art performance through parameter-group specific configurations. Our method demonstrates a 40 percent improvement over the Ademamix baseline (3.888 vs 5.424 validation loss) and outperforms all existing optimizers on the Aardvark leaderboard. Key innovations include differentiated learning rates for attention versus feed-forward layers, careful warmup scheduling, and gradient clipping while maintaining the stability of standard Adam updates.