StableAutoLR: Adaptive Learning Rate Optimization with Gradient Stability for Language Models

View PDF

Abstract:We present StableAutoLR, an optimizer for transformer language models that combines loss-aware learning rate adaptation with gradient stability mechanisms. On the FineWeb benchmark with a 134M parameter Qwen model, StableAutoLR achieves a validation loss of 4.518, improving upon AdamW's 4.926 while maintaining comparable computational efficiency. Our key contributions include: (1) a dynamic learning rate adaptation rule responsive to both loss trends and gradient statistics, (2) a stability-preserving gradient clipping mechanism, and (3) empirical validation of the optimizer's performance across different training phases. We provide complete implementation details and ablation studies to support reproducibility.