StableAdamW: Variance-Stabilized Optimization for Language Models

View PDF

Abstract:We present StableAdamW, a modified version of AdamW that addresses training instability through controlled variance clipping of second moment estimates. While the performance improvement over AdamW is modest (4.919 vs 4.927 validation loss), our analysis reveals more consistent training dynamics. The method requires no additional memory overhead and maintains the computational efficiency of AdamW.