LAMVS: Layer-Adaptive Momentum Variance Scaling for Language Models

View PDF

Abstract:We present LAMVS (Layer-Adaptive Momentum Variance Scaling), a novel optimization method for training large language models. LAMVS extends AdamW by introducing layer-specific learning rate scaling and variance stabilization techniques. Through extensive experiments on the FineWeb benchmark using a 134M parameter Qwen 3 architecture, we demonstrate that LAMVS achieves a validation loss of 4.822, outperforming the AdamW baseline (4.9266) and other recent optimization approaches. Our ablation studies reveal that attention layers benefit most from increased learning rates (1.5x), while embedding layers perform best with standard rates. The paper includes complete implementation details, training dynamics analysis, and discussion of limitations to facilitate reproducibility and future research.