Attentive Spectral Momentum: \\ Theoretical Foundations and Empirical Analysis

View PDF

Abstract:We present Attentive Spectral Momentum (ASM), an optimizer for transformer language models that combines adaptive momentum with theoretically-grounded parameter-specific adjustments. Building on recent work in spectral analysis of transformer gradients, ASM provides a principled approach to optimizing attention layers while maintaining full FSDP compatibility. On the FineWeb benchmark with a 134M parameter Qwen architecture, ASM achieves a validation loss of 4.85, outperforming AdamW (4.93) while demonstrating superior training stability. Comprehensive ablation studies validate our design choices, and we provide theoretical analysis of ASM's convergence properties. The optimizer's simplicity and compatibility with distributed training make it practical for real-world applications.