Curvature-Adaptive Muon Optimizer: Lessons from a Negative Result

View PDF

Abstract:This paper presents a detailed empirical evaluation and analysis of the Curvature-Adaptive Muon Optimizer (CAMuon), a novel optimization approach combining adaptive momentum with curvature information and periodic orthogonalization. While our theoretical framework suggested potential benefits from incorporating Hessian information and orthogonal updates, experimental results on a 134M parameter transformer model demonstrated significant underperformance compared to baselines, achieving a validation loss of 9.932 versus 3.537 for Muon and 4.927 for AdamW. Through comprehensive implementation details, failure analysis, and comparisons with recent optimizer variants, we identify key challenges in adapting second-order methods for large-scale language model training and provide concrete recommendations for future research directions.