Spectral Momentum Optimization for Language Models

View PDF

Abstract:This paper investigates spectral momentum optimization for language models. Our final simplified StableMomentum optimizer achieved a validation loss of 8.114, underperforming standard baselines. We document the implementation challenges and lessons learned.