Analysis of Spectral Momentum Optimizer

View PDF

Abstract:This paper analyzes a spectral momentum optimizer for transformers that achieved worse performance (loss=9.85) compared to standard baselines like AdamW (loss=4.93). We identify key challenges in applying spectral methods to transformer optimization.