SpectralLion: Spectral Processing Meets Sign-Based Optimization for Language Models

View PDF

Abstract:We introduce SpectralLion, a novel optimizer combining spectral processing techniques with sign-based updates for training large language models. Our method processes gradients through singular value decomposition before applying sign-based updates inspired by the Lion optimizer. On the FineWeb benchmark with a 134M parameter model, SpectralLion achieves a validation loss of 4.521, representing an 8.2\% improvement over AdamW (4.927) and 26\% improvement over Lion (6.114). While computationally more expensive than standard optimizers due to SVD operations, SpectralLion demonstrates that spectral processing can meaningfully improve optimization when combined with sign-based updates.