OrthoLion: A Novel Geometric Approach to Transformer Optimization

View PDF

Abstract:This paper introduces OrthoLion, a new optimization algorithm combining orthogonal weight updates with sign-based adaptation for large language models. Through extensive experiments on the FineWeb benchmark, we demonstrate our method achieves a validation loss of 5.859, showing improved stability over Lion (6.114) while remaining competitive with adaptive methods. We provide theoretical analysis of our layer-aware geometric constraints and comprehensive ablation studies validating our design choices.