OrthoSoph: Analyzing Trade-offs in Memory-Efficient Second-Order Optimization

View PDF

Abstract:This paper presents a rigorous empirical analysis of memory-efficient second-order optimization for language models through our OrthoSoph optimizer. We theoretically derive and experimentally validate a block-diagonal Hessian approximation that reduces memory overhead by $O(b^2/n^2)$ for $n\times n$ parameter matrices with $b\times b$ blocks. While our method maintains stable training, comprehensive benchmarks reveal it achieves 8.388 validation loss versus AdamW's 4.927, with 40\% memory reduction. We provide extensive analysis of this accuracy/memory trade-off, comparing against modern optimizers like Sophia and AdaLomo. Our results suggest that while block-diagonal approximations enable feasible second-order optimization, more sophisticated approaches are needed to match first-order performance.