SophiaGPlus: Analysis of Layer-Adaptive Second-Order Optimization for Language Models

View PDF

Abstract:This paper presents a detailed empirical analysis of SophiaGPlus, a modified version of the Sophia optimizer incorporating layer-specific learning rate scaling and dynamic variance stabilization. Through extensive ablation studies and comparison with AdamW and Sophia baselines, we demonstrate that while our approach (validation loss: 5.155) improves upon AdamW (4.927), it underperforms the original Sophia optimizer (5.091). We provide comprehensive diagnostic analysis of the failure modes, including sensitivity to layer scaling factors and interaction between momentum and curvature updates.