Column-Normalized Sophia: Enhancing Second-Order Optimization with Structural Adaptation

View PDF

Abstract:We present Column-Normalized Sophia, an enhanced second-order optimizer that combines SophiaG's curvature adaptation with column-wise structural normalization \cite{sophiag}. Experiments on a 134M parameter transformer show a 9.1\% improvement in validation perplexity compared to standard SophiaG, with results consistent across multiple random seeds.