\documentclass[10pt]{article}
\usepackage[utf8]{inputenc}
\usepackage{graphicx}
\usepackage{booktabs}

\title{Re-evaluating AdamW Optimizer Modifications for Transformer Language Models}
\author{Aardvark}
\date{\today}

\begin{document}

\maketitle

\begin{abstract}
This paper presents a comprehensive empirical evaluation of various AdamW optimizer modifications for transformer-based language models. Through systematic experimentation, we demonstrate that many proposed modifications to the base AdamW optimizer fail to provide consistent improvements in model convergence or final performance. Our study evaluates four optimizer variants, including novel approaches involving orthogonal gradient processing and layer-specific momentum adaptation. Despite extensive tuning, our best-performing variant achieved a validation loss of 6.572, underperforming both the AdamW baseline (4.927) and state-of-the-art methods (3.537). These results suggest that fundamental improvements to adaptive optimization may require approaches beyond incremental modifications to existing methods.
\end{abstract}

\section{Introduction}
The optimization of transformer language models remains a challenging problem in deep learning research. While AdamW has become the standard optimizer for many natural language processing tasks, numerous modifications have been proposed to improve its performance. However, the effectiveness of these modifications is often inconsistent across different architectures and datasets.

Our work systematically evaluates several AdamW variants, beginning with two novel approaches: Adaptive Orthogonal Scaling (AOS) and Layer-Adaptive Momentum (LAM). Through iterative refinement, we arrived at simplified AdamW variants incorporating gradient clipping and learning rate warmup. Our comprehensive experiments on a 134M parameter transformer model reveal that these modifications, while providing stable training, fail to surpass the performance of carefully tuned baselines.

This paper makes the following contributions:
\begin{itemize}
\item A systematic evaluation of AdamW modifications including novel orthogonal gradient processing and layer-specific adaptation techniques
\item Empirical demonstration that complex modifications often degrade rather than improve optimizer performance
\item Guidelines for practitioners on effective optimizer configuration for transformer models
\end{itemize}

\section{Related Work}
Recent work in optimizer design has explored several directions for improving transformer training:

\subsection{Orthogonal Gradient Methods}
Building on the success of orthogonal regularization in deep learning, several works have proposed incorporating orthogonal constraints into optimization. OrthoAdam \cite{2511.00018} introduces orthogonal gradient processing during the update step, while Adaptive Orthogonal Momentum \cite{2511.00049} combines momentum with orthogonal projection. Our AOS variant explores similar concepts but with a simplified implementation.

\subsection{Layer-wise Adaptation}
The varying learning dynamics across transformer layers have motivated layer-specific optimization approaches. Layer-Adaptive Dual Momentum \cite{2510.00115} proposes different momentum factors for attention and feed-forward layers, an approach we extend in our LAM optimizer. However, our results suggest these methods require extremely careful tuning to be effective.

\subsection{Stability Enhancements}
Training stability remains a key challenge in large language models. StableAdam \cite{2510.00111} introduces gradient normalization and clipping mechanisms similar to those we evaluate. Our work confirms these techniques can improve stability but may not necessarily lead to better final performance.

\section{Methodology}
We evaluated four optimizer variants on a 134M parameter transformer model trained on the FineWeb dataset:

\subsection{Optimizer Variants}
\begin{enumerate}
\item \textbf{AOS (Adaptive Orthogonal Scaling)}: Combines AdamW with lightweight orthogonal gradient processing and automatic scaling factors
\item \textbf{LAM (Layer-Adaptive Momentum)}: Implements layer-specific momentum factors (0.95 for attention, 0.9 for MLP, 0.85 for embeddings)
\item \textbf{AdamWPlus}: AdamW with gradient clipping and cosine learning rate decay
\item \textbf{AdamWFinal}: Simplified AdamW with linear warmup and gradient clipping
\end{enumerate}

\subsection{Experimental Setup}
All experiments used consistent hyperparameters:
\begin{itemize}
\item Learning rate: 3e-4
\item $\beta_1$: 0.9, $\beta_2$: 0.999
\item Weight decay: 0.01
\item Warmup steps: 3000
\item Gradient clipping: 2.0
\end{itemize}

Training was conducted on the FineWeb dataset using a 134M parameter transformer with Qwen 3 architecture. We tracked both training and validation loss throughout the optimization process.

\section{Results}
Our experimental results reveal several key findings:

\begin{table}[h]
\centering
\caption{Validation Loss Comparison}
\label{tab:results}
\begin{tabular}{lc}
\toprule
Method & Validation Loss \\
\midrule
Muon (Baseline) & 3.537 \\
AdamW (Baseline) & 4.927 \\
AOS & 5.963 \\
LAM & 8.074 \\
AdamWPlus & 5.812 \\
AdamWFinal (Ours) & 6.572 \\
\bottomrule
\end{tabular}
\end{table}

As shown in Table \ref{tab:results}, all our proposed variants underperformed the AdamW baseline. The LAM optimizer, despite its sophisticated layer-specific adaptation, showed particularly poor convergence. Our final simplified AdamW variant achieved better stability than LAM but still failed to match the baseline performance.

\section{Discussion}
Our negative results provide several important insights for the machine learning community:

\subsection{Optimizer Complexity}
The consistent underperformance of our more complex variants (AOS and LAM) suggests that intricate modifications to AdamW may often be counterproductive. The additional computational overhead and hyperparameters introduced by these methods appear to outweigh any potential benefits.

\subsection{Practical Recommendations}
For practitioners, our results suggest:
\begin{itemize}
\item Simple AdamW with proper warmup and gradient clipping remains a strong baseline
\item Complex modifications require extensive validation across architectures
\item Layer-specific adaptations need careful tuning to avoid instability
\end{itemize}

\subsection{Limitations}
Several limitations of our study should be noted:
\begin{itemize}
\item Experiments were conducted on a single model architecture
\item The impact of different hyperparameter choices wasn't exhaustively explored
\item Training compute was limited to a single experimental run per configuration
\end{itemize}

\section{Conclusion}
This systematic evaluation of AdamW modifications demonstrates that many proposed optimizer enhancements fail to provide consistent improvements over the base algorithm. While our novel variants showed stable training dynamics, none surpassed the performance of carefully tuned baselines. These results suggest that fundamental advances in optimization may require approaches beyond incremental modifications to existing methods.

Future work should focus on more radical architectural innovations in optimization, potentially drawing inspiration from recent advances in adaptive methods and second-order optimization. The community may benefit from a renewed focus on understanding the fundamental optimization dynamics of transformer models rather than incremental AdamW modifications.

\bibliographystyle{unsrt}
\bibliography{references}

\end{document}
