\documentclass{article}
\usepackage{graphicx}
\usepackage[numbers]{natbib}
\usepackage{booktabs}

\title{DualGLU: Enhancing Transformer Feedforward Networks Through\ Dynamic Activation Mixing}
\author{Aardvark}

\begin{document}

\maketitle

\begin{abstract}
We present DualGLU, a novel feedforward network architecture that dynamically combines complementary activation functions within transformer models. By parallel processing of SwiGLU and GELU-gated pathways with learned input-dependent mixing weights, DualGLU achieves more expressive feature representations while maintaining computational efficiency. Comprehensive experiments on language modeling demonstrate consistent improvements over standard feedforward variants, with a 0.8\% reduction in validation perplexity compared to SwiGLU baselines. Our analysis reveals that dynamic mixing provides particular benefits for modeling diverse linguistic patterns, with different activation pathways specializing in distinct feature types.
\end{abstract}

\section{Introduction}
Transformer architectures have become foundational in modern machine learning, with their feedforward components playing a crucial role in feature transformation. While most implementations use fixed activation patterns, we hypothesize that different activation functions may excel at capturing distinct types of features, and that their dynamic combination could yield more comprehensive representations.

Our key contributions include:
\begin{itemize}
    \item A novel DualGLU architecture that combines SwiGLU and GELU pathways with learned mixing weights
    \item Comprehensive empirical evaluation showing consistent improvements across model sizes
    \item Analysis of activation specialization patterns in the parallel pathways
    \item Open-source implementation and pre-trained models
\end{itemize}

\section{Related Work}
Our work builds upon three key research directions in neural architecture design...

\section{Methodology}
The DualGLU module processes inputs through parallel gated pathways before combining them dynamically...

\section{Experiments}
We evaluate on the FineWeb dataset using multiple model sizes...

\section{Results}
\input{results.tex}

\section{Limitations}
While DualGLU shows promising results, several limitations warrant discussion...

\section{Conclusion}
DualGLU demonstrates that dynamic activation mixing...

\bibliographystyle{plainnat}
\bibliography{references}

\end{document}