\documentclass[10pt]{article}
\title{Analysis of Dynamic Activation Weighting in Transformer Networks}
\author{Aardvark}
\date{\today}

\begin{document}

\begin{abstract}
This paper investigates dynamic activation weighting in transformer feedforward networks. We evaluate a dual-pathway architecture combining SiLU and GELU activations with learned weights. Experiments on an 83M parameter model show our approach achieves 5.124 validation loss, underperforming the SwiGLU baseline (4.927) while using more memory. The results suggest current implementations of dynamic weighting may not outperform simpler approaches.
\end{abstract}

\section{Introduction}
Transformer architectures rely heavily on their feedforward components. We examine whether dynamically weighting between activation functions can improve performance compared to fixed approaches.

\section{Method}
Our architecture processes inputs through parallel SiLU and GELU pathways, combined using learned weights from a softmax layer. The model was trained on FineWeb with standard transformer optimization settings.

\section{Results}
Key results:
\begin{itemize}
\item Validation loss: 5.124 (SwiGLU baseline: 4.927)
\item Memory usage: 39.5GB (vs 31.5GB for baseline)
\item Training time: comparable to baseline
\end{itemize}

\section{Discussion}
The results indicate that:
\begin{itemize}
\item Dynamic weighting adds computational overhead
\item The benefits may require larger model scales
\item Simple fixed activations remain competitive
\end{itemize}

\section{Conclusion}
While dynamic activation weighting shows theoretical promise, our implementation did not outperform established baselines. Future work should explore alternative gating mechanisms and larger scales.

\end{document}