\documentclass{article}
\title{Polynomial Activations in Transformer Networks}
\author{Aardvark}
\date{\today}

\begin{document}

\maketitle

\begin{abstract}
We evaluate quadratic polynomial activations as alternatives to gated mechanisms in transformers. Our experiments show polynomial activations achieve a validation loss of 4.891, improving upon the SwiGLU baseline (4.9266). The results demonstrate polynomial activations can provide comparable performance with simpler implementations.
\end{abstract}

\section{Introduction}
Transformer architectures rely heavily on their feedforward sub-layers. While gated units are standard, we examine polynomial activation functions as simpler alternatives. Our work provides:
\begin{itemize}
\item Evaluation of quadratic polynomial activations
\item Comparison of configurations
\item Analysis of performance tradeoffs
\end{itemize}

\section{Method}
We use quadratic polynomial activations:
\[ f(x) = a_0 + a_1x + a_2x^2 \]
Initialized with:
\begin{itemize}
\item $a_0 = 0.0$
\item $a_1 = 1.0$
\item $a_2 \sim N(0, 0.1)$
\end{itemize}

\section{Results}
Our experiments show:
\begin{center}
\begin{tabular}{|l|l|}
\hline
Method & Validation Loss \\ \hline
Polynomial (Ours) & 4.891 \\ 
SwiGLU Baseline & 4.9266 \\ \hline
\end{tabular}
\end{center}

\section{Conclusion}
Polynomial activations provide modest improvements over SwiGLU with simpler implementation. While not state-of-the-art, they remain an attractive option for resource-constrained applications.

\end{document}