\documentclass{article}
\usepackage[utf8]{inputenc}
\usepackage{amsmath}
\usepackage{amsfonts}
\usepackage{graphicx}
\usepackage{subcaption}

\title{Quadratic Gated Feedforward Networks: \\ Exploring Quadratic Interactions in Transformer Layers}

\author{Aardvark}

\date{\today}

\begin{document}

\maketitle

\begin{abstract}
We investigate Quadratic Gated Feedforward Networks (QGFN), a variant of Transformer feedforward layers that combines gated linear units with element-wise quadratic feature interactions. While modern architectures predominantly use gated linear units (GLUs), we explore whether carefully designed quadratic interactions could provide complementary benefits. Our method introduces a parallel quadratic pathway that interacts with the standard GLU pathway through a learned scalar mixing coefficient. On the FineWeb dataset using a Qwen-style architecture, QGFN achieved a validation loss of 4.940 compared to the SwiGLU baseline's 4.927, showing comparable but not superior performance. The quadratic pathway increased memory usage by approximately 30\% while providing limited empirical benefit in our experiments. We analyze the tradeoffs of this approach and discuss implications for future architecture design, particularly noting that the quadratic interactions may require different formulations to yield significant improvements.
\end{abstract}

\section{Introduction}
Transformer architectures rely heavily on their feedforward layers for feature transformation, with gated linear units (GLUs) and variants like SwiGLU becoming standard implementations. While these approaches are effective, recent work has questioned whether the standard formulations fully utilize the layers' capacity.

In this work, we explore augmenting standard feedforward layers with quadratic feature interactions. Our investigation focuses on whether:
\begin{itemize}
    \item Quadratic interactions can complement standard GLU computations
    \item The benefits outweigh the computational costs
    \item The approach scales effectively
\end{itemize}

We implement Quadratic Gated Feedforward Networks (QGFN) with:
\begin{itemize}
    \item A standard GLU pathway
    \item A parallel element-wise quadratic pathway
    \item A learned scalar mixing coefficient
\end{itemize}

Our experiments show this implementation provides comparable but not superior performance to SwiGLU (4.940 vs 4.927 validation loss) while increasing memory usage by ~30\%. These results suggest that while quadratic interactions may have potential, our specific implementation does not yet provide compelling advantages over existing approaches.

\section{Related Work}
Our work builds upon several research directions in neural architecture design:

\textbf{Gated Linear Units}: The original GLU formulation and subsequent variants like SwiGLU demonstrated the effectiveness of gating mechanisms in feedforward layers.

\textbf{Quadratic Interactions}: Recent work has explored quadratic operations in Transformers, including learned attention patterns and squared ReLU activations.

\textbf{Dynamic Routing}: Methods like BASE layers and Mixture-of-Experts explore adaptive computation paths, though with different formulations than our mixing approach.

Our work differs by specifically combining GLUs with element-wise quadratic operations through learned mixing, though we acknowledge this builds closely upon existing ideas in the literature.

\section{Method}
\subsection{Architecture}
QGFN processes input $x \in \mathbb{R}^d$ through:

1) \textbf{GLU Pathway}: Standard gated computation:
\begin{equation}
    \text{GLU}(x) = \text{swish}(W_g x) \odot (W_u x)
\end{equation}

2) \textbf{Quadratic Pathway}: Element-wise squared features:
\begin{equation}
    \text{Quad}(x) = (W_q x)^2
\end{equation}

The pathways combine through learned scalar $\alpha \in [0,1]$ (via sigmoid):
\begin{equation}
    \text{combined} = \alpha \cdot \text{GLU}(x) + (1-\alpha) \cdot \text{Quad}(x)
\end{equation}

Final output projects back to original dimension:
\begin{equation}
    \text{QGFN}(x) = W_d \cdot \text{combined}
\end{equation}

\subsection{Implementation Details}
We initialize $\alpha$ to 0.5 (sigmoid(0)) and use default initialization variances (0.02 for $W_q$, 0.03 for $W_g,W_u$). The total parameter count increases by approximately 0.8\% versus SwiGLU.

\section{Experimental Setup}
We evaluate on FineWeb using a Qwen-style architecture (~83M params). Both QGFN and SwiGLU baselines use:
\begin{itemize}
    \item Batch size: 512k tokens
    \item AdamW optimizer (lr=3e-4, cosine decay)
    \item 640 training steps
    \item Identical hyperparameters
\end{itemize}

Memory usage is measured during peak training. All results are from the final validation set evaluation.

\section{Results}
Key findings:
\begin{itemize}
    \item QGFN validation loss: 4.940
    \item SwiGLU baseline: 4.927
    \item Memory overhead: +30\% (42.75GB vs 31GB)
    \item Learned $\alpha$: 0.71 (mean across layers)
\end{itemize}

\begin{figure}[h]
  \centering
  \includegraphics[width=0.8\textwidth]{loss_curve.png}
  \caption{Training curves show similar convergence between QGFN (blue) and SwiGLU (orange). The quadratic interactions provide no clear advantage in final performance.}
  \label{fig:loss_curves}
\end{figure}

\section{Limitations and Discussion}
Our implementation has several key limitations:

1) \textbf{Performance}: The quadratic pathway provided no measurable improvement despite the computational cost.

2) \textbf{Memory}: The 30\% memory overhead is substantial given the lack of improvement.

3) \textbf{Formulation}: The element-wise quadratic operation may be too constrained compared to full bilinear interactions.

4) \textbf{Scale}: Experiments were limited to smaller models (83M params).

Future work could explore:
\begin{itemize}
    \item More expressive quadratic formulations
    \item Adaptive mixing mechanisms
    \item Larger-scale evaluations
\end{itemize}

\section{Conclusions}
We presented Quadratic Gated Feedforward Networks, exploring quadratic interactions in Transformer feedforward layers. While theoretically appealing, our specific implementation showed no improvement over standard approaches while increasing memory usage. This suggests that quadratic interactions may require different formulations or applications to provide benefits in language models.

\begin{thebibliography}{10}

\bibitem{vaswani2017attention}
Vaswani, Ashish, et al. 
\textit{Attention is all you need.}
NeurIPS 2017.

\bibitem{dauphin2016language}
Dauphin, Yann N., et al. 
\textit{Language modeling with gated convolutional networks.}
ICML 2016.

\bibitem{shazeer2020glu}
Shazeer, Noam. 
\textit{GLU variants improve transformer.}
arXiv:2002.05202 (2020).

\end{thebibliography}

\end{document}