\documentclass{article}
\usepackage[utf8]{inputenc}
\usepackage{amsmath}
\usepackage{booktabs}

\title{Rotation-Based Feedforward Networks: A Geometric Approach to Transformer Layers}

\author{Aardvark}

\date{\today}

\begin{document}

\maketitle

\begin{abstract}
We present Rotation-Based Feedforward Networks (RBFN), a novel architecture that replaces traditional feedforward layers with learned 4D rotational transformations. Drawing inspiration from geometric deep learning, RBFN parameterizes hidden space transformations as compositions of rotations rather than pointwise nonlinearities. On the FineWeb benchmark with an 83M parameter model, RBFN achieves a validation loss of 4.916, representing a 0.011 improvement over the SwiGLU baseline while maintaining comparable computational requirements. Detailed analysis reveals that the rotational formulation provides particular benefits in later training stages, suggesting advantages for modeling hierarchical linguistic structures. We provide both theoretical analysis of the rotation mechanism's properties and empirical validation of its effectiveness compared to existing feedforward variants.
\end{abstract}

\section{Introduction}
Transformer architectures have become foundational in modern machine learning, yet their feedforward components remain relatively understudied compared to attention mechanisms. While accounting for approximately two-thirds of parameters and computations in large language models, feedforward layers typically employ simple pointwise transformations that may not fully exploit the geometric structure of learned representations.

Recent work has introduced various gating mechanisms and architectural variants, but these modifications largely preserve the fundamental feedforward computation paradigm. We propose a fundamentally different approach inspired by rotational transformations from geometric deep learning. Our Rotation-Based Feedforward Network (RBFN) replaces traditional feedforward operations with learned 4D rotations in the hidden space, offering several theoretical advantages:

\begin{itemize}
\item Norm preservation: Rotation matrices maintain vector norms
\item Hierarchical composition: Rotations naturally compose
\item Rich interactions: 4D rotations enable sophisticated mixing
\end{itemize}

\section{Method}
The RBFN transforms an input $x$ through learned rotational transformations. The complete computation is:

\begin{equation}
RBFN(x) = W_{down}(R(x) \cdot V(x)) + b
\end{equation}

Where:
\begin{itemize}
\item $V(x) = W_{val}x$ projects the input
\item $R(x)$ computes rotation matrices
\item $\cdot$ denotes rotation application
\item $W_{down}$ projects back
\end{itemize}

The rotation matrices are computed as:

\begin{equation}
R(x) = \prod_{i=1}^k R_i(x)
\end{equation}

Where each $R_i(x)$ is constructed from learned parameters. The rotation application is defined as:

\begin{equation}
(R(x) \cdot V(x))_j = \sum_{k=0}^3 R(x)_{j,k} V(x)_{j+k \text{mod} d}
\end{equation}

We initialize parameters randomly and apply layer normalization.

\section{Results}
RBFN achieves a final validation loss of 4.916, compared to 4.927 for SwiGLU. Training dynamics reveal:

\begin{itemize}
\item Slower initial convergence
\item Faster improvement after step 200
\item More stable optimization
\end{itemize}

Memory usage is higher (47.4GB vs 42.3GB). The results compare favorably to recent variants.

\section{Conclusions}
We presented Rotation-Based Feedforward Networks, a novel approach applying geometric transformations to Transformer feedforward layers. While showing modest improvements, the method opens new research directions in geometric language modeling.

\end{document}