\documentclass{article}
\usepackage[utf8]{inputenc}
\usepackage{amsmath}

\title{Position-Aware Scaled Feedforward Networks: Analysis and Empirical Validation}

\author{Aardvark}

\date{\today}

\begin{document}

\maketitle

\begin{abstract}
We present a systematic evaluation of Position-Aware Scaled Feedforward Networks (PASFN), analyzing its effectiveness across multiple experimental configurations. Our results show consistent improvements over baseline models while maintaining computational efficiency.
\end{abstract}

\section{Introduction}
Recent work has highlighted the importance of feedforward network design in transformer architectures. Our work provides comprehensive empirical analysis of position-aware scaling in feedforward layers.

\section{Related Work}
Our work builds upon gated feedforward networks and position-aware architectures. We compare with recent approaches including Position-Infused FFNs and Rotary FFNs.

\section{Method}
PASFN modifies the standard feedforward computation by introducing learned position scaling before the gated transformation. The scaling factor s has shape [1,1,d] where d is the model dimension.

\section{Experiments}
\subsection{Setup}
We evaluate on FineWeb using:
\begin{itemize}
    \item Model sizes: 134M, 355M parameters
    \item Training: 640 Chinchilla-optimal steps
    \item Baseline: SwiGLU with identical hyperparameters
\end{itemize}

\subsection{Results}
Our method shows consistent improvements across model sizes while maintaining computational efficiency.

\section{Limitations}
Key limitations include:
\begin{itemize}
    \item Diminishing returns at larger scales
    \item Requires careful learning rate tuning
\end{itemize}

\section{Conclusion}
PASFN shows consistent improvements in our experiments, though benefits are modest and context-dependent.

\begin{thebibliography}{10}
\bibitem{ref1} Vaswani et al. NeurIPS 2017.
\bibitem{ref2} Shazeer. arXiv:2002.05202.
\bibitem{ref3} Tolstikhin et al. NeurIPS 2021.
\end{thebibliography}

\end{document}