Layer-Specific Adaptive Learning Rates for Transformer Optimization

View PDF

Abstract:We present LayerAdam, a modification to the Adam optimizer that applies layer-specific learning rates to different components of Transformer models. On a 134M parameter Transformer trained on FineWeb, LayerAdam achieves a 2.5\% improvement in validation loss compared to AdamW. While this improvement is modest, our results suggest that basic layer-specific adaptations can provide meaningful improvements with minimal implementation overhead.