Preprint
As Language Models Scale, Low-order Linear Depth Dynamics Emerge
ArXiv.org
Cornell University
03/13/2026
DOI: 10.48550/arxiv.2603.12541
Abstract
Large language models are often viewed as high-dimensional nonlinear systems and treated as black boxes. Here, we show that transformer depth dynamics admit accurate low-order linear surrogates within context. Across tasks including toxicity, irony, hate speech and sentiment, a 32-dimensional linear surrogate reproduces the layerwise sensitivity profile of GPT-2-large with near-perfect agreement, capturing how the final output shifts under additive injections at each layer. We then uncover a surprising scaling principle: for a fixed-order linear surrogate, agreement with the full model improves monotonically with model size across the GPT-2 family. This linear surrogate also enables principled multi-layer interventions that require less energy than standard heuristic schedules when applied to the full model. Together, our results reveal that as language models scale, low-order linear depth dynamics emerge within contexts, offering a systems-theoretic foundation for analyzing and controlling them.
Details
- Title: Subtitle
- As Language Models Scale, Low-order Linear Depth Dynamics Emerge
- Creators
- Buddhika Nettasinghe - University of IowaGeethu Joseph - Delft University of Technology
- Resource Type
- Preprint
- Publication Details
- ArXiv.org
- DOI
- 10.48550/arxiv.2603.12541
- ISSN
- 2331-8422
- Publisher
- Cornell University; Ithaca, New York
- Language
- English
- Date posted
- 03/13/2026
- Academic Unit
- Business Analytics
- Record Identifier
- 9985147089002771
Metrics
1 Record Views