Logo image
Back
TransformerFAM: Feedback attention is working memory
Preprint   Open access

TransformerFAM: Feedback attention is working memory

Dongseong Hwang, Weiran Wang, Zhuoyuan Huo, Khe Chai Sim and Pedro Moreno Mengibar
arXiv (Cornell University), p.26 pages
05/07/2024
DOI: 10.48550/arxiv.2404.09173
url
https://doi.org/10.48550/arXiv.2404.09173View
Preprint (Author's original)This preprint has not been evaluated by subject experts through peer review. Preprints may undergo extensive changes and/or become peer-reviewed journal articles. Open Access

Abstract

While Transformers have revolutionized deep learning, their quadratic attention complexity hinders their ability to process infinitely long inputs. We propose Feedback Attention Memory (FAM), a novel Transformer architecture that leverages a feedback loop to enable the network to attend to its own latent representations. This design fosters the emergence of working memory within the Transformer, allowing it to process indefinitely long sequences. TransformerFAM requires no additional weights, enabling seamless integration with pre-trained models. Our experiments show that TransformerFAM significantly improves Transformer performance on long-context tasks across various model sizes (1B, 8B, and 24B). These results showcase the potential to empower Large Language Models (LLMs) to process sequences of unlimited length.
 Computer Science - Artificial Intelligence Computer Science - Computation and Language Computer Science - Learning

Details

Metrics

9 Record Views
Logo image