Preprint
Slaves to the Law of Large Numbers: An Asymptotic Equipartition Property for Perplexity in Generative Language Models
arXiv.org
Cornell University
05/22/2024
DOI: 10.48550/arxiv.2405.13798
Abstract
We propose a new asymptotic equipartition property for the perplexity of a large piece of text generated by a language model and present theoretical arguments for this property. Perplexity, defined as a inverse likelihood function, is widely used as a performance metric for training language models. Our main result states that the logarithmic perplexity of any large text produced by a language model must asymptotically converge to the average entropy of its token distributions. This means that language models are constrained to only produce outputs from a ``typical set", which we show, is a vanishingly small subset of all possible grammatically correct outputs. We present preliminary experimental results from an open-source language model to support our theoretical claims. This work has possible practical applications for understanding and improving ``AI detection" tools and theoretical implications for the uniqueness, predictability and creative potential of generative models.
Details
- Title: Subtitle
- Slaves to the Law of Large Numbers: An Asymptotic Equipartition Property for Perplexity in Generative Language Models
- Creators
- Raghu MudumbaiTyler Bell
- Resource Type
- Preprint
- Publication Details
- arXiv.org
- DOI
- 10.48550/arxiv.2405.13798
- eISSN
- 2331-8422
- Publisher
- Cornell University; Ithaca, New York
- Language
- English
- Date posted
- 05/22/2024
- Academic Unit
- Electrical and Computer Engineering
- Record Identifier
- 9984628216602771
Metrics
15 Record Views