Why Compression Is Prediction: The Information-Theoretic View of LLMs
Compression is prediction, and prediction is compression: these are two sides of the same information-theoretic coin. Any model that accurately predicts the next token can be turned into a lossless data compressor, and any compressor can be turned into a generative model. This guide explains why that equivalence holds, how DeepMind and Meta proved it in 2023, and why it reframes how you should think about large language models. What Does It Mean That Compression Is Prediction? Compression is prediction because a good predictor implicitly assigns low code lengths to likely sequences, which is exactly what a compressor does. When you predict “the” follows “of the”, you are acting like an entropy coder that gives “the” a short code because it is probable. The tighter your predictions match the real distribution of data, the fewer bits you need to store that data. ...