Shannon Information and Kolmogorov Complexity
Peter Grunwald, Paul Vitanyi
Abstract
We compare the elementary theories of Shannon information and Kolmogorov complexity, the extent to which they have a common purpose, and where they are fundamentally different. We discuss and relate the basic notions of both theories: Shannon entropy versus Kolmogorov complexity, the relation of both to universal coding, Shannon mutual information versus Kolmogorov (`algorithmic') mutual information, probabilistic sufficient statistic versus algorithmic sufficient statistic (related to lossy compression in the Shannon theory versus meaningful information in the Kolmogorov theory), and rate distortion theory versus Kolmogorov's structure function. Part of the material has appeared in print before, scattered through various publications, but this is the first comprehensive systematic comparison. The last mentioned relations are new.
Create a lesson
Related papers
Galois Hulls of Generalized Roth-Lempel Codes and Their Applications to EAQECCs
Xuefei Wu, Qi Liu, Yingchun Chen et al.
Maximum Entropy Probability Distributions on Spheres with Fixed Mean Busemann Function and Holomorphic-Information-Geometric Model of Cognition
Vladimir Jacimovic
Norm-One Torus Decompositions and Decoding of Gashkov-Sidel'nikov Codes
Minjia Shi, Shitao Li, Yuhong Xia et al.
A Mirror Vanishing Band for Weight Distributions of Binary Linear Codes
Xianmang He
Tri-Hybrid Beamforming Design for Large-Scale MIMO ISAC Systems
Tianyu Fang, Mengyuan Ma, Markku Juntti et al.
Equivalence Between Nested Gibbs Measures and Log-Linear Combinations of Gibbs Measures
Yaiza Bermudez, Samir M. Perlaza, Iñaki Esnaola