Statistical mechanics of lossy compression using multilayer perceptrons
Kazushi Mimura, Masato Okada
Abstract
Statistical mechanics is applied to lossy compression using multilayer perceptrons for unbiased Boolean messages. We utilize a tree-like committee machine (committee tree) and tree-like parity machine (parity tree) whose transfer functions are monotonic. For compression using committee tree, a lower bound of achievable distortion becomes small as the number of hidden units K increases. However, it cannot reach the Shannon bound even where K -> infty. For a compression using a parity tree with K >= 2 hidden units, the rate distortion function, which is known as the theoretical limit for compression, is derived where the code length becomes infinity.
Create a lesson
Related papers
A Human-AI Theorem Connecting Spontaneous and Field-Induced Mechanisms of Collective Behavior in One Dimension
Weiguo Yin
Exact joint eigenvalue densities of non-Hermitian random matrices are Calogero scattering states
Zhenyu Xiao, Ze Chen, Yifei Liu et al.
Duality between the level statistics of Hermitian and non-Hermitian random matrices
Ze Chen, Zhenyu Xiao, Yifei Liu et al.
Nonlinear Fluctuating Hydrodynamics from Interacting Noisy Quantum Matter
Alexios Christopoulos, João Costa, Stefano Scopa et al.
Bayesian Tracking of a Diffusing Target in Two and Three Dimensions
Ewan McCulloch, Adam Nahum
Overcoming critical slowing down in frustrated spin systems by learned multiscale sampling
Gabriele Bandini, Giulio Biroli, Patrick Charbonneau et al.