Learning Curves for Mutual Information Maximization
Robert Urbanczik
Abstract
An unsupervised learning procedure based on maximizing the mutual information between the outputs of two networks receiving different but statistically dependent inputs is analyzed (Becker and Hinton, Nature, 355, 92, 161). For a generic data model, I show that in the large sample limit the structure in the data is recognized by mutual information maximization. For a more restricted model, where the networks are similar to perceptrons, I calculate the learning curves for zero-temperature Gibbs learning. These show that convergence can be rather slow, and a way of regularizing the procedure is considered.
Create a lesson
Related papers
Attention over noisy examples: generalization and memorization in the exponential Hopfield network
Francesco Alemanno
Finite-size geometric crossover toward fragmentation of the potential energy shell in the quantum random energy model
Ishita Modak, Vatsana Tiwari, Soumya Bera
Testing Replica Symmetry Breaking in the three-dimensional Edwards-Anderson model by using vanishing fields
Miguel Aguilar-Janita, Silvio Franz, Victor Martin-Mayor et al.
Deep Learning of Phase Transitions for Bilinear-Biquadratic Spin-1 Chains from Correlation Aspects
Ming-Chiang Chung, Yu-Ting Kuo, Ian McCulloch et al.
Resolving Structural Uncertainty in Stochastic Dynamical Systems via Ensemble-Based Bayesian Inference
Francesco Zambelli, Valeria D'Andrea, Manlio De Domenico
Seismology of Glasses
Zhiyu Cao, Peter G. Wolynes