Thermodynamics of Forgetting in a Bounded Hopfield Network

Abstract

A memory that keeps learning must also forget. The Hopfield network does not: beyond a critical number of stored patterns it falls into a state of total confusion and retrieves none of them. Parisi showed in 1986 that bounding the synaptic strength cures this pathology, since the clipping of the couplings gradually erases the oldest memories and leaves room for the new ones. In this paper we provide a thermodynamic description of the resulting network, known as the Hopfield-Parisi model. Starting from the learning rule, we show that the network is equivalent to a Hopfield model whose patterns are weighted according to their age, superimposed on a Sherrington-Kirkpatrick spin glass, both governed by the synaptic bound: a larger bound makes the memory longer, but also noisier. The theory of nonlinear networks developed in the eighties already provides, through the heuristic replica method, the equations that govern this model at any temperature, but to the best of our knowledge they have been solved only at zero temperature. We rederive them at the replica symmetric level through Guerra's interpolation and solve them at finite temperature, obtaining the phase diagram of the model together with its memory horizon, namely the largest age at which a pattern can still be retrieved. The solution shows that forgetting acts on the signal alone, the noise being the same for patterns of every age, and that the horizon is maximal at an intermediate bound which is essentially insensitive to temperature. Monte Carlo simulations of networks trained with the original learning rule support these predictions.