Beyond storage capacity in a single model neuron: Continuous replica symmetry breaking

Abstract

A single McCulloch-Pitts neuron, that is, the simple perceptron is studied, with focus on the region beyond storage capacity. It is shown that Parisi's hierarchical ansatz for the overlap matrix of the synaptic couplings with so called continuous replica symmetry breaking is a solution, and as we propose it is the exact one, to the equilibrium problem. We describe some of the most salient features of the theory and give results about the low temperature region. In particular, the basics of the Parisi technique and the way to calculate thermodynamical expectation values is explained. We have numerically extremized the replica free energy functional for some parameter settings, and thus obtained the order parameter function, i.e., the probability distribution of overlaps. That enabled us to evaluate the probability density of the local stability parameter. We also performed a simulation and found a local stability density closer to the theoretical curve than previous numerical results were.