Self-organized annealing in laterally inhibited neural networks shows power law decay

Abstract

In this paper we present a method which assigns to each layer of a multilayer neural network, whose network dynamics is governed by a noisy winner-take-all mechanism, a neural temperature. This neural temperature is obtained by a least mean square fit of the probability distribution of the noisy winner-take-all mechanism to the distribution of a softmax mechanism, which has a well defined temperature as free parameter. We call this approximated temperature resulting from the optimization step the neural temperature. We apply this method to a multilayer neural network during learning the XOR-problem with a Hebb-like learning rule and show that after a transient the neural temperature decreases in each layer according to a power law. This indicates a self-organized annealing behavior induced by the learning rule itself instead of an external adjustment of a control parameter as in physically motivated optimization methods e.g. simulated annealing.