Hessian-information geometric formulation of a class of deterministic neural network models

Abstract

In this paper a class of dynamical systems describing deterministic neural network models are formulated from a viewpoint of differential geometry. This class includes the Hopfield model and gradient systems, and is such that the so-called activation functions induce information and Hessian geometries. In this formulation, it is shown that the phase space compressibility of a dynamical system belonging to this class is written in terms of the Laplace operator defined on Hessian manifolds, where phase space compressibility is associated with a volume-form of a manifold, and expresses how such a volume-form is compressed along the vector field of a dynamical system. Since the sigmoid function, as an activation function, plays a role in the study of neural network models, such compressibility is explicitly calculated for this case. Throughout this paper, the so-called dual coordinates known in information geometry are explicitly used.