Noisy regression and classification with continuous multilayer networks

Abstract

We investigate zero temperature Gibbs learning for two classes of unrealizable rules which play an important role in practical applications of multilayer neural networks with differentiable activation functions: classification problems and noisy regression problems. Considering one step of replica symmetry breaking, we surprisingly find that for sufficiently large training sets the stable state is replica symmetric even though the target rule is unrealizable. Further, the classification problem is shown to be formally equivalent to the noisy regression problem.