A High-order Backpropagation Algorithm for Neural Stochastic Differential Equation Model

Abstract

Neural stochastic differential equation model with a Brownian motion term can capture epistemic uncertainty of deep neural network from the perspective of a dynamical system. The goal of this paper is to improve the convergence rate of the sample-wise backpropagation algorithm in neural stochastic differential equation model which has been proposed in [Archibald et al., SIAM Journal on Numerical Analysis, 62 (2024), pp. 593-621]. It is necessary to emphasize that, improving the convergence order of the algorithm consisting of forward backward stochastic differential equations remains challenging, due to the loss of information of Z term in backward equations under sample-wise approximation and the limitations of the forward network form. In this paper, we develop a high-order backpropagation algorithm to improve the training accuracy. Under the convexity assumption, the result indicates that the first-order convergence is achieved when the number of training steps is proportional to the cubic number of layers. Finally, numerical examples illustrate our theoretical results.