Brain-inspired, interpretable, resonant recurrent neural networks

Abstract

Traditional artificial neural networks consist of nodes with non-oscillatory dynamics. Biological neural networks, on the other hand, consist of oscillatory components embedded in an oscillatory environment. Motivated by this feature of biological neurons, we describe a neural network framework with explicit damped, oscillatory node dynamics. We express the oscillatory dynamics using two history dependent terms to connect these dynamics with standard recurrent neural network formulations, apply physical constraints from observed brain dynamics to choose the oscillatory frequencies, and stationary constraints to reduce the number of free parameters. We then optimize and illustrate network performance by classifying hand-written digits and simulated neuronal spike train activity and show that these oscillatory network elements support accurate classification with few trainable parameters. Choosing oscillator frequencies according to a proposed theory for brain rhythms improves classification accuracy compared to alternative frequency configurations and compared to standard recurrent neural network frameworks with comparable numbers of parameters. Compared to existing approaches, the proposed resonant recurrent network (RRN) utilizes oscillatory dynamics expressed as a straightforward extension of standard recurrent neural networks, produces interpretable features for classification, and performs well with few parameters when oscillator frequencies follow a configuration observed in vivo. We propose that RRNs may serve as efficient, biologically inspired building blocks to achieve complex goals in biological and artificial neural networks.