General transformation neural networks: A class of parametrized functions for high-dimensional function approximation

Abstract

We propose a novel class of neural network-like parametrized functions, i.e., general transformation neural networks (GTNNs), for high-dimensional approximation. Conventional deep neural networks sometimes perform less accurately on learning problems trained with gradient descent, especially when the target function is oscillatory. To improve accuracy, we generalize the neuron's affine transformation to a broader class of functions that can capture complex shapes and offer greater capacity. Specifically, we discuss three types of GTNNs in detail: the cubic, quadratic and trigonometric transformation neural networks (CTNNs, QTNNs and TTNNs). We perform an approximation error analysis of GTNNs, presenting their universal approximation properties for continuous functions, error bounds for Barron-type functions and error bounds of deep architectures. Several numerical examples of regression problems are presented, demonstrating that CTNNs/QTNNs/TTNNs achieve higher accuracy than conventional fully connected neural networks.