Random Gradient-Free Optimization in Infinite Dimensional Spaces

Abstract

We propose a new gradient-free method for infinite-dimensional optimization in Hilbert spaces that requires only the computation of directional derivatives. Though functional optimization is often solved through finite-dimensional gradient descent over a parametrization, such as neural networks, we instead propose to leverage the functional nature of the optimization problem to enable provable guarantees. However, infinite-dimensional gradients are often hard to compute in practice, rendering naïve functional gradient descent intractable. To overcome this limitation, our framework leverages only directional derivatives and a pre-basis for the Hilbert space, i.e., a linearly independent set whose span is dense. This resolves the tractability issue, as pre-bases are much more accessible than full orthonormal bases or reproducing kernels -- which may not even exist -- and individual directional derivatives can be computed using automatic differentiation. We showcase the use of our method to solve partial differential equations à la physics-informed neural networks (PINNs), where it effectively enables provable convergence.