A New Perspective of Accelerated Gradient Methods: The Controlled Invariant Manifold Approach

Abstract

Gradient Descent (GD) is a ubiquitous algorithm for finding the optimal solution to an optimization problem. For reduced computational complexity, the optimal solution $\mathrm{x^*}$ of the optimization problem must be attained in a minimum number of iterations. For this objective, the paper proposes a genesis of an accelerated gradient algorithm through the controlled dynamical system perspective. The objective of optimally reaching the optimal solution $\mathrm{x^*}$ where $\mathrm{\nabla f(x^*)=0}$ with a given initial condition $\mathrm{x(0)}$ is achieved through control.