Approaching nonsmooth nonconvex optimization problems through first order dynamical systems with hidden acceleration and Hessian driven damping terms

Abstract

In this paper we carry out an asymptotic analysis of the proximal-gradient dynamical system \begin{equation*}\left\{ \begin{array}{ll} \dot x(t) +x(t) = \prox_{\gamma f}\big[x(t)-\gamma\nabla\Phi(x(t))-ax(t)-by(t)\big],\\ \dot y(t)+ax(t)+by(t)=0 \end{array}\right.\end{equation*} where $f$ is a proper, convex and lower semicontinuous function, $\Phi$ a possibly nonconvex smooth function and $\gamma, a$ and $b$ are positive real numbers. We show that the generated trajectories approach the set of critical points of $f+\Phi$, here understood as zeros of its limiting subdifferential, under the premise that a regularization of this sum function satisfies the Kurdyka-\L{}ojasiewicz property. We also establish convergence rates for the trajectories, formulated in terms of the \L{}ojasiewicz exponent of the considered regularization function.