A $\mathcal{VU}$-calculus for composite functions and the $\mathcal{U}$-Hessian of partly smooth functions

Abstract

We assemble a finite-dimensional \(\mathcal{VU}\)-calculus for composite nonsmooth functions whose outer function is convex: the chain rule, separable and convex sums, a strictly differentiable perturbation of a lower semicontinuous (lsc) term, the model \(δ_X+f_0+θ\circ F\), and a finite maximum of \(C^1\) functions. The same algebra yields an \(\varepsilon\)-\(\mathcal{VU}\) chain rule for a proper outer approximation of the subdifferential. On the set \(\mathcal{R}_{h,F}\) of points at which the convex chain rule holds, the subspaces \(\mathcal{V}f\) and \(\mathcal{U}f\), an orthonormal frame of \(\mathcal{U}f\), and the \(\mathcal{U}\)-gradient \(\bar g_u\) are written in terms of the factors. If \(f\) is \(C^1\)-partly smooth at \(\bar{x}\), that gradient is \(\bar g_u=U_f^\top\nabla_{\mathcal{M}}f(\bar{x})\). If \(f\) is \(C^2\)-partly smooth and \(0\in\ri\partial f(\bar{x})\), the convex \(\mathcal{U}\)-Hessian \(H_U\) (when \(f\) is convex) or the local matrix \(H_\varepsilon\) (when \(f\) is prox-regular at \(\bar{x}\) for \(0\)) equals the Gram matrix \(U_f^\top\nabla^2_{\mathcal{M}}f(\bar{x})\,U_f\); the same matrix, written along the active manifold in a continuous frame, depends continuously on the base point. If in addition \(f\) is \(C^2\)-partly smooth at \(\bar{x}\), then under prox-regularity and subdifferential continuity at \(\bar{x}\) for \(0\), \(H_\varepsilon\succ 0\) is equivalent to tilt stability of \(\bar{x}\) and to strong metric regularity of \(\partial f\) at \((\bar{x},0)\), with \(\lip\bigl((\partial f)^{-1}\bigr)(0\mid\bar{x})=\|H_\varepsilon^{-1}\|\). The calculus and the test are illustrated on a hinge composite and two elementary tests of \(H_U\succ 0\).