Some convergent results for Backtracking Gradient Descent method on Banach spaces
Abstract
Our main result concerns the following condition: Condition C. Let X be a Banach space. A C1 function f:X→ R satisfies Condition C if whenever \xn\ weakly converges to x and n→∞||∇ f(xn)||=0, then ∇ f(x)=0. We assume that there is given a canonical isomorphism between X and its dual X*, for example when X is a Hilbert space. Theorem. Let X be a reflexive, complete Banach space and f:X→ R be a C2 function which satisfies Condition C. Moreover, we assume that for every bounded set S⊂ X, then x∈ S||∇ 2f(x)||<∞. We choose a random point x0∈ X and construct by the Local Backtracking GD procedure (which depends on 3 hyper-parameters α ,β ,δ 0, see later for details) the sequence xn+1=xn-δ (xn)∇ f(xn). Then we have: 1) Every cluster point of \xn\, in the weak topology, is a critical point of f. 2) Either n→∞f(xn)=-∞ or n→∞||xn+1-xn||=0. 3) Here we work with the weak topology. Let C be the set of critical points of f. Assume that C has a bounded component A. Let B be the set of cluster points of \xn\. If B A= , then B⊂ A and B is connected. 4) Assume that X is separable. Then for generic choices of α ,β ,δ 0 and the initial point x0, if the sequence \xn\ converges - in the weak topology, then the limit point cannot be a saddle point.