Two-point residual step size gradient methods for linear systems

Abstract

Kaczmarz-type methods often converge slowly when the coefficient matrix is ill-conditioned. In particular, when the smallest singular value is small, every row of the coefficient matrix is nearly orthogonal to the corresponding right singular direction, so that the iteration stalls along that direction. To break this slow-convergence regime, we borrow the delayed-step-size idea of the Barzilai-Borwein (BB) method and, within the framework of the residual-based surrogate-hyperplane Kaczmarz method, propose two new step sizes for gradient-descent-type methods solving linear systems: a delayed residual step size \(α_k^{\mathrm{DR}}\) and a residual-difference step size \(α_k^{\mathrm{RD}}\), which correspond, respectively, to the BB1 and BB2 step sizes in convex quadratic optimization. We prove that \(α_k^{\mathrm{RD}}\leα_k^{\mathrm{DR}}\), and we analyze the convergence of the resulting method in a canonical model with two singular values: although the step sizes are non-monotone, the residual norm is shown to decrease after finitely many steps. Numerical experiments on some ill-conditioned linear systems illustrate the effectiveness of the proposed methods.