An Adaptive Mixed Precision and Dynamically Scaled Preconditioned Conjugate Gradient Algorithm

Abstract

We propose an adaptive mixed precision and dynamically scaled preconditioned conjugate gradient algorithm (AMP-PCG). It dynamically adjusts the precision for storing vectors and computing, exploiting low precision when appropriate, while maintaining a convergence rate and accuracy comparable to that of double precision PCG. Our mixed precision strategy consists of three main components: (1) The residual and matrix-vector product are initially computed in double precision, and the algorithm switches these to single precision based on the chosen convergence tolerance and an estimate of the residual gap. (2) Depending on the eigenvalue distribution, the preconditioned residual and search direction are either in half precision throughout the iterations or initially in double precision and then stepwise reduced to single and half precision. (3) A dynamically scaled residual is used at every iteration to mitigate underflow in half precision. We provide theoretical support for our estimates and we demonstrate the effectiveness of AMP-PCG through numerical experiments, highlighting both its robustness and the significant performance gains (1.63x speedup) achieved compared to double precision PCG on a GPU.