Exascale-ready HemeLB for large-eddy simulation of full-human arterial blood flow

Abstract

Direct numerical simulation (DNS) of blood flow through the whole human arterial tree is limited by the number of lattice sites that must be advanced on a high-performance computer. We address this limit by combining the lattice Boltzmann solver HemeLB with Smagorinsky large-eddy simulation (LES). HemeLB runs with the same scaling behaviour on AMD, Intel and NVIDIA graphics processing units (GPUs) and sustains about 10 trillion lattice-site updates per second (TLUPS) on both the Frontier and Aurora exascale supercomputers. In stenotic aortic flow, LES on lattices five and ten times coarser than a 20~$μ$m DNS reproduces the wall pressure and the cross-sectional pressure and velocity, and its plane-averaged pressure differs from that of the DNS by at most 0.75~mmHg, below 1\% of the local pressure. At 100~$μ$m, LES reduces the estimated cost of a full-human simulation by a factor of about 625 relative to DNS. We use this resolution to compute blood flow from the head to the feet in a full-human arterial model with 295 million fluid sites, one inlet at the aortic root and 432 Windkessel outlets, at about 15--18 node-hours per cardiac cycle on two Frontier nodes. A single simulation resolves every region simultaneously, from the cerebral arteries to the arteries of the feet. The pressure pulse grows as it travels from the aorta to the feet, the shear stress concentrates in the branches rather than in the aorta, the pulse reaches the outlets later with increasing distance from the heart, and the flow in the legs and feet is triphasic, as in Doppler measurements of the human leg. These results bring patient-specific whole-body lattice Boltzmann simulations within reach of current supercomputers and pave the way for ensemble simulations towards a standard human model.