Machine Learning Closure Audits for LSST Photometric Supernova Cosmology

Abstract

Modern and next generation supernova cosmology analyses rely on end to end simulations to train photometric classifiers, characterise selection effects, validate light curve models, and calibrate distance bias corrections. Standard closure tests based on global Hubble diagram summaries can miss multivariate structure that remains in post correction residuals. We introduce a supervised machine learning closure audit that tests whether measured observables can predict the bias corrected Hubble residuals $Δμ$. We apply the audit to LSST Type Ia supernova simulations from two independent analyses: the M23 mock data sets of Mitra et al. (2023), with spectroscopic redshift and photometric redshift samples, and the predominantly photometric LSST like M25 simulation of Mitra et al. (2025). Standard one dimensional Redshift binned diagnostics explain <1% of the residual variance ($R^2 < 0.01$), suggesting apparent closure. In contrast, out of fold LightGBM models recover up to 98.2% of the variance in the simulated residuals ($R^2 = 0.982$), revealing structured residual predictability. Applying the same audit directly to the real Dark Energy Survey 5 Year (DES 5YR) spectroscopic sample yields a held out $R^2 = 0.725$. SHAP feature attribution rankings are highly consistent between independently trained M25 and DES models (Spearman $ρ= 0.802$), with apparent magnitudes, signal to noise ratios, and redshift dominating the shared predictive hierarchy. The resulting scorecard provides a diagnostic protocol for comparing mock ensembles and real observations, identifying residual non closure before cosmological parameters are unblinded.