Graph-Based Testing of Conditional Mean Independence with Applications to Large Language Model Evaluation

Abstract

We study whether a predictor $X$ provides additional information for predicting a scalar response $Y$ after adjustment for other variables $Z$, formalized by testing $\mathsf{H}_0:\E(Y\mid X,Z)=\E(Y\mid Z)$. Existing methods often require estimating nuisance functions involving $X$, which can be difficult for high-dimensional or structured predictors. We propose a graph-based test that extends the edge count principle to covariate adjustment. It represents $X$ through pairwise dissimilarities, estimates only the baseline regression $\E(Y\mid Z)$, and aggregates products of held-out residuals across graph edges. We develop sample-splitting and two-fold cross-fitting versions. Under regularity conditions, both have standard normal null limits, yielding analytic $p$-values, and are consistent under graph-smooth and graph-rough alternatives. Simulations with vector, image, and network predictors show good size control and power advantages over competing procedures, with cross-fitting improving power across the main alternatives. Applications examine human quality ratings, mathematical error labels, and LLM response preferences. Our method detects associations with quality ratings and an LLM judge's preferences after adjustment for text length and other recorded characteristics.