Realized versus Population Reliability: Anytime-Valid Self-Consistency Certification under Correlated Reasoning
Terence D. Agbeyegbe, Toneukarin G. Agbeyegbe
Abstract
Self-consistency certification treats repeated sampling from a language model as sequential evidence accumulation and stops when an anytime-valid e-process crosses a threshold, guaranteeing finite-sample Type I error control by Ville's inequality. Prior constructions assume sampled rollouts are conditionally independent. We show this assumption fails in practice: reasoning streams are positively correlated through shared reasoning prefixes or across-prompt heterogeneity, changing not merely certificate efficiency but the hypothesis certified. Modeling dependence via a per-unit latent state (the random-effects / intraclass-correlation structure) induces exchangeable, equi-correlated increments with correlation ρ. Correlation splits the single null into a realized null about the drawn belief and a population null about average model behavior, which coincide only when ρ=0. We show: (i) the standard betting e-process remains anytime-valid for the realized null, not the population null; (ii) read as a population-null test, a suitable mixture has asymptotic crossing probability at least π=P(μ(Z)>0), potentially exceeding nominal ε; (iii) the within-unit mean's effective sample size saturates at 1/ρ, so extra within-unit rollouts cannot restore population validity; and (iv) a prompt-level e-process restores population validity, with optimal allocation at one rollout per prompt under a pure rollout budget. Simulations and Claude Haiku 4.5 rollouts confirm this regime dependence: on GSM8K (ρ=0.435, θ=+0.96) the gap is dormant, whereas on AIME (ρ=0.480, θ=-0.15) the single-unit rule crosses on 24% of prompts at nominal ε=0.05, while the prompt-level procedure declines (0.00). This correctness result identifies which reliability claim an anytime-valid certificate actually supports under dependence.
Create a lesson
Related papers
Additive Fréchet Regression for Random Objects
Changwon Choi, Wookyeong Song, Hans-Georg Müller et al.
Spatiotemporal Risk-Based Hotspot Dynamics Assessment
José M. Angulo, Ana E. Madrid
Optimal Transport Rank Regression for Robust Multivariate-Response Regression
Yizhe Ding, Yifan Jiang, Runze Li et al.
Statistical Inference with Ill-Conditioned Random Weight Neural Network Sieves
Giovanni Ballarin, Lukas Gonon, Lyudmila Grigoryeva
Causal Effect Estimation Under Uncertain Statistical Surrogacy
Yi Liu, Alexander W. Levis, Shu Yang et al.
Elementwise Precision-weighted Compositional Data Analysis by Weighted Alternating Least Squares
Jan Graffelman