Excess Separability: Nuisance-Controlled Residual-Stream Probing for Benchmark Contamination Detection

Abstract

Benchmark contamination is diagnosed with n-gram overlap, likelihood-based membership inference, or canary strings, and each needs something usually unavailable: the training corpus, a well-chosen test statistic, or foresight at release. A recent alternative reads it off a linear probe on internal activations. We show the natural way to do this does not work, specify one that survives measurement, then find that the correction making it work carries more variance than the null it is tested against. The protocol reports a zero-sum contrast on the depth profile of probe accuracy, recentred on a level-matched placebo baseline, tested against a label-permutation null, with the reference set twice the size of the suspect set. Each choice replaces a simpler alternative we rejected on measurement. Reporting the level of excess separability rather than its shape makes the false positive rate track the size of the analyst's own control set, 0.03 to 0.99 under a true null. Contrasting against a flat depth profile rejects a true null 0.72 of the time when surface decodability rises with depth, and loses all power when it falls. On real transformers the protocol fails a test the simulations did not pose. The recentring subtracts an estimate, and the permutation null holds it fixed. Re-estimated across split seeds on four audits of contaminated checkpoints, its standard deviation is 1.30 to 1.56 times the null's own in every arm: what is subtracted to remove a bias is more variable than what it corrects. The one nominally significant result, p = 0.0075, becomes 0.0745 once that variance is propagated, and no verdict is issued. The simulations missed this because their surface key is the covariate driving item variation; on real text it is a proxy, and degrading key quality in simulation reproduces it. We add a companion measurement and a widened null. No arm shows contamination.