Dead text or binding clause? Measuring and restoring constraint influence in black-box LLM dialogues
Haoyuan Zhu
Abstract
Multi-turn dialogues let users revoke constraints as easily as impose them, but revocation does not reliably take effect: models keep enacting withdrawn requirements (occasionally beneath comments asserting their removal), a failure we call behavioral relapse, or revocation inertia. No existing instrument measures this influence per clause, predicts it before delivery, or repairs it under matched budgets. closes the three gaps through the model API alone: a contract ledger pairs every constraint with an executable checker, records revocations as tombstones, and compiles the net constraint state ahead of time into a single specification; a sequential ablation probe measures per-clause adherence and incremental behavioral effect; a repair ladder operates under token- and attempt-matched budgets. On ( HumanEval tasks, verified checkers), relapse at an 8B operating point climbs from to as constraint load grows, while stronger models sit at floor. Under matched checkers, model, and budget, ahead-of-time compilation significantly reduces relapse against a no-ledger verifier-retry baseline (, 95\% CI , p ); adaptive ladder interventions stacked on top add no detectable gain (95\% confidence excludes gains ≥ ). The probe predicts relapse before delivery (AUROC ); a one-sentence tombstone note recovers about a third of the compilation effect and survives a placebo control. At delivery overhead and of API compute for every result, revocation failure becomes a measurable, predictable, and repairable property of dialogue state rather than an invisible one.
Create a lesson
Related papers
MUSE: Benchmarking Large Vision-Language Models on Multi-Modal Understanding in Situated Education
Luyao Zhu, Xun Wei Yee, Wei Li et al.
Infinite-Parameter LLMs: Generating and Adapting Weights from Live Data
Jinli Hu, Ross M. Clarke, Yichuan Zhang et al.
Compositional Policy Violations: When Step-Level Compliance Fails In Agentic AI Workflows
Ashwini Kurady, Sri Sai Charith Grandhi, Rajesh Gupta et al.
CERA-MoA: Co-Evolving Routing Mechanisms with Continually Learning LLM Agents
Jiaxuan Jiang, Liyuan He, Zhixuan Fang
Version- and Scope-Aware Question Answering over Normative Documents: A Deployed System and an End-to-End Evaluation at Production Scale
Liuyin Wang, Shuaipeng Jin, Jiwei Shi et al.
Clueing up LLMs with Tool-Augmented Deductive Reasoning
Rebecca Ansell, Autumn Toney-Wails