Right Reason, Wrong Advice: Evaluating Recourse in LLM Explanations of Denied Requests
Arunkumar V
Abstract
Language models increasingly write the explanations people receive when a rule system denies their request. If several rules deny the request independently, an explanation can name a true reason and still leave the person denied after they fix it. We call the missing property recourse validity: changing what the explanation cites, without changing what was asked for, can flip the decision. We compute it exactly, and a hitting-set argument predicts from the policy alone when a correct reason fails. In six of seven rule systems ported from access control and US benefit eligibility, a single correct reason almost never gives recourse when several reasons apply. Across 20 LLMs, checks of whether the cited reasons are correct, computed or asked of an LLM judge, are close to chance at spotting this failure, as the argument predicts. A strong judge asked whether acting on the explanation would work does spot it. Stronger and reasoning models do better but still fail on many denials with four or more independent reasons. Asking for every rule helps every open model with room to improve. Explanations of eligibility decisions should be built from the decision function and judged by whether acting on them works.
Create a lesson
Related papers
Auditors Struggle to Catch Sabotage In End-to-End Transcripts of AI Safety Research
Keshav Shenoy, Anthony Zang, Eric Gan et al.
How Much of SAR Vehicle Recognition Is the Vehicle? Background Shortcuts, Angle Gaps and Partial Libraries on MSTAR and ATRNet-STAR
Mousa Shaya
Chain-of-Thought Controllability Can Reduce Chain-of-Thought Monitorability
Zheng-Xin Yong, Benjamin Arnav, Miles Wang et al.
SciResearch-Eval: Evaluating Scientific Judgment in Autonomous Experimental Agents
Brian T. Weston, Alondra Torres Contreras, Aldair E. Gongora
Procedure without Judgment: What Small Language Models Learn under an Exact Verifier
Arunkumar V, Manoranjan Gandhudi, Gangadharan G. R. et al.
Safe Resemblance, Silent Propagation: Limits of Text-Similarity Guardrails for AI Agent Safety
Charles Xiao