Characterizing and Detecting Unfaithful Work Reports by Coding Agents
Wenhao Yang, Runzhi He, Minghui Zhou, Pengfei Gao, Dayi Lin, Qingwei Lin, Dongmei Zhang
Abstract
Coding agents increasingly carry out long-horizon development tasks, autonomously planning and executing multi-step actions that span files, tools, and sessions. Because developers rarely follow each action, and agents resuming the work depend on records of what was done, the agent's work report often becomes the primary account of what was changed and established. When the report contains false or unsupported claims, it gives recipients false assurance: incomplete work may be accepted without verification, incorrect assumptions about the current state can propagate to later steps and agents, and evaluators that credit stated success may reward the report rather than the work. Yet little is known about what unfaithful reporting looks like in real development work, or how developers and agents handle it once it surfaces. Starting from real developer--agent conversations in SWE-chat, we conduct a systematic mixed-methods analysis of self-admitted unfaithful reports. We identify eleven subtypes in four categories: misreported observations, misattributed evidence, overstated task success, and faulty technical explanations, with overstated task success the most common. Handling these problems involves on average 4.3 further user turns and 12.71 minutes of recorded response and execution time, and users express dissatisfaction more often. Among 435 work reports that we separately sample and annotate, \% contain at least one unfaithful claim. We also develop FaithDetector, a detector for unfaithful reporting informed by the observed patterns. On naturally sampled reports, FaithDetector attains \% macro-F1, about 13 percentage points above the baseline. Together, our findings and FaithDetector provide empirical evidence and an automated method for checking reporting faithfulness, informing agent evaluation and training feedback.
Create a lesson
Related papers
Adopting Embedded AI Through an Innovation Team
Viktoria Stray, Kristina Kantima Østvold, Agnes Hunstad Birkemo et al.
From Code Review Comments to Reuse in Production: An Industrial Study of Extracted Rules
Mutsuha Asada, Yuhei Nakasaka, Keito Tabuchi et al.
Copilot or Culprit? How AI Is Changing Open-Source Bug Reporting and Maintenance
Yinuo Liu, Yunlong Lyu, Zichao Luo et al.
Technology-Enabled Fuel Supply Chain Resilience: An Integrated Information System for Real-Time Monitoring and Decision Support
Mohamed Loutis, Esma Aïmeur
Behavioral Containment: Source-to-Binary Function Matching for 1-Day Vulnerability Detection
Yifei Yang, Xuejiao Luo, Penghui Li
Ontology of Software Maintenance
Luis Olsina, Pablo Becker, Maria Fernanda Papa