DataTrace: Benchmarking Data Agents for Evidence-Grounded Root-Cause Localization
Tung Sum Thomas Kwok, Xiaotian Lin, Yuyu Luo
Abstract
Database agents diagnose pipeline failures, but a correct root cause is trustworthy only when it is supported by evidence the agent inspected. Existing benchmarks usually grade the final query or answer but fail to distinguish an evidence-based diagnosis from an unsupported correct answer. We introduce DataTrace, a benchmark for evidence-grounded root-cause localization over typed graphs of database artifacts. DataTrace uses controlled fault injection to derive root causes and evidence paths within 83 runnable environments without human annotations, and scores root-cause correctness and coverage of the supporting path separately. We also introduce CoSE, a generator-solver protocol in which an external verifier admits reproducible faults and checks both the submitted answer and the inspected evidence. Our evaluation shows that seven proprietary models achieve average root-cause accuracy 48%, with a mean per-model conditional unsupported rate of 18%. Fine-grained analysis shows that this gap varies sharply with fault mechanism, dependency distance, and evidence modality. After in-loop training on a matched held-out batch, the share of grounded correct diagnoses rises from 83.7% to 94.6%, while root-cause accuracy rises from 29.2% to 36.7%. These results show that while final-answer accuracy can hide unsupported diagnoses, evidence coverage provides an actionable signal to improve database agents. Code and data are available at https://hkustdial.github.io/datatrace/.
Create a lesson
Related papers
Bridging the Discovery Gap in Solid Dataspaces with a Semantic Data Catalog
Florian Hölken, Alexander Paulus, Tobias Meisen et al.
GenSPARQL: A Generative Extension of SPARQL for Incomplete Knowledge Graphs
Jingcheng Wu, Yi Wang, Hongkuan Zhou et al.
Standardized Tolerance for Adaptive Reuse and Repair: A Method for HNSW Migration Under Embedding Model Change Scenarios
Hengzhao Ma, Chenyang Zhu, Bin Wang et al.
The American Medical Directory Panel: Comparing AI-Assisted and LLM Transcription with Manual Data Entry at Scale
Christian M. Dahl, Sophie Li, Rasmus Emil Nyholm et al.
CRAFT: Constrained Route planning with Adaptive Framework and Template-based code generation
Shahd Elmahallawy, Ahmed Eldawy
ProvCards: Artifact-Oriented Provenance for Heterogeneous Scientific Workflows
Tanja Auge, Marinus Holitschke, Meike Klettke