A Taxonomy of Failing Bug Reproduction Tests on SWT-Bench
Luca Filippelli, Hamed Taherkhani, Hung Viet Pham, Hadi Hemmati
Abstract
Automatic Program Repair (APR) tools require high-quality test generation models to validate candidate patches. Despite the advancements of automated bug reproduction testing tools, they continue to struggle with issues of higher complexity. These struggles may be caused by a framework's lack of problem-specific context. We address the unclassified nature of these framework failures by providing a taxonomy of bug reproduction tests. This study uses the AssertFlip and OpenHands frameworks, the two highest-scoring open-source tools on SWT-Bench Verified. The significance of model selection on the causes of failure is also explored by utilizing GPT-5-mini and GPT-4o-mini on both tools. The error traces of these frameworks are categorized to determine granular modes of failure. Through this process, mechanical and implementation failures were the most common failure classes. Overall, a tool's generation approach has a significant impact on precision and coverage, while the selection of the model alters the quality and quantity of generated tests. Specifically, more powerful models with higher reasoning capability tended to present fewer failures due to a lack of technical or problem-specific context. Our experiment also shows evidence that our taxonomy can be used as concrete guidance to augment the issue description to improve the generation quality.
Create a lesson
Related papers
Verify the Verifier: Specification-Grounded LLM Cross-Checking for Bugs That Escape UVM Scoreboards
Ana E. Sanchez Villalobos, Michael D. Quinn, Aakash Tyagi et al.
Anticipating Future Build Results with BuildPredictor
Christian Macho, Katharina Stengg
Static Analysis Guidance for Secure LLM Code Generation
Daniele Lunghi, Alberick Euraste Djiré, Zacharie Chenail-Larcher et al.
SWE-Recreation: Benchmarking Coding Agents for Software Recreation
Zhiyao Xu, Zhanjie Ding, Hongyao Yu et al.
SkillBridge: Cost-Aware Diagnosis and Repair of Cross-OS Compatibility Defects in Agent Skills
Anjie Xu, Zixing Wang, Yifeng Cai et al.
Barriers to GenAI Integration During Large-Scale Agile Transformation: An Exploratory Case Study
Nils Brede Moe, Marthe Berntzen, Astri Barbala et al.