SPARK: Semantic-preserving Prompt Mutation for Code Generation using Consensus-based Verification Framework
Alireza Daghigh, Chung-Yu Wang, Luca Filippelli, Hamed Taherkhani, Hadi Hemmati, Hung Viet Pham
Abstract
Automated code generation with large language models (LLMs) has improved in recent years. With the lack of test cases in real-world usages, generated code is not automatically verifiable and often requires manual validation. Previous studies have leveraged consensus verification, where code solutions and corresponding test cases are generated and evaluated based on functional consistency. Building on this, SPARK incorporates semantic-preserving mutations to problem descriptions such as paraphrasing, grammatical adjustments, function renaming, and other alterations to enhance the robustness and correctness of LLM-generated code. For each mutated description, SPARK generates a set of code solutions and test cases. These candidate solutions are evaluated using consensus-based verification, where the solutions' quality is measured by how consistently they pass generated tests. We assess the effectiveness of SPARK on three benchmarks (LiveCodeBench, BigCodeBench, and DLBench) across three LLMs (o4-mini, DeepSeek-R1-Basic, and Gemini-2.5-Pro). Our evaluation demonstrates that SPARK outperforms state-of-the-art consensus-based verification approaches by up to 4.1%. Our generalization experiment highlights the practical potential of SPARK as an effective strategy for enhancing the reliability and correctness of LLM-based code generation systems in scenarios where traditional validation resources are scarce, while our qualitative analysis shows insight into how SPARK's mutation helps LLMs generate better code solutions.
Create a lesson
Related papers
Verify the Verifier: Specification-Grounded LLM Cross-Checking for Bugs That Escape UVM Scoreboards
Ana E. Sanchez Villalobos, Michael D. Quinn, Aakash Tyagi et al.
Anticipating Future Build Results with BuildPredictor
Christian Macho, Katharina Stengg
Static Analysis Guidance for Secure LLM Code Generation
Daniele Lunghi, Alberick Euraste Djiré, Zacharie Chenail-Larcher et al.
SWE-Recreation: Benchmarking Coding Agents for Software Recreation
Zhiyao Xu, Zhanjie Ding, Hongyao Yu et al.
SkillBridge: Cost-Aware Diagnosis and Repair of Cross-OS Compatibility Defects in Agent Skills
Anjie Xu, Zixing Wang, Yifeng Cai et al.
Barriers to GenAI Integration During Large-Scale Agile Transformation: An Exploratory Case Study
Nils Brede Moe, Marthe Berntzen, Astri Barbala et al.