Does ISO-Grounded NFR Specification Improve LLM Code Generation? A Comparison of Rich and Structured Interventions against a Natural-Language Baseline
Joào Pedro Monteiro Pereira, Vinicius Cardoso Garcia
Abstract
In LLM-based code generation, Non-Functional Requirements (NFRs) are often specified as terse one-line phrases. We ask whether grounding those specifications in ISO/IEC 25010 Quality Model, either as rich natural-language prose (NL-rich) or as structured JSON (Structured), improves code generated on HumanEval/HumanEval-ET compared to a RobuNFR-style one-line baseline (NL-simple). We evaluate four NFRs (performance, error handling, code smell, readability) with ten prompt variations per condition under a fixed model snapshot and paired non-parametric analysis. Primary finding: ISO-grounded enrichment improves static quality proxies (unreadability density falls across all four NFRs (e.g., Performance 0.88 -> 0.69 for NL-rich)) and reduces sensitivity to prompt wording, but does not reliably improve functional correctness; for error handling, extended-test pass rate decreases, suggesting tension between defensive coding patterns and exact-output benchmarks. Secondary finding: when ISO content is held constant, NL-rich and Structured differ negligibly in correctness (|delta| <= 0.023), indicating that semantic content matters more than JSON-vs-prose format. Practitioners should invest in standard-grounded NFR content rather than serialization form. A fully traceable replication package is provided.
Create a lesson
Related papers
ProgramDistill: From Interactive Web Apps to Verifiable Reference-Guided SWE Tasks
Jeonghye Kim, Minseon Kim, Young Jin Kim et al.
Evaluating the Health of Open-Source Smart City Platforms
Rodrigo Bravo Simões, Fernando Brito e Abreu, Vasco Amaral
From Component Snapshots to Lifecycle Traces: Agent-Based Software Composition Analysis
Chaofan Li, Zhengduo Xue, Chengxiang Li et al.
A Study on the Impact of Natural Language Differences in Prompts on Automatic Code Generation Using LLMs
Haruka Tokumasu, Masanari Kondo, Alexander Serebrenik et al.
A Study of the Reliability of Agentic AI-Generated Programs
Ayesha Shafique, Barton P. MIller, Elisa R. Heymann
Relationally Guided Use Case Modeling with LLMs
Guangyu Wang, Bangqi Li, Ji Wu et al.