Noise Floor Audit for Agent Benchmarks
Yihang Chen, Pin Qian, Su Wang, Chong Peng, Huan Xu, Xiyang Wu, Yiqi Sun
Abstract
We audit measurement variability for 3 native tool-calling endpoints across 2 providers on the official BFCL multiple and parallel categories, using matched AST grading. At temperature 0, reruns are nearly deterministic across Groq endpoints and a thinking-enabled Gemini setting: ever-flip fractions are 0.7%, 2.0%, and 2.7%, with mean run correlations of 0.997, 0.966, and 0.961. Semantics-preserving prompt perturbations create the larger floor on all endpoints, with median perturbation paired SDs 11x to 58x larger than rerun paired SDs. The failure character also shifts: malformed-output failures account for 30%, 7%, and <1% of task failures, so marginal accuracy hides not only stability but also failure mode.
Create a lesson
Related papers
Monitoring and Discovering Reward Hacking with Internal Representations during LLM Evaluations
Leon Bergen, Usha Bhalla, Andrew Lee et al.
Reporting Practice Matters: The Impact of Reference Choice on Chest X-ray Report Evaluation
Daniel P. Jeong, Charles Q. Li, Hossein Hosseiny et al.
Long-Lived Characters, Local Inference: Incremental Memory Maintenance for Game NPCs
Zimu Xu
Beyond Outcomes: Dual-View Relational Learning for Efficient Agent Benchmarking
Xinshuai Guo, Junjie Wu, Dolly Deng et al.
How Much is a Human Right Worth? ECtHR-NPD: A Benchmark for Predicting Non-Pecuniary Damage Awards
Yanyi Pu, Damian A. Gonzalez-Salzberg, Zheng Yuan et al.
Structured Claim-Level Discourse Representations for Dense Health Narratives
Farnoushsadat Nilizadeh, Elham Pourabbas Vafa, Shirin Nilizadeh et al.