Benchmarking Blunders and Things That Go Bump in the Night
Neil J. Gunther
Abstract
Benchmarking; by which I mean any computer system that is driven by a controlled workload, is the ultimate in performance testing and simulation. Aside from being a form of institutionalized cheating, it also offer countless opportunities for systematic mistakes in the way the workloads are applied and the resulting measurements interpreted. Right test, wrong conclusion is a ubiquitous mistake that happens because test engineers tend to treat data as divine. Such reverence is not only misplaced, it's also a sure ticket to production hell when the application finally goes live. I demonstrate how such mistakes can be avoided by means of two war stories that are real WOPRs. (a) How to resolve benchmark flaws over the psychic hotline and (b) How benchmarks can go flat with too much Java juice. In each case I present simple performance models and show how they can be applied to correctly assess benchmark data.
Create a lesson
Related papers
How the Dirty Set Moves: Characterizing Schedule Reuse in Incremental Sparse Matrix Multiplication
Hunter Horne, Eric Spencer, Alex Henningfield et al.
PrismStencil: Transforming Stencil Computations into Matrix Multiplications
Xianmeng Jiang, Yuan Liang, Zixuan Wang et al.
Fast and Flexible Spiking Network Simulations on Massively Distributed On-Chip Memory Accelerators
Catherine Mia Schöfmann, Jan Finkbeiner, Jan Vogelsang et al.
How Reliable Are Large Language Models for Soil Organic Carbon Assessment? An Australian Benchmark
Yingxuan Cheng, Tong Li, Yayong Li et al.
AlignPred: Error-Controlled KV Cache Compression via Alignment and Predictive Coding
Xu Yang, Jiapeng Zhang, Zhiwei Zuo et al.
DynaSPLIT: Strong tail-optimal scheduling for the M/G/n with heavy-tailed job sizes
Zhouzi Li, Mor Harchol-Balter, Alan Scheller-Wolf