Inference of Component Effect on System Performance

Abstract

In a computer system, multiple components--such as the CPU, memory, and others--work together as a system whose performance can be directly measured. However, the effect of a component under investigation (CUI), e.g., CPU, on system performance cannot be directly measured and can only be inferred. Accurately inferring CUI effect on system performance is a critical issue. Our experiments reveal that the general-purpose rigorous methodologies, like Design of Experiments (DoE), Randomized Controlled Trials (RCTs), and a single-purpose empirical methodology, like SPEC CPU2017, can not address this issue effectively and efficiently. We propose a rigorous methodology to address this issue: First, we identify a self-contained system (SCS) under the context of which we can completely understand how CUI and other essential components affect the system performance, and then we use a structural causal model methodology to represent and infer the causal effect of CUI on the system performance. We utilize this methodology and verify its correctness in the context of CPU design and evaluation. Through theoretical analysis and pioneering controlled experiments, we systematically compare our methodology against three established methodologies: SPEC CPU2017, DoE, and RCTs. The results show that our methodology can achieve its goal effectively and efficiently, whereas others exhibit inherent limitations.