Flaky Test Recognition when Testing CPSs Using Hybrid Models

Abstract

Cyber-Physical Systems (CPSs) have many applications, ranging from simple thermostat systems to autonomous driving systems and medical devices. Like all systems, they need to function correctly. To this end, validation techniques such as testing that can effectively reveal faults are required. Also, CPSs usually operate in uncertain environments where various expected/unexpected events can affect their behaviors. These events together with timing and synchronization incidents may result in various/different CPS behaviors and may cause a CPS to pass a test under some conditions and fail it under others. Such test cases are called ``flaky'' test cases and we call the conditions that are responsible for them ``flaky conditions'' and call these behaviors ``flaky behaviors''. When test cases are flaky, testing results are unreliable. To achieve more reliable test results, engineers may attempt to recognize flaky test cases and remove them. In this work, beginning with a test case generation and execution technique called HyTest that we created previously, we integrate TReVa, a new technique that validates testing results provided by HyTest and FlaRe, a new technique that recognizes flaky test cases and flaky conditions using hybrid models during the early stages of CPS development in just one additional round of testing. We present the results of an empirical study evaluating the effectiveness of our new approach (which we call HyTestTF). Our results show that correctly differentiate flaky test cases from non-flaky test cases