Efficient Screening Designs for Expensive Black-box Models with Qualitative and Quantitative Factors

Abstract

Computationally expensive black-box models often involve a large number of input factors with complex interactions and varying importance. Experimental design techniques can be used for quickly identifying the important factors, which can make the optimization of a complex computer model or the training of an expensive machine learning model more efficient. Existing screening designs for black-box models focus mainly on continuous factors, with the maximum one-factor-at-a-time (MOFAT) design being a recent example. In this work, we extend the design to incorporate multiple types of factors, including nominal, ordinal, and discrete-numeric. We first identify the properties leading to optimal screening, where qualitative and quantitative factors should be treated differently. Based on these properties, we propose practical algorithms to efficiently construct MOFAT designs for all types of factors. The usefulness of the design is demonstrated by both numerical experiments and an application to hyperparameter tuning in machine learning models.