Investigating Performance Trends of Simulated Real-time Solar Flare Predictions: The Impacts of Training Windows, Data Volumes, and the Solar Cycle

Abstract

This study explores the behavior of machine learning-based flare forecasting models deployed in a simulated operational environment. Using Georgia State University's Space Weather Analytics for Solar Flares benchmark dataset (Angryk et al. 2020a,b), we examine the impacts of training methodology and the solar cycle on decision tree, support vector machine, and multilayer perceptron performance. We implement our classifiers using three temporal training windows: stationary, rolling, and expanding. The stationary window trains models using a single set of data available before the first forecasting instance, which remains constant throughout the solar cycle. The rolling window trains models using data from a constant time interval before the forecasting instance, which moves with the solar cycle. Finally, the expanding window trains models using all available data before the forecasting instance. For each window, a number of input features (1, 5, 10, 25, 50, 120) and temporal sizes (5, 8, 11, 14, 17, 20 months) were tested. To our surprise, we found that for a 20-month window, skill scores were comparable regardless of the window type, feature count, and classifier selected. Furthermore, reducing the size of this window only marginally decreased stationary and rolling window performance. This implies that, given enough data, a stationary window can be chosen over other window types, eliminating the need for model retraining. Lastly, a moderately strong positive correlation was found to exist between a model's false positive rate and the solar X-ray background flux. This suggests that the solar cycle phase has a considerable influence on forecasting.