Using superpixels for interpretable feature reduction in large 2D diffraction datasets

Abstract

Large 2D diffraction datasets, consisting of hundreds or thousands of measurements, are commonly acquired with 4D-STEM electron diffraction or at synchrotron X-ray beamlines. Machine learning and artificial intelligence offer great promise for analyzing these datasets. However, the sheer volume of data presents a significant data processing bottleneck. Cropping the detector and pixel binning are standard ways to reduce data size. Here we propose grouping and averaging pixels into superpixels of variable area. High-information regions are sampled densely, while low-information areas are collected into larger superpixels. In the process, symmetries in the data are captured and exploited, making this approach suitable for preprocessing 2D diffraction data. We compare two variance-minimizing methods: K-means clustering (top-down) and agglomerative clustering (bottom-up) and demonstrate superior scaling and interpretability for the bottom-up method. As these methods are distance-based, we demonstrate that the construction of superpixels can be accelerated using Gaussian random projection. Finally we show over 100-fold acceleration for phase mapping with Non-negative matrix factorization on a 4D-STEM dataset when superpixels are used as a preprocessing step.