Statistical Inference with Ill-Conditioned Random Weight Neural Network Sieves
Giovanni Ballarin, Lukas Gonon, Lyudmila Grigoryeva
Abstract
Random weight neural networks (RWNNs) are shallow networks whose hidden-layer parameters are randomly generated and then kept fixed, while only the output coefficients are estimated. Conditional on the sampled parameters, RWNN regression is a linear sieve problem with random approximation space. We show that such random sieves cannot remain uniformly well-conditioned as their dimension grows. For parametrized feature families satisfying polynomial covering and Lipschitz conditions, the eigenvalues of the population covariance matrix are proven to have polynomial upper bounds determined by the covering geometry of the parameter space, leading to a vanishing smallest eigenvalue. We develop approximation, estimation, and inference theory that explicitly treats the ill-conditioning. Under a β-Hölder source condition, we obtain both faster-than-Monte-Carlo L2 approximation rates and new L∞ approximation bounds based on stratified sampling. We then derive improved consistency guarantees for ridge-regularized least squares driven by ridge effective dimension, spectral, and leverage-based stability conditions. For inference, we establish asymptotic linearity and Gaussian limits for linear functional directions while allowing the smallest eigenvalue of the population covariance matrix to vanish. Nondegeneracy is directional: we characterize it with a ridge-normalized Rayleigh quotient and extend it to smooth functionals via Riesz representation. Consistency of ridge-adjusted variance estimators is proved, yielding feasible studentized inference. Numerical experiments illustrate the deterioration of random-feature conditioning and the finite-sample behavior of the resulting estimation and inference procedures.
Create a lesson
Related papers
Additive Fréchet Regression for Random Objects
Changwon Choi, Wookyeong Song, Hans-Georg Müller et al.
Realized versus Population Reliability: Anytime-Valid Self-Consistency Certification under Correlated Reasoning
Terence D. Agbeyegbe, Toneukarin G. Agbeyegbe
Spatiotemporal Risk-Based Hotspot Dynamics Assessment
José M. Angulo, Ana E. Madrid
Optimal Transport Rank Regression for Robust Multivariate-Response Regression
Yizhe Ding, Yifan Jiang, Runze Li et al.
Causal Effect Estimation Under Uncertain Statistical Surrogacy
Yi Liu, Alexander W. Levis, Shu Yang et al.
Elementwise Precision-weighted Compositional Data Analysis by Weighted Alternating Least Squares
Jan Graffelman