Statistical learning theory and Occam's razor: Regularization
Tom F. Sterkenburg
Abstract
The principle of Occam's razor, which instructs us to prefer simplicity in inductive inference, has attracted much scrutiny both in the philosophy of science and in machine learning. In either field, however, a justification for the principle has been elusive. In this paper, building on an earlier "core argument," I spell out a justification from statistical learning theory for the procedure of regularization: for trading off fit for simplicity. The means-ends argument is that in order to profit from theoretical reliability and "what-you-see-is-what-you-get" guarantees, one must implement a certain preference for simplicity over fit. This is a genuine methodological justification, which neither collapses to a purely pragmatic principle that we prefer simplicity for its own sake, nor to an ontological assumption that the truth is simple.
Create a lesson
Related papers
A General Kernel Framework for Non-CND Distance Measures Using |D|-Dimensional Sparse Landmark Embeddings
Marcus M. Noack, Maher B. Alghalayini, Mark D. Risser
Fast Learning Rates for Physics-Informed Kernel Methods
Luc Brogat-Motte, Joachim Bona-Pellissier, Giacomo Meanti et al.
Rank and computation of the pathlifting Jacobian of a DAG ReLU network
Manon Verbockhaven
Preservation of Log-Concavity and Convergence of Wasserstein-Fisher-Rao Gradient Flows
Francesca Romana Crucinio, Sahani Pathiraja
Generalized DCCQ: From Binary Quotients to Multinomial Simplex Geometry and Critical-Strip Coordinates
Y. Kenan Yılmaz
Bracketing Uncertainty in Clustering Under the Manifold Hypothesis
Savik Kinger, Luciano Dyballa, Steven W. Zucker