Two hidden layer suffice: the caterpillar net
Jakob Paul Zimmermann
Abstract
Every continuous piecewise linear (CPWL) function is a finite signed sum of maxima of affine functions, making maximum representation a central problem in the study of the expressiveness of ReLU networks. A now-disproved conjecture predicted logarithmic depth for maxima. Inspired by the proof by Brunck et al. (2026) that two hidden layers suffice for general CPWL functions, we give the Caterpillar Net, a nonrecursive construction for the maximum of N inputs with two hidden layers of widths O(N2) and 2O(N N). The construction has a direct geometric interpretation.
Create a lesson
Related papers
Text-to-Image Generative Modeling Without Language Pretraining
Keshigeyan Chandrasegaran, Yue Zhao, Armin W. Thomas et al.
WeAct: Dynamic Weight Switching as Agent Actions
Rui Shi, Xunjian Yin, Jiacheng Sang et al.
Kronecker Delta Rule for Long Context Linear Attention
Anamaria-Roberta Hartl, Levente Zólyomi, Michael List et al.
Next Best Action: Causal Constrained Offline RL for Business Action Targeting
Surya Ramachandiran, Yiming Li, Juan Mancilla-Caceres et al.
Scaling Parameter-Efficient Fine-Tuning for Brain Encoding
Zhuofan Josh Ying, Guanglong Sun, Nolan Ho et al.
Learning to Refer: Inference-Time Collaboration in Decentralized Federated Learning
Yuheng Ma, Chendi Wang, Qiong Zhang