pipette
ESEspañol

Phase Transition from Random Features to Fully Trainable Networks

Xin He, Huan Lao, Xiliang Lu, Shijun Zhang

Preprint

In the authors' words

Constructing accurate and computationally tractable approximation spaces is a fundamental task in scientific computing. Partially trainable neural networks bridge fixed random features and fully adaptive representations. This paper quantifies a phase transition pattern in approximation bounds and observes analogous behavior in training errors: a rapid but smooth initial improvement followed by diminishing returns, with a sharper transition under greater initialization mismatch. Under polynomial spectral decay with exponent and leverage compatibility, we establish a squared approximation rate for two-layer networks of width and trainable proportion , where . The bound holds uniformly over the unit ball of a reproducing kernel Hilbert space (RKHS) with probability at least . The constants and quantify the fixed and trainable groups' sampling mismatches relative to the target RKHS. For spherical biased-ReLU features in dimension , we verify these assumptions with and for initialization scale , and under ridge leverage sampling. For and , a trainable proportion of approximately captures of the reduction in the approximation bound from to . A structured three-layer example illustrates how a few trainable first-layer neurons exploit fixed downstream blocks through nonlinear composition. Numerical experiments across diverse settings, including networks with up to 15 layers, exhibit similar phase transitions in training errors.

Main resultThe abstract does not state a limitation.

Appeared: Friday, October 23. arXiv. Preprint, not yet peer-reviewed.