pipette
ENEnglish

Early Memory Selection for Balanced Adam

Alberto Fern\'andez-Hern\'andez, Cristian P\'erez-Corral, Jose I. Mestre, Manuel F. Dolz, Enrique S. Quintana-Ort\'i

PreprintCódigo disponible

En palabras de los autores

We propose a method for choosing the shared memory parameter in Adam from a short pilot training. The selected remains fixed during the subsequent full training. A local model of Adam's normalized direction balances sampling variability against the delay introduced by averaging past gradients. This balance gives a cubic memory rule, whose two coefficients are estimated from gradient probes at a few pilot checkpoints. The estimator uses the numerator and denominator jointly, preserving their covariance. With a 200-update pilot and sixteen probe gradients at each of four checkpoints, a seed-matched retrospective evaluation on eleven vision and language workloads reduces mean relative validation gap by 40.7% and worst-quarter mean gap by 44.3% against the grid representative of shared . The mean gap is also 32.3% lower than that of the best constant chosen across all eleven workloads.

Resultado principalEl resumen no menciona limitaciones.

Apareció: miércoles, 7 de octubre. arXiv. Preprint, todavía sin revisión por pares.

Comentario de los autores: Includes theoretical proofs and reproducibility appendices. Code and data: https://github.com/AlbertoFdezHdez/Adam_beta_rule_cubic