An Accuracy--Information Tradeoff for Loss-Difference Conditional Mutual Information
En palabras de los autores
Loss-difference conditional mutual information (ld-CMI) uses the smallest of the standard observations in the supersample hierarchy of generalization bounds: it measures what a learner's loss differences reveal about which candidate of each pair it was trained on. Accuracy is known to force information into the model; data processing does not carry such lower bounds to losses. We show, by bounding three moments of the loss differences, that accuracy also forces ld-CMI. For linear predictors with a smooth convex loss of nonzero slope at zero, such as the logistic loss, plus a regularizer whose curvature and growth are both of power , on product distributions over a scaled sign cube in dimension at least linear in , every proper learner with expected excess risk at most on these distributions at the optimal sample size has worst-case ld-CMI of order bits, and bits under Gaussian noise of standard deviation on the loss differences. The same holds without a regularizer, at . Consequently, range-scaled ld-CMI bounds cannot vanish on these distributions, although every proper learner's generalization gap is . We also show that model-level information does not determine noisy loss-difference information, and that the growth, slope and dimension conditions are needed, the last up to a logarithm.
Apareció: jueves, 8 de octubre. arXiv. Preprint, todavía sin revisión por pares.
Comentario de los autores: 61 pages, of which 8 pages main text. Code, data and the Lean 4 formalization are in the ancillary files