Calibrating Reproduced Claims in Recommender Systems
En palabras de los autores
Reproduction studies can produce mixed outcomes. Reported values may differ while the ordering of the compared methods remains the same, a result may hold only under some experimental conditions, or a released implementation may fail to reproduce a result that the model can still reach. The terms repeatability, reproducibility, and replicability describe how a follow-up study relates to the original experiment, but not which parts of the original claim are supported by the new results. We introduce claim calibration as a way of stating the strongest claim supported by a follow-up study, together with the conditions under which it holds and the parts that remain untested. We apply this perspective to five original--follow-up paper pairs from recommender-systems research. The cases show that agreement in numerical values, method rankings, statistical results, and overall conclusions does not always coincide, and that follow-up studies often support only part of the original claim. Based on these observations, we propose a Claim Evidence Profile for reporting the original claim, its scope, the reproduction target, the reported results, the calibrated claim, and the parts of the original claim that remain unresolved.
Apareció: jueves, 24 de septiembre. arXiv. Preprint, todavía sin revisión por pares.
Comentario de los autores: Accepted to the Workshop Methodology First - Rethinking Research Assessment in RecSys (FRAME) September 28, 2026, Minneapolis, Minnesota, USA