pipette
ESEspañol

Two-Sided Conformal Evidence Sets for Retrieval-Augmented Generation

Ben Jenkins, Mihaela Cardei

Preprint

In the authors' words

Retrieval-augmented generation (RAG) can surface passages that support a claim, contradict it, or fail to address it, yet existing calibrated grounding methods are typically one-sided and model only supporting evidence. We introduce Conformal Evidence Sets (CES), a two-sided framework that returns separate support and refutation sets with finite-sample inclusion coverage of at least on each side under exchangeability; a Bonferroni union yields joint coverage of at least . On labeled candidate pools from the contrastive VitaminC benchmark, CES closely tracks nominal coverage across , with maximum deviations of percentage points for support and for refutation, and reaches essentially the same empirical coverage--size frontier as a test-tuned fixed-threshold heuristic without any test exposure. Stress tests mark the boundary of that guarantee: under FEVER distribution shift both sides under-cover, by up to percentage points, and support-side group-stratified recalibration substantially reduces the support gap. Scorer swaps, hard-negative retrieval, cross-dataset transfer, and a live end-to-end pilot expose a clear separation: when exchangeability holds and relevant evidence is retrieved, evidence inclusion remains calibrated even with an imperfect scorer, whereas the support/refute verdict tracks scorer quality directly. Calibrated evidence coverage and reliable verdicts are therefore distinct properties and should be reported separately.

Main resultLimitation the authors admit

Appeared: Tuesday, November 10. arXiv. Preprint, not yet peer-reviewed.

Authors' comment: Accepted at the 2026 IEEE International Conference on Tools with Artificial Intelligence (ICTAI 2026). 8 pages