Beyond Mean Foils: Auditing Worst-Foil Specificity in Frozen CLIP Region Explanations
In the authors' words
A region can overlap a target object yet contribute more to another class. We test regions selected by Cluster-based Concept Importance (CCI) in frozen CLIP. Across COCO and VOC with two checkpoints, 41.08-64.78% of regions that pass overlap and mean-contrast checks fail against the strongest competing class. Removing competitors annotated in the image leaves 39.69-63.64% failing. We then test all eight candidate regions per image. An alternative passes the test for 6.25-7.84% of failures on COCO and 27.40-31.15% on VOC. Requiring it to preserve the original target-score drop within reduces these rates to 0.16-0.98%. Available regions and target-drop tolerance constrain repair; relaxing the tolerance increases repair opportunities.
Appeared: Thursday, September 24. arXiv. Preprint, not yet peer-reviewed.
Authors' comment: 5 pages, 2 figures, 6 tables