Don't Read the Log: Execution Traces Contaminate Verifiers in Video-Generation Agents
En palabras de los autores
Agentic video-generation systems close a loop between a generator and a verifier: an LLM plans shots, calls a text-to-video model, and a multimodal judge decides whether the result satisfies the request. To diagnose where a long workflow fails, recent harnesses deliberately show the judge more than the video-the agent's execution trace, its plan, the narration it synthesized. We ask whether this auxiliary text moves the judge's verdict on purely visual requirements, holding the frames fixed. On a benchmark of 109 generated two-event clips with manual labels, in which the requested event is either visibly completed or visibly missing, a trace that reports a successful tool call makes three open-weight Qwen-VL judges (7B, 8B, 32B) accept -- of the failures, up from -- without text, and a contradicting trace makes them reject up to of correct clips; an instruction to "use only the frames" does not remove the effect. Frontier closed judges are essentially unmoved on the same clips, showing that the vulnerability is a property of the judge's learned trust in tool logs rather than of the task. Plan-derived text carries no clip-specific information, so it can only shift a judge's operating point, and in a repair loop that shift becomes a cap on the true pass rate that no repair policy can exceed; the cap matches simulation to two decimals. In the loop, contamination is exploited without any adversarial agent: an honest LLM planner that always regenerates ends with a judge pass rate of and a human-labelled pass rate of , and a pipeline in which a cheap checker writes its verdict into the trace launders that checker's errors into a stronger final judge ( false accepts).
Apareció: viernes, 25 de septiembre. arXiv. Preprint, todavía sin revisión por pares.