Detecting Inconsistencies in Model Specifications with LLM-as-Verifier Reasoning
In the authors' words
Model specifications define how large language models (LLMs) should behave, guiding alignment training, inference-time behavior, and evaluation. Yet these specifications may themselves contain defects: two individually reasonable principles may prescribe incompatible behavior when applied to the same situation, leaving no response that satisfies both. Detecting such inconsistencies is challenging. Formalizing natural-language specifications risks losing subtle distinctions, while behavior-based testing cannot reliably distinguish specification defects from differences in model behavior. We introduce VeriSpec, the first approach to directly detect inconsistencies in model specifications by auditing the specification text itself. Our key insight is to preserve the specification in natural language while using an LLM as a verifier. VeriSpec extracts structured, context-aware rules, constructs a topic-guided graph to cluster behaviorally related rules at the same authority level, and applies LLM-as-verifier reasoning to detect inconsistencies. Applying VeriSpec to the OpenAI Model Spec, we extract 405 rules and manually validate five inconsistencies, all reported to its developers, who responded positively and have initiated internal discussions. Compared with five baselines, VeriSpec identifies the most validated inconsistencies, achieves the highest precision (38.5%), and incurs the lowest cost per validated inconsistency ($11.12). These results establish direct specification auditing as a practical complement to behavioral alignment evaluation, catching defects at the source before they shape any model. The code is available at https://github.com/HIPREL-Group/VeriSpec.
In plain words AI summary
Model specifications guide how large language models should behave, but two principles may conflict in the same situation. The authors created VeriSpec, which uses an LLM as a verifier to find inconsistencies directly in natural-language specification text. Applied to the OpenAI Model Spec, VeriSpec extracted 405 rules and found five validated inconsistencies, which were reported to developers. VeriSpec achieved 38.5% precision and cost $11.12 per validated inconsistency, outperforming five baselines. The authors suggest direct specification auditing can catch defects before they shape any model behavior.
Written by Kimi K2.5 using only the abstract above. Jev checked that every sentence is supported by the abstract sentences it cites, and removed any that were not. Point at a sentence to highlight its source.
Appeared: Friday, October 2. arXiv. Preprint, not yet peer-reviewed.