When Skills Tell Agents Not to Ask: A Large-Scale Study of Oversight Directives in Agent Skills
En palabras de los autores
Agent skills, reusable instruction packages centred on a SKILL.md file, extend AI agents on demand, and a single line in a widely copied skill reaches every agent that loads it. Some lines are oversight-loosening directives: they tell the agent to proceed without approval, to bypass checks with --no-verify, or to keep working without checking in. Although such directives can remove a checkpoint from every agent that loads them, no prior work has examined them. We present the first large-scale study of oversight-loosening directives, analyzing 1.53 million English skills from GitSkills. We split skills into units (sentences, bullets, table cells, and code lines), develop a codebook that labels oversight directives by polarity, mechanism, and scope, label 2,097 units by hand, and train a detector to scale the analysis. Our results show that about one in seventeen skills (about 88,000 skills) contains a loosening directive. These directives most often loosen oversight through permission flags, and most of them (70.1%) apply unconditionally rather than only in specific contexts such as CI. Compared with skills that have the same amount of oversight language, skills that run shell code, install packages, or use sudo have 1.7-1.8 times the odds of containing a loosening directive, an association that holds across all our robustness checks, and they do not compensate with more tightening. Loosening spreads through a few upstream skills rather than through edits by their adopters: 2.4% of the skill families that carry loosening account for half of all copies of loosening-carrying skills, and 1.1% of distinct loosening units for half of all loosening occurrences. Open-source skill scanners catch at most 46% of loosening directives, with a precision below 0.4, while the LLM behind them, prompted with our codebook instead, catches 61% at a precision of 0.71.
En palabras simples Resumen con IA
Las habilidades de agente son paquetes de instrucciones reutilizables que extienden a los agentes de IA, y algunas contienen líneas que indican a los agentes omitir la aprobación o eludir verificaciones. Los autores analizaron 1.53 millones de habilidades de GitSkills y encontraron que aproximadamente una de cada diecisiete contiene una directiva que afloja la supervisión. La mayoría de las directivas de aflojamiento usan banderas de permiso, y el 70.1% se aplican incondicionalmente en lugar de solo en contextos específicos. Las habilidades que ejecutan código de shell, instalan paquetes o usan sudo tienen 1.7-1.8 veces más probabilidades de contener una directiva de aflojamiento. El aflojamiento se propaga principalmente a través de algunas habilidades originales: el 2.4% de las familias de habilidades representan la mitad de todas las copias de habilidades que llevan aflojamiento.
Escrito por Kimi K2.5 usando solo el abstract de arriba. Jev verificó que cada oración esté respaldada por las oraciones del abstract que cita y eliminó las que no. Pasá el cursor por una oración para ver de dónde sale.
Apareció: viernes, 30 de octubre. arXiv. Preprint, todavía sin revisión por pares.
Comentario de los autores: preprint