pipette
ESEspañol

DrivingBench: Can Vision-Language Models Drive a Toyota Corolla?

Aditya Ramabadran, Simon Mahns, Tobias Gessler

PreprintClaims a big stepEasy to read

In the authors' words

Frontier models excel at many digital benchmarks, yet their ability to drive a real car, an everyday human skill, remains largely untested. We present DrivingBench, to our knowledge the first benchmark where general-purpose vision-language models must drive a real car. Through three tools, the models see camera frames from a Toyota Corolla and directly command its steering and velocity around a parking lot cone course at low speeds. The car may continue moving while the model thinks and new commands replace the currently running one, so inference latency is part of the task, testing the models' abilities to observe, act, monitor, recover, and complete a long-horizon objective under such constraints. We benchmark GPT-6 Astra, Claude Fable 5.1, GPT-5.6 Sol, and Grok 4.6 in vendor-native harnesses (Codex, Claude Code, Cursor) with up to three attempts each in one conversation; Astra is the only model to finish the course, on its second attempt, with no other attempt passing 50% of the course. Two of the four models improved materially across attempts with retained context. We also detail the design principles behind our action interface, and show how the tool output format and the framing of the task combined to determine whether models would drive at all or refuse. We release our harness, prompts, course map, and traces with video and telemetry for reproducibility.

Main resultThe abstract does not state a limitation.

In plain words AI summary

The researchers built DrivingBench, the first test where general-purpose vision-language models must drive a real car. The models see camera images from a Toyota Corolla and directly command its steering and speed around a cone course at low speeds. Inference latency is part of the task, testing the models' abilities to observe, act, monitor, recover, and complete a long-horizon objective. Only GPT-6 Astra finished the course, on its second attempt; no other attempt by any model passed 50% of the course. Two of the four models improved materially across attempts with retained context.

Written by Kimi K2.5 using only the abstract above. Jev checked that every sentence is supported by the abstract sentences it cites, and removed any that were not. Point at a sentence to highlight its source.

Appeared: Thursday, October 1. arXiv. Preprint, not yet peer-reviewed.