Turing-like tests for language and vision demonstrate a narrowing gap between humans and AI
In the authors' words
As artificial intelligence (AI) becomes increasingly embedded in daily life, ascertaining whether an agent is human is critical. Here we systematically benchmark AI’s ability to imitate humans in three language tasks (image captioning, word association and conversation) and three vision tasks (colour estimation, object detection and attention prediction), collecting data from n = 636 humans and 37 AI agents. We also conducted 72,191 Turing-like tests with 1,916 human judges and 10 AI judges. Current AIs are approaching the ability to convincingly impersonate humans and deceive human judges in both language and vision. Even simple AI judges outperformed humans in distinguishing AI from human responses. Imitation ability showed minimal correlation with conventional AI performance metrics, suggesting that passing as human is an important independent evaluation criterion. The large-scale Turing-like datasets and metrics introduced here offer valuable benchmarks for assessing human-likeness in AI and highlight the importance of rigorous, quantitative imitation tests for AI development. Zhang et al. systematically compare humans and AI across three language and three vision tasks.
In plain words AI summary
The authors tested 37 AI agents and 636 humans on six tasks: three language tasks and three vision tasks. Current AI systems are getting close to convincingly pretending to be human in both language and vision tasks. Simple AI judges did better than humans at spotting which responses came from AI. The authors suggest that passing as human is a separate and important way to evaluate AI.
Written by Kimi K2.5 using only the abstract above. Jev checked that every sentence is supported by the abstract sentences it cites, and removed any that were not. Point at a sentence to highlight its source. 1 sentence was removed for not being supported by the abstract.
Appeared: Wednesday, October 28. Nature Human Behaviour. Peer-reviewed journal.