pipette
ESEspañol

Benchmarking four large language models on generating and selecting hypotheses that match published findings in 50 studies of ageing biology

Y. Poteshkin, E. Udodov, M. Batin

Preprint

In the authors' words

Objective We asked how often hypotheses generated by large language models for 50 research tasks matched the withheld finding of the published study, and whether model forecasts could identify the matching hypotheses. Methods We built tasks from 50 peer-reviewed studies in ageing biology published after the models' stated training cutoffs; each task gave the background and a research question and withheld the study's finding (the reference finding). GPT-6 Astra, Grok 4.5, Claude Opus 5.5, and Gemini 4 Argon each generated five hypotheses per task by three methods: a single request, a multistep pipeline, and the Omega Point method without literature search (2997 hypotheses). A hypothesis matched the reference finding if at least two of three judge models from other developers scored it at least 3 (primary threshold) or at least 4 (strict threshold) on a 0 to 5 scale; refusals counted as non-matches. A selection rule, chosen from 112 variants evaluated on the Grok 4.5 hypotheses, kept only the hypotheses for which all four predictor models forecast experimental confirmation; the predictors never saw the published finding. The controls were recorded random answers and each generator's first-listed hypothesis. Confidence intervals (95% CIs) were calculated by cluster bootstrap over tasks. Results Of the hypotheses generated by a single request, 13.2% to 18.0% matched the reference finding, compared with 11.2% to 18.8% for the multistep pipeline and 2.8% to 7.6% for the Omega Point method. With a single request, the hypothesis listed first matched in 37.5% of cases (75/200), and the hypotheses in positions 2 to 5 matched in 10.1% (81/800). Among the hypotheses selected unanimously, the match rate was 73.7% (14/19) for Grok 4.5, 47.4% (9/19) for Gemini 4 Argon, and 63.6% (14/22) for Claude Opus 5.5, compared with 12.1% to 17.9% for the random control and 37.5% to 40.0% for the first-listed hypotheses. The selected lists captured 27.3% to 35.0% of all matching hypotheses. Pooled across the three sets, the match rate among selected hypotheses exceeded that among first-listed hypotheses by 23.2 percentage points (pp; 95% CI [13.0, 32.6]). When Gemini 4 Argon re-extracted the reference findings, the match rate among selected hypotheses was 59.6% (34/57), compared with 61.4% (35/57) against the original reference findings for the same tasks. In a manual audit with the generator, judge scores, and selection status hidden, our classifications agreed with the judges' at the primary threshold in 81/100 cases (Cohen's kappa 0.62). Agreement was 89.7% after weighting the sample strata by their size in the corpus. We classified 19 of 30 sampled selected hypotheses as matches, and the judges classified 20 of 30. Conclusions In the evaluated sets, unanimous selection by the four predictors increased the match rate among selected hypotheses; the resulting match rate depended on the matching threshold. Generators listed a matching hypothesis first more often than in later positions, and 50 of the 60 selected hypotheses were listed first. The advantage over the first-listed hypotheses and over the best single predictor at equal list length was statistically significant for the Grok and Opus hypotheses but not for the Gemini hypotheses, and the pairwise differences of the advantage between generators were not statistically significant (p >= 0.16 after Holm correction). We selected the rule using the Grok hypotheses and evaluated it on the Gemini and Opus hypotheses for the same 50 tasks; it has not been tested on new tasks.

Main resultLimitation the authors admit

Appeared: Thursday, November 19. bioRxiv. Preprint, not yet peer-reviewed.

DOI: 10.64898/2026.11.17.767323