Why Predicting Neuroscience Is Unreasonably Hard
Neuroscience spans psychology and biology, from cellular and molecular processes to anatomy, physiology, and behavior. Its breadth makes it difficult for any one researcher to hold a complete picture of what the field knows. Psychology Today’s overview of neuroscience describes the field’s wide scope.
Researchers also use a diverse range of methods, including brain imaging, cellular studies, and pharmacological interventions. Findings can be difficult to anticipate across that methodological and biological range. BrainBench was designed to test that challenge directly: how well can people and AI systems predict the results reported by neuroscience studies? UCL’s report on the BrainBench study
BrainBench: A Test Built to Expose Expert Blindness
A team led by Dr. Ken Luo at UCL, with senior author Professor Bradley Love, developed BrainBench, a benchmark for comparing how well people and AI systems forecast neuroscience research findings. The work was published in Nature Human Behaviour. UCL’s study report
In the benchmark, participants were given a study’s abstract and research question, then asked to choose which of two candidate results matched the study’s actual finding. The human group comprised 171 neuroscience experts who passed an expertise screening test. The researchers also evaluated 15 large language models, including versions from the Llama, Galactica, Falcon, and Mistral families. UCL’s study report; Psychology Today’s account of BrainBench
The test covered five areas: behavioral and cognitive neuroscience; systems and circuits; neurobiology of disease; development, plasticity, and repair; and cellular and molecular neuroscience. The researchers selected papers that, as far as they could determine, had not been included in the models’ training data. This screening was intended to reduce the possibility that a model was simply recalling a paper’s published result. UCL’s study report; Psychology Today’s account of BrainBench
The Results: AI Leads by Nearly 20 Percentage Points
Across the benchmark, the language models averaged 81.4% accuracy in identifying the actual study result, compared with 63.4% for the human experts. The researchers also examined experts who reported the highest level of expertise in the relevant neuroscience area. That subgroup scored about 66%—higher than the full human group, but still below the models’ average. UCL’s study report; Psychology Today’s account of BrainBench
The team also fine-tuned a version of the open-source Mistral model on two decades of neuroscience literature, producing a specialized model they called BrainGPT. BrainGPT reached 86% accuracy, compared with 83% for the general-purpose Mistral version evaluated in the study. UCL’s study report; Psychology Today’s account of BrainBench
The study also reported a relationship between model confidence and correctness: models were more likely to be right when more confident. That result suggests confidence could be relevant to human–AI collaboration, but it does not establish that a model can reliably recognize every situation in which it is wrong. UCL’s study report
What the Benchmark Does, and Doesn’t, Show
BrainBench tests predictions of reported study outcomes based on abstracts; it does not test whether a model can design experiments, explain biological mechanisms, or establish that a reported result is true. The forced-choice task is also narrower than the open-ended judgments researchers make in practice. The benchmark’s findings should therefore be understood as evidence about performance on this particular prediction task, not proof that AI understands the brain better than neuroscientists. UCL’s study report; Psychology Today’s account of BrainBench
The results are consistent with language models capturing patterns in scientific literature that can help forecast study outcomes. Such systems might assist researchers in considering relevant precedent or identifying predictions that warrant scrutiny. But a forecast is not an explanation: neuroscience still depends on experiments, careful measurement, replication, and expert interpretation.
Love argued that the models’ performance raises questions about novelty in science, saying that “a great deal of science is not truly novel, but conforms to existing patterns of results in the literature.” That is his interpretation of the findings, not a separate conclusion established by the benchmark. Psychology Today’s account of BrainBench
Why This Matters for the Future of Neuroscience
BrainBench shows that, on this benchmark, language models predicted reported neuroscience study results more accurately than the participating experts. It does not establish that models can replace scientific expertise. Rather, it offers a reason to investigate whether computational tools can help researchers identify patterns in a large and growing literature—and to examine what information those tools rely on. UCL’s study report
Love has predicted that scientists will use AI tools to help design effective experiments and suggested that the approach could extend beyond neuroscience. Those are expectations about possible future uses, not outcomes tested by BrainBench. If prediction tools do help researchers prioritize experiments, their value will depend on transparency, validation, and clear limits; a high forecast accuracy on one task does not by itself show that an experiment is unnecessary. Psychology Today’s account of BrainBench
The central lesson is not that expertise is obsolete. It is that carefully evaluated computational tools may help researchers notice patterns that are difficult to discern from individual experience alone—and that their predictions should be tested and interpreted alongside scientific judgment.
Sources
- University College London, via ScienceDaily: “AI can predict study results better than human experts, researchers find” (November 27, 2024), https://www.sciencedaily.com/releases/2024/11/241127140027.htm
- Psychology Today, “The Future Brain” blog: “AI Predicts Neuroscience Study Results Better Than Experts,” https://www.psychologytoday.com/us/blog/the-future-brain/202412/ai-predicts-neuroscience-study-results-better-than-experts
- Psychology Today, “Neuroscience” (background on the field), https://www.psychologytoday.com/us/basics/neuroscience
Explore how psychology and cognitive science shaped artificial intelligence.