Introduction: LLM-Powered Decoding of Human Thought
A new study combines Large Language Models and behavioral mathematics to analyze human decision-making text data at scale, using fine-tuned LLMs to decode thousands of free-text justifications and validate them against objective choice mathematics. The research, published by a team at SynoSys and reviewed in PNAS, demonstrates that verbal reports — when analyzed with LLMs — become a reliable, scalable data source that reveals how and why people make choices.
The Experiment: Gambling, Choices, and Written Justifications
During simulated gambling rounds with shifting risk parameters, participants were blocked from simply clicking a choice; they were required to actively describe their internal thought processes and subjective justifications in their own words after every single round. This design produced a rich dataset of free-text explanations, each paired with a precise mathematical record of the choice the participant made. The experiment showed that people's self-insights, when captured systematically, are highly reliable.
"Our understanding of human behavior, including decision making, can be deepened by asking people to elaborate on their thought processes," says lead author Dr. Kamil Fuławka, researcher at SynoSys. "However, the systematic analysis of such free-text data requires scalable and rigorous analytical frameworks — an endeavor that can now be supported by LLMs."
Building the Algorithmic Codebook
Drawing on decades of established behavioral finance and decision-making theories, the researchers built an extensive taxonomy of possible human rationales, ranging from hyper-focused optimization of the best possible outcome ("maximax heuristics") to intense avoidance of maximum devastation ("minimax loss aversion"). This codebook gave the team a structured vocabulary for tagging the thousands of text entries that would follow.
In the experiment, participants took part in gambling and had to explain each decision in their own words. To analyze these explanations, the researchers drew on existing theories and models of decision making to develop a large set of possible decision reasons, such as focusing on the best possible outcome or avoiding a big loss.
LLMs as Scalable Qualitative Auditors
Rather than relying on human research assistants to manually read and tag thousands of individual journal sheets, the team deployed fine-tuned LLMs. The models functioned as high-speed qualitative auditors, reading the free-text data at scale and instantly tagging the exact psychological reasons driving each entry. This approach unlocked the potential of verbal reports at a scale that manual coding could never achieve.
"Looking only at a person's final choice is like trying to understand a complex murder mystery by only looking at the very last page of the book. Traditional behavioral economics tracks what button someone clicks or what product they buy, but it remains completely blind to the hidden thought processes, doubts, and mental shortcuts that led to that action."
Validating Text Against Mathematical Choices
To verify that the LLM's text classifications weren't hallucinated or arbitrary, the researchers cross-referenced the model's text tags with objective mathematical modeling of the participants' actual physical choices. The choice equations validated the text profiles with remarkable precision, confirming that what people said they were doing aligned perfectly with how they acted. This validation layer was the most brilliant and vital part of the study.
How did the researchers make sure the AI wasn't just making things up when it read people's explanations? The team used objective choice mathematics as a strict validation shield. They took the psychological reasons the AI discovered in a participant's text, like "trying to avoid a catastrophic loss", and plugged that exact rationale into a separate mathematical algorithm tracking the participant's actual physical gambling behavior. The math and the text aligned perfectly: the choices people made matched up precisely with the written motivations the AI extracted from their journals, proving the LLM's scale is highly accurate and scientifically sound.
Dynamic Shifting of Decision Strategies
The unified data clearly demonstrated that human decision-making strategies are not permanent personality traits. Instead, individuals are highly adaptive: they dynamically and systematically shift their reasoning profiles from round to round based on how a problem is presented. This finding challenges the idea that decision styles are fixed and shows that context shapes strategy in real time.
The unified data clearly demonstrated that human decision-making strategies are not permanent personality traits. Instead, individuals are highly adaptive: they dynamically and systematically shift their reasoning profiles from round to round based on how a problem is presented.
A New Toolkit for Public Policy
Dr. Fuławka stresses that this automated analytical framework opens up vast horizons for studying human behavior inside complex, real-world ecosystems. By allowing researchers to parse massive amounts of free-text public feedback, policymakers can better understand how communities interpret and simplify complicated trade-offs in public health, economic planning, and technological adaptation.
In such settings, people's own explanations may be especially valuable for revealing how they simplify complex problems, focus on particular pieces of information, and adaptively use simple decision strategies. The framework presented in the study shows how LLMs can help researchers analyze these explanations at scale, opening new opportunities to study human decision making in more realistic and complex environments.
Key Questions Answered
Q: Why do scientists need an LLM to study how people make decisions? Can't they just look at what people choose?
Looking only at a person's final choice is like trying to understand a complex murder mystery by only looking at the very last page of the book. Traditional behavioral economics tracks what button someone clicks or what product they buy, but it remains completely blind to the hidden thought processes, doubts, and mental shortcuts that led to that action. While researchers have always wanted to read people's written explanations to understand their true motives, manual reading is a massive bottleneck. By using an LLM as a high-speed, automated reader, scientists can now analyze thousands of detailed personal journals instantly, giving them an unprecedented look inside the human mind at a massive scale.
Q: How did the researchers make sure the AI wasn't just making things up when it read people's explanations?
This was the most brilliant and vital layer of the SynoSys experiment. To ensure the LLM was accurately mapping human psychology rather than generating "hallucinations," the team used objective choice mathematics as a strict validation shield. They took the psychological reasons the AI discovered in a participant's text, like "trying to avoid a catastrophic loss", and plugged that exact rationale into a separate mathematical algorithm tracking the participant's actual physical gambling behavior. The math and the text aligned perfectly: the choices people made matched up precisely with the written motivations the AI extracted from their journals, proving the LLM's scale is highly accurate and scientifically sound.
Q: How can this new AI framework change how governments design public policies or financial tools?
Most monumental choices in life, like planning for retirement, choosing a cancer treatment, or adapting to new technology, involve incredibly messy trade-offs that don't fit into a simple multiple-choice question. Historically, governments and banks have struggled to analyze mass public surveys because coding open-ended feedback is too slow. This new framework allows public policy analysts to deploy LLMs to immediately read and quantify thousands of complex, written responses from real citizens. This reveals exactly how the public simplifies complicated problems, what specific pieces of information they focus on, and how to design safer, clearer, and more supportive programs that match real human behavior.
Conclusion
The framework presented in the study also demonstrates how LLMs can help researchers analyze these explanations on a large scale, thereby opening up innovative possibilities for studying human decision-making processes in more realistic and complex environments. By combining verbal reports, LLMs, and rigorous mathematical modeling, the research shows that people's own insights are a valuable data source — and that the reasons people rely on are not fixed, but shift systematically with the structure of the decision problem.
This article was edited by a Neuroscience News editor. Journal paper reviewed in full.
Related Research
- Why Every Security & Compliance Analyst Should Care About How LLMs Map Human Decision-Making
- When Your Mind Rewrites the Rules: How LLMs Prove Decisions Are Shaped by Problems, Not Personality
