Beyond the Atomic Trace: Why Cohort-Based Monitoring is the Future of AI Evaluation
We spend an inordinate amount of time chasing the "perfect" AI conversation trace. We scrutinize the token generation, the latency, the tool-calling precision. It feels good. It’s quantifiable. But as leaders from LangChain, Conviva, and CoreWeave recently pointed out at VB Transform 2026, it’s also fundamentally misleading.
Scoring individual traces in isolation is like judging the health of a crowd by interviewing one person. You might find a perfectly healthy individual, but if you look at the aggregate data, you might discover an epidemic. Relying on single-output evaluation doesn't just miss the point; it hides broken products. The industry is currently trapped in the "atomic fallacy," believing that if the individual atom of the conversation works, the molecule of the product is sound. It isn't.
The Illusion of Atomic Precision
The atomic nature of LLM evaluation—grading each prompt-response pair in a vacuum—is the industry’s greatest temptation. It’s easy to benchmark. It’s easy to automate. It creates a neat, satisfying spreadsheet of scores that teams can proudly present to management.
But as Hui Zhang, CTO of Conviva, aptly put it, the industry is forcing itself to pick between two poisons: ungrounded, scalable automation (like LLM-as-judge) or unscalable, human-verified accuracy. When we optimize for the former, we lose the context of the user journey.
"You have scalable but ungrounded—you grade the outcome, you grade the work," Zhang said. "It still is very difficult to ground it, and then you use humans and that's just not scalable."
The conversation trace is a snapshot, not a movie. It tells you what happened in that sliver of time, but it doesn't tell you why, to whom, or what happened before.
Building Evals as Living Product Requirements
That gap—a conversation that scores well but still signals a broken product—is what teams tirelessly try to close by building an exhaustive evaluation suite before they launch. Harrison Chase, CEO of LangChain, argues that this approach is counterproductive.
"We sometimes see teams that have almost eval paralysis," Chase noted. "They feel that they can't launch because they haven't achieved some perfect, static eval set. The best teams launch and then iterate."
Instead of treating evals as a one-time final exam, teams should treat them as a living product specification—a real-time product requirements document (PRD).
Emmanuel Turlay, director of engineering at CoreWeave, echoes this sentiment from a developer’s perspective. He recounted trying to reach 100% test coverage, only to find that his production environment was still riddled with bugs. A robust test suite doesn't guarantee a robust product if the suite itself is built on the wrong assumptions.
The solution is not to build a massive, exhaustive pre-launch test. It is to implement broad, always-on monitoring that detects real-world failure patterns as they occur, then build targeted, offline evaluation sets around those specific, observed problems.
Contrastive Analysis: Uncovering Systemic Issues
If individual trace scoring is flawed, what replaces it? The answer lies in contrastive analysis, a method of comparing user cohorts against a baseline to identify signals that simply aren't visible in a single conversation.
Zhang provided a compelling retail example. Imagine a shopper asking an agent for a running shoe. The agent asks qualifying questions, and the shopper completes the purchase. Scored individually, this trace looks perfect.
However, when you zoom out to look at the entire population of shoppers in that category, you might notice something: the clarification ratio—the frequency with which the agent asks follow-up questions—is three times higher than the baseline. Similarly, the rate at which shoppers abandon the agent to finish their purchase elsewhere might be five times higher than expected.
A single trace will never reveal this. You need that cohort-based comparison to see that a specific category has a measurable, debuggable, and systematic problem. Furthermore, the industry is missing a critical piece of the puzzle: the data surrounding the conversation—the events that happen before, between, and after the interaction.
Finding the Right-Sized Judge
Once you’ve successfully used contrastive analysis to pin down where the product is failing, the next question is how to monitor it efficiently. Turlay suggests a simple rule: start with the most capable model to prove the task is solvable, then downsize. If it won't work on a top-tier model, it won't work on a smaller one.
Once a failure pattern is clearly defined, you don't need to run every traffic interaction through your heaviest, most expensive judge model. Fine-tuning smaller models—like those from the Qwen family, as LangChain demonstrated—can achieve performance comparable to top-tier closed models for specific tasks, often with 10 to 100 times the cost reduction.
And, crucially, not every guardrail requires a large language model. As Chase pointed out, many of the most robust guardrails in tools like Claude Code are simply regex-based patterns. They’re efficient, predictable, and frankly, often perform better than a small, hallucination-prone LLM-as-judge.
Accountability, Trust, and the Human-in-the-Loop
The final tension, of course, is the role of human oversight. Does LLM-as-judge replace the human in the loop? Absolutely not.
Accountability is the bottleneck. Drawing on his experience in the self-driving car industry, Turlay highlighted that even in highly automated environments, a human must ultimately endorse the system’s safety. High-stakes domains like legal, finance, and healthcare will require this for years to come.
Automation eventually runs at a scale that beats human accuracy, but as Zhang noted, humans remain the essential guardians of corner cases and anomalous behaviors. And beyond simple error-checking, Chase argues that human involvement is essential for the fundamental processes of memory, learning, and trust-building.
The goal isn't to remove the human; it’s to free them to focus on the systematic, high-level issues that machines aren't yet equipped to solve, rather than checking individual traces one by one. The future of AI evaluation isn't in perfecting the trace; it's in understanding the system.
