If you are evaluating your enterprise AI agents by pulling 50 chat logs into a dashboard and asking an LLM judge to grade them one by one, you are likely flying blind. A conversation can look polite, helpful, and accurate in isolation while hiding a complete breakdown in your underlying product flow.
That core disconnect took center stage at VB Transform 2026. Harrison Chase (CEO of LangChain), Hui Zhang (CTO and co-founder of Conviva), and Emmanuel Turlay (Director of Engineering at CoreWeave) broke down why single-trace scoring fails at scale. The consensus across infrastructure, evaluation, and observability leaders is clear: moving beyond pilot projects requires shifting from individual trace scoring to contrastive cohort analysis, environment-aware benchmarking, and automated eval engineering.
The Hidden Trap of Isolated Trace Scoring
Standard AI evaluation relies on sampling—pulling a representative slice of conversation traces and scoring each from start to finish. It feels intuitive. But scoring isolated interactions creates a dangerous false positive rate.
Hui Zhang illustrated this with a retail scenario. A customer asks an agent for running shoe recommendations ahead of a half marathon. The agent asks several clarifying questions, suggests a pair, and the user eventually buys the shoes. Evaluated on its own, an LLM judge gives that trace high marks across answer quality, accuracy, and tone.
When Conviva analyzed that interaction across the full user cohort for the category, two critical problems emerged:
- Clarification ratio: The agent asked 3x more follow-up questions than baseline for that specific footwear category.
- External completion rate: Customers abandoned the agent to complete their purchase elsewhere at 5x the baseline rate.
"You have scalable but ungrounded, whether it's agents as judge or LLMs as judge—you grade the outcome, you grade the work," Zhang noted during the panel. "It still is very difficult to ground it, and then you use humans and that's just not scalable."
Neither metric was visible in a single trace. Both pointed to severe friction in the user experience. Looking at isolated conversations misses what happens before, between, and after user sessions—the broader behavioral context that determines product health.
Eval Paralysis vs. Living Product Specifications
Faced with unexpected production failures, engineering teams often react by trying to construct a massive, exhaustive test suite before releasing updates. Harrison Chase warned that this impulse frequently backfires.
"We sometimes see teams that have almost eval paralysis," Chase said. "They're like, 'this is an eval set, I can't launch it.' The best teams launch and then iterate."
Instead of treating evaluation suites as static QA gates, top teams treat evals as a living product requirements document (PRD). In Chase's view, evals define what an agent should and should not do as product requirements evolve.
Emmanuel Turlay reached the same conclusion from his work at CoreWeave and prior experience in high-stakes autonomous systems. "I was trying to reach 100% coverage for my tests, and I still had bugs in production," Turlay recalled.
Turlay recommended deploying wide, always-on online monitoring to capture failure classes directly from production traffic. Once real production failure modes surface, developers can build targeted offline evaluation benchmarks to prevent regressions.
Environment Grounding and Artifact Benchmarking
Simple text-in, text-out LLM evaluations fall apart when applied to autonomous agents. Agents do not just write text; they execute code, query databases, modify local filesystems, and call external APIs.
To evaluate real agent performance, the execution environment must be treated as a first-class component of the test. As detailed in LangChain's work on How We Benchmark Deep Agents, evaluation requires sandboxed execution environments running inside isolated Docker containers using frameworks like Harbor.
Judging an agent requires inspecting the artifacts and state changes created during task execution—not just evaluating the final text response. LangChain structures its evaluation across three specialized benchmark domains:
- Harbor-Index: 82 complex tasks distilled from thousands of candidates, testing long-horizon software engineering, data analysis, and tool usage.
- 𝜏³-bench: 30 multi-turn conversational tasks that evaluate agent decision-making against simulated user interactions.
- ContextBench: 30 retrieval tasks where the agent must navigate and join information across a sandboxed document corpus.
Because non-deterministic agents exhibit natural behavioral variance, reliable benchmarking requires running tasks multiple times. To maintain high engineering velocity, teams run a lightweight benchmark subset—roughly 8x faster and 6x cheaper—for daily iteration, saving full benchmark suites for major harness updates like prompt trimming or middleware refactoring.
Right-Sizing Judge Models and Guardrails
Running top-tier frontier models like Claude 3.5 Sonnet or GPT-4o as continuous judges over every production trace is economically unviable. CoreWeave's Turlay advised starting with the most capable frontier model to prove a task can be evaluated automatically, then working down the cost curve.
Once an evaluation pattern is proven, teams can sample a small fraction of production traffic and offload straightforward evaluation tasks to fine-tuned open-source models.
LangChain put this into practice by fine-tuning an open-source Qwen model to detect "perceived error"—moments where a user signals that an agent made a mistake. By combining hand-labeled traces with model distillation, Chase reported achieving accuracy on par with Sonnet while reducing inference costs by 10x to 100x depending on serving infrastructure.
Not every behavioral check requires an LLM. Chase pointed out that Anthropic's Claude Code relies on standard regular expressions (regex) for many of its core guardrails. Simple, deterministic pattern matching often outperforms LLM judges in speed, cost, and reliability.
Automated Eval Engineering and the Human Guarantee
Writing containerized evaluation tasks manually creates a major development bottleneck. To streamline this process, LangChain introduced the Eval Engineering Skill, enabling AI coding tools to inspect repository architecture, mine production traces, and generate standardized Harbor eval tasks automatically.
The most effective evals emerge through an interactive loop with human engineers rather than single-shot generation. Inspecting both the agent trajectory (messages and tool calls) and the verifier trajectory (reasoning and scores) is essential to catch "reward hacking"—where an agent exploits proxy metrics, such as over-citing sources or claiming unperformed actions, to pass tests without solving the task.
This creates a continuous development flywheel:
- Mine traces: Extract failure modes, tool errors, and unexpected user drop-offs from production logs.
- Build containerized evals: Generate executable Harbor tasks with isolated Docker environments and deterministic verification scripts.
- Refine agent harness: Adjust system prompts, fine-tune models, or alter middleware configurations.
- Re-evaluate: Run parallel benchmark suites to verify improvements without introducing regressions.
Despite rapid advances in automated evaluation, human sign-off remains indispensable for high-stakes applications. Turlay highlighted his experience building autonomous vehicle software, where data processing and model retraining operated on tight two-week cycles—yet human engineers still had to sign off before code reached production vehicles.
Whether in healthcare, finance, or enterprise infrastructure, human oversight provides the ultimate accountability layer while generating the feedback data agents need to learn over time.
Sources
- A single AI agent conversation can look perfect and still be broken, leaders from LangChain, Conviva and CoreWeave said at VB Transform 2026
- How We Benchmark Deep Agents
- Towards Automating Eval Engineering
