The narrative around AI productivity is intoxicatingly simple: plug in a Large Language Model (LLM), watch the code generation rates soar, and celebrate the ROI.
But if you talk to seasoned engineering leaders—like Zillow’s Senior Vice President of Engineering, Toby Roberts—that narrative is a dangerous oversimplification. At this year's VB Transform 2026, Roberts shared a sobering reality: Zillow’s reported 40% increase in code shipping wasn't a product of magic AI pixie dust. It was credible, he argued, precisely because the team had rigorously measured their performance against DORA (DevOps Research and Assessment) metrics years before they deployed a single agent in production.
This serves as a vital wake-up call for every enterprise betting on AI to overhaul their development lifecycle. Without that pre-existing baseline, you aren’t measuring AI productivity; you’re measuring noise.
The Illusion of "AI-First" Engineering
It’s easy to understand why so many teams jump straight to the "AI-First" strategy. The promise is immediate, the pressure from stakeholders is intense, and the technology is undeniably powerful.
However, treating AI as a panacea for engineering efficiency is a mistake. When you lack a clear understanding of your software delivery lifecycle—from planning and coding to build, test, and release—you lose the ability to isolate the specific impact of your AI tools. Was that 40% speed-up really down to the coding assistant, or was it a result of a process optimization effort you ran last quarter? Without valid historic data, the answer remains a guess.
As Zillow recognized, you need to know how you work before you change how you work.
DORA Metrics as the Ground Truth
For the uninitiated, DORA metrics—which include measures like deployment frequency, lead time for changes, change failure rate, and time to restore service—provide the industry gold standard for quantifying software delivery performance.
Zillow’s strategy was simple: treat these DevOps lifecycle metrics as the immutable truth. By establishing these baselines before the AI push, they constructed a defensible framework. When they turned on their AI-assisted tooling, they weren't grasping at anecdotal sentiment; they were tracking concrete, historical performance metrics compared to their known delivery baseline. This is the difference between data-driven engineering and marketing-driven fiction.
Moving from Single Model APIs to Persistent Context
Once you’ve established your baselines, the challenge shifts to architecture. For Zillow, the hardest problem wasn't the AI model itself—it was maintaining persistent context across the complex, multi-month journeys of their customers.
Zillow touches roughly 80% of U.S. residential transactions. A customer could move from skimming property listings on their phone to engaging with a loan officer and eventually a real estate agent. Relying on a single chatbot interface to bridge that journey is naïve; it lacks the necessary continuity.
Instead of routing customers through a one-size-fits-all model API, Zillow built its own architecture harness. By leveraging 20 years of ML history, they prioritized using smaller, task-specific fine-tuned models over one general-purpose behemoth. This isn't just about efficiency—it’s about coherence. Enterprise AI must understand the nuances of the business, not just the nuances of human language.
Controlling the Cost of Enterprise Intelligence
Centralization is often frowned upon in agile environments, but in the context of enterprise AI, it is a necessary governance strategy.
Zillow operates thousands of Glean agents in production, executing tens of thousands of tasks across the company. As Glean CEO Arvind Jain pointed out, the alternative to this centralized approach is a fragmented sprawl where finance, legal, and marketing teams each rebuild their own connections to the same data sources. This not only invites redundancy; it creates massive security and operational debt.
Centralization as a Governance Strategy
The power of a platform like Glean lies in its ability to centralize this integration work once, via an AI gateway (such as the Model Context Protocol). This centralization transforms how the enterprise handles cost. By implementing smart model routing—sending most tasks to smaller, cheaper models—and utilizing precomputed context, Zillow can significantly slash token consumption.
As Jain noted, simply having a powerful model isn't enough to deliver enterprise-ready automation. "You do have to connect it with your enterprise context," he said. And that connection needs to be managed efficiently to avoid burning unnecessary compute resources on token assembly.
Why Permission Inheritance Isn't Enough
Perhaps the most critical, often overlooked aspect of enterprise AI is security. It is tempting to trust your existing permission structures within the enterprise context layer. After all, if a user doesn't have access to the file, the agent shouldn't have access to it either, right?
Zillow’s approach warns against this complacency. Even with a permissions-aware platform, they implemented hard compliance rules and standing checks over their most sensitive data categories. The enterprise data landscape is messy. Trusting inherited permissions inside a new AI context layer is an invitation to data leaks.
Conclusion: A Foundation for Long-Term Rigor
The lesson from Zillow is clear: Enterprise AI is not a shortcut out of engineering discipline. If anything, it demands more rigor.
To achieve meaningful, defensible ROI from agentic AI, you must build on the bedrock of solid engineering metrics, thoughtful architecture that prioritizes persistent context, and centralized governance that secures data beyond simple inheritance models.
AI tools will continue to evolve, and their capabilities will undeniably grow. But your process baselines and your governance architecture? Those are the infrastructure that will determine whether your AI initiative is a sustainable success or a costly, unmeasured experiment.
