The Metric Obsession That’s Killing Strategy
A founder I spoke to last month pulled up his dashboard, grinning. Forty-one tests. Running right now. He was proud. I asked him to name three of those forty-one that had changed a real business decision in the past quarter.
He went quiet.
That’s the trap. When AI makes it effectively free to generate variants, it also makes it almost impossible not to run tests. But velocity isn't value. Growth teams are increasingly trapped by "experimentation theater"—launching dozens of concurrent A/B tests because the tools have made building variants fast, while failing to extract actual, meaningful strategy from the results. If your growth velocity is measured solely by test volume and not by strategic impact, you’re just building a faster machine to be wrong. You aren't learning; you're just generating noise.
The Miro-Reforge Pivot: Judgment as the New Bottleneck
The AI boom has fundamentally shifted what’s actually hard. A few years ago, the headache was coding the variant, briefing the designer, and hooking up the tracking pixels. Now, you can spin up that variant in seconds. The bottleneck has moved entirely to judgment.
This is exactly why Miro’s acquisition of Reforge—a move aimed squarely at solving what happens when teams move faster but in the wrong direction—was so critical. Miro's CEO, Andrey Khusid, nailed it when he said the biggest opportunity isn’t just moving quicker, but ensuring you’re moving in the right direction. AI has completely eroded the technical execution barrier. The result? Teams are moving fast, but they’re building the wrong features because they’ve skipped the actual strategy part.
When you're building products at scale—whether you're in Productivity software, Cloud computing, or Gaming—the challenge isn't "can we build this feature?" it's "should we?" Reforge founder Brian Balfour has been clear: reliance on gut intuition isn't enough anymore. You need to master AI-assisted evaluations and rigorous strategy, not just the technical plumbing of deployment.
Plumbing Belongs to Machines, Verdicts Belong to Humans
If machines handle the plumbing, then why are we letting them handle the strategy?
The industry-wide move toward automated statistical analysis—highlighted by high-profile deals like the $1.1 billion Statsig acquisition—shows we've solved the statistical labor problem. Modern platforms now handle SDK deployments, warehouse-native computations, and variance reduction with impressive speed. This is a blessing for Computing productivity.
But there’s a massive catch. While these tools can tell you if a variant is better, they can't tell you if the variant matters to your business.
Automated engines are great at optimizing micro-conversions. But if you’re building a gaming engine or enterprise apps in the Cloud, optimizing for the wrong metric—like page-level clicks rather than long-term retention or lifetime value—is a quick way to destroy revenue. The ultimate hypothesis, the core metric guardrails, and the final scale-or-kill judgments still have to be human-in-the-loop tasks. When you outsource your metrics definition to an autonomous model, don't be surprised when you end up with a dashboard full of vanity metrics and a roadmap that’s heading nowhere.
Building a High-Threshold Framework
Real experimentation isn't a factory-line process; it’s a high-stakes filtering system. If it’s cheap to run, you have to make it expensive to enter.
Stop treating every idea as a test proposal. Instead, score potential tests against three specific factors:
- Size of a Win: If this works, does it move the needle on a real metric, like revenue, churn, or core activation? If not, it’s not worth running.
- Confidence Level: What do we already know? If we’re launching a random test on a hunch, the bar should be sky-high.
- Cost to Run: What’s the total organizational cost of managing this test, including the analysis time? If a test takes longer to analyze than it takes to build, you shouldn't be running it.
Microsoft’s own approach to AI-driven productivity shows the path: use the tools to scale your speed, but rely on human-governed strategy to decide which tests are worth the effort.
Consolidation is your friend. If you’re currently running 41 tests, consolidate down to 6—maybe even fewer. If a test doesn't pass the high-threshold filter, kill it immediately to stop the noise. Maintaining a disciplined weekly review to force a scale-or-kill decision isn't optional; it's the only way to ensure your AI-powered experimentation framework actually changes your business instead of just inflating your dashboard numbers.
The goal isn't more tests. The goal is better decisions. Start acting like it.
