The landscape of product development is undergoing a profound transformation, with experimentation moving from a specialized function to a central strategic imperative for companies striving for sustained innovation. This was the resounding message delivered at the recent Circus Austin Experimentation Conference, where industry leaders converged to discuss the evolving role of rigorous testing and learning in an era defined by rapid technological advancements, particularly artificial intelligence. The keynote emphasized that as AI dramatically reduces the cost and time of building, the critical challenge shifts to "figuring out what to build," making comprehensive experimentation the indispensable engine for product success and business outcomes.
A New Paradigm in Product Operating Models
The discussion at the conference centered on the distinction between traditional "project models" and the more agile "product operating model." In the outdated project model, product roadmaps are dictated by stakeholders, focusing on delivering specific features and outputs. Conversely, the product model empowers teams with problems to solve, granting them autonomy to devise solutions that achieve measurable business results and outcomes. Experimentation, while present in both, primarily serves optimization in the project model. However, within the product model, it becomes the cornerstone of innovation, enabling teams to validate hypotheses and iterate rapidly towards impactful solutions. As Amazon founder Jeff Bezos famously articulated, "If you want to innovate, you simply must experiment."
The advent of AI further amplifies this necessity. While AI is streamlining the mechanics of planning, running, and evaluating experiments, its broader impact underscores the strategic importance of "building to learn." The conference highlighted that the traditional approach of rapid development followed by release into the wild often leads to frustrated customers, eroded trust, high churn, and, crucially, a lack of corresponding improvement in desired business outcomes—a phenomenon some refer to as the "AI productivity paradox." This growing awareness of high failure rates and the imperative to deliver real value is pushing experimentation to the forefront of organizational competencies.
A Historical Arc of Discovery: From Qualitative Roots to Quantitative Revolution
The journey of product experimentation traces back decades, evolving significantly with technological progress. Early forms of product discovery, dating back to the 1980s, were predominantly qualitative. Pioneers in the field engaged in iterative design processes, creating and testing physical or digital prototypes with small groups of customers. This initial phase focused on understanding user needs, usability, and fundamental value propositions through direct observation and feedback.
A pivotal shift occurred in the late 1990s with the explosive growth of the internet. The digital realm provided an unprecedented volume of user data, paving the way for quantitative experimentation. A seminal moment recalled by many early practitioners involved the ability to conduct A/B tests on live services with millions of daily active users. This capability allowed product teams to scientifically compare different designs or features, moving beyond opinion-based decisions. The guiding principle of this era, famously articulated by former Netscape CEO Jim Barksdale, "If we have data, let’s look at data. If all we have are opinions, let’s go with mine," perfectly encapsulated the new data-driven ethos. The internet not only enabled data collection but also democratized the tools for product teams to pursue innovative ideas and validate them with empirical evidence, thereby challenging hierarchical decision-making.
Today, AI represents the third major inflection point, fundamentally altering the calculus of product development. The time and cost associated with building and shipping new capabilities have plummeted, allowing companies to deploy more features faster than ever before. However, this increased output has not always translated into better outcomes, making the strategic role of experimentation — discerning what to build — more critical than ever.
The Three Pillars of Product Work and Their Experimental Imperatives
Leading product organizations strategically deploy different types of experiments across three distinct categories of product work:
- Creating New Products ("Zero to One"): This involves developing entirely new offerings where the risks are highest. Beyond typical product discovery risks like value, usability, viability, and feasibility, new products demand a comprehensive assessment of cost structures, monetization strategies, and go-to-market approaches to achieve product-market fit. Experiments in this phase are often primarily qualitative, leveraging prototypes and user interviews to rapidly learn and iterate, guiding significant shifts in direction before substantial investment in full-scale development. The emphasis is on deep insights from early interactions rather than statistical significance from large user bases.
- Improving Existing Products – Innovation: Once a product is established, the focus shifts to sustaining and growing the business through continuous innovation. This encompasses creating new value by expanding problem-solving capabilities for customers or the company, enhancing existing solutions, or targeting adjacent markets. This work combines both rapid qualitative learning through prototypes and early-stage quantitative experiments, leveraging the existing customer base to validate new features and improvements. The goal is to continuously expand the product’s value proposition.
- Improving Existing Products – Optimization: This category aims to maximize the potential of a live product by reducing friction, improving clarity, and optimizing elements like pricing. Characterized by primarily quantitative, data-driven experiments, this work often involves A/B testing minor changes. While individual impacts might seem small, their aggregation across thousands or millions of users can yield substantial business results. This is where traditional CRO (Conversion Rate Optimization) teams often focus their efforts.
Across all three types of work, the overarching goal of experimentation remains the same: to maximize learning. Quantitative tests establish causality and quantify outcomes, while qualitative tests uncover why customers behave as they do, identifying underlying motivations and revealing new opportunities. As Product Discovery Coach Tom Chi eloquently states, "Learning why it worked is way more important than whether it worked. Why it worked is an enduring property… it comes from inside the person you’re trying to serve."
AI’s Impact on the Experimentation Landscape
Artificial intelligence is not just a subject of experimentation but also a powerful tool for enhancing it. In product discovery, AI assists across numerous steps, from analyzing and synthesizing vast user data to generating live-data prototypes and even automating aspects of test administration and result evaluation. For optimization, AI is rapidly automating tasks such as generating experiment variants, calculating power and sample sizes, checking instrumentation, monitoring key metrics, and detecting anomalies.
However, the integration of AI also introduces new complexities, particularly when building or improving AI-driven (non-deterministic) products. Here, product discovery must incorporate rigorous evaluation (eval) processes, as many judgments about AI product quality are inherently subjective. Furthermore, quantitative A/B tests for AI products can be more expensive and time-consuming, with conclusions potentially less durable due to the dynamic nature of AI models. The strategy shifts to using evals for earlier quality assessment, reserving A/B tests for establishing core value propositions.
Organizational Evolution: Bridging the Qualitative-Quantitative Divide
Historically, experimentation functions within companies have often been siloed. User research teams, typically within product design, focused on qualitative methods. Data analysis and data science groups, often aligned with product management, handled quantitative work. A third group, CRO or optimization teams, sometimes sat within product growth or marketing. This fragmentation, however, hinders the holistic learning necessary for true innovation.
The conference underscored the urgent need to address this organizational politics and integrate these functions. A common challenge is the cultural bias towards either qualitative or quantitative methods. However, leading companies recognize that both are essential, creating a synergy where "1 + 1 > 2." AI is now blurring the lines, enabling qualitative experiments to scale, and bringing the rigor of experimentation professionals to subjective eval processes. The ability to understand both what happened and why it happened is paramount for sustained success.
The second critical dimension revolves around who runs the experiments. Three models typically emerge:
- Agency Model: Product teams request experiments from a centralized service organization, receiving findings after the fact. This can lead to detachment and a lack of ownership over the learnings.
- Autonomous/Democratized Model: Product teams run experiments themselves. While fostering ownership, this model often suffers from a lack of expertise within individual teams, leading to flawed methodologies or misinterpretations.
- Hybrid/Enablement Model: This approach, widely advocated by experts, empowers product teams to own their experimentation while receiving coaching, support, and services from a central team of experimentation professionals.
The hybrid model offers significant advantages. It ensures product teams internalize learnings, as they are directly involved in the process, while benefiting from expert guidance on experimental design, managing test subjects, and reporting. This model also addresses a long-standing complaint from experimentation professionals: "not being in the room where it happens." By working embedded within product teams, they actively influence strategic decisions and contribute directly to outcome creation, rather than merely reporting on past results. The enablement charter of such a team is to equip product teams with the knowledge and tools to conduct intelligent and effective experiments, elevating the entire organization’s competency.
The Future: Experimentation as a Core Business Competency
As AI continues to drive down the cost of building, the competitive advantage will increasingly lie in the ability to effectively discover and validate what to build. This elevates experimentation to a core competency for all product organizations committed to innovation. For many companies, this entails a significant evolution: moving beyond a niche data science or CRO team, often measured by the volume of A/B tests, to a pervasive product company competency measured by tangible business and customer impact.
This transformation necessitates embracing the full spectrum of qualitative and quantitative experimentation, not just for value capture (optimization) but crucially for value creation (innovation). It means expanding the scope beyond merely proving outcomes to actively influencing them, and shifting from determining "which version won" to discerning "what we should build and why."
The future demands experimentation professionals with deep expertise across the entire range of qualitative and quantitative techniques applicable to diverse product work. These professionals will serve as strategic partners, assisting product teams not just in validating solutions but in collaboratively creating products that inherently drive desired outcomes. The Circus Austin Experimentation Conference highlighted that this shift is not merely an operational adjustment but a strategic imperative, positioning experimentation as the central pillar of effective product discovery and sustainable success in the AI-driven economy. Companies that fail to adapt risk being outmaneuvered by competitors who master the art and science of continuous learning and validated innovation.
