Austin, TX – The annual Circus Austin Experimentation Conference recently convened, bringing together product leaders, data scientists, and innovators to discuss the evolving landscape of product development. A central theme emerged from the keynote address, highlighting a critical shift in how organizations perceive and utilize experimentation. The discourse underscored that experimentation, traditionally confined to product optimization or marketing A/B tests, is rapidly expanding its scope to become the core competency for product organizations, especially in an era increasingly dominated by Artificial Intelligence. This paradigm shift emphasizes a holistic approach to learning, moving beyond mere outcome validation to actively driving innovation and strategic product discovery.
Redefining Experimentation in the Product Operating Model
The keynote presentation, delivered to a captivated audience of hundreds, meticulously laid out a broader definition of experimentation. Far from being a niche activity, an experiment, in its most fundamental sense, is a structured endeavor designed to learn something new. This learning can range from establishing causality, proving an outcome, understanding user confusion, or even comprehending compliance constraints. Crucially, many experiments are proactively deployed to prevent the costly development of the wrong solutions, saving significant resources and time.
The discussion prominently featured the distinction between the traditional "project model" and the modern "product operating model." In the outdated project model, stakeholders dictate product roadmaps filled with features and projects, tasking product teams primarily with output delivery. Experimentation in this context is often relegated to post-launch optimization—tweaking existing features for marginal gains.
In stark contrast, the product operating model empowers product teams with problems to solve, granting them autonomy to devise solutions that achieve specific business outcomes. Here, experimentation transcends optimization, becoming an indispensable tool for continuous innovation. As Amazon founder Jeff Bezos famously stated, "If you want to innovate, you simply must experiment." This philosophy underpins the product model, where consistent innovation is the linchpin for sustained success and outcome delivery. The integration of AI further amplifies this need, as the speed and cost of building capabilities decrease, placing an unprecedented premium on knowing what to build.
A Historical Arc: From Qualitative Insights to AI-Driven Discovery
The evolution of product experimentation offers a compelling chronology of technological and methodological advancements. Early forms of product discovery in the 1980s were predominantly qualitative. Pioneer product developers relied heavily on creating prototypes and conducting direct user tests, observing reactions and gathering feedback to refine concepts. This "build-to-learn" approach, though rudimentary by today’s standards, was foundational for understanding user needs and identifying potential pitfalls before significant investment. These initial qualitative experiments, though lacking the statistical rigor of later methods, were crucial for generating profound insights and guiding fundamental design choices.
A pivotal shift occurred with the advent of the internet in the late 1990s. The digital realm ushered in an era of unprecedented data collection capabilities. The keynote speaker recounted a personal anecdote from 1997 at Netscape Communications, where a design dispute between an executive and a product designer was resolved using an A/B test. Leveraging the platform’s millions of daily active users, the team could statistically determine which design variant performed better. This moment crystallized the power of quantitative experimentation, making data-informed decisions viable on a large scale. Jim Barksdale, then CEO of Netscape, encapsulated this new ethos with his memorable dictum: "If we have data, let’s look at data. If all we have are opinions, let’s go with mine." This principle resonated deeply, empowering product teams to overcome subjective biases with empirical evidence and attribute outcomes directly to specific changes.
Today, Artificial Intelligence is again reshaping the landscape. While AI streamlines the mechanics of planning, running, and evaluating many types of experiments, its more profound impact lies in redefining product development itself. The ease and speed of building and deploying new capabilities have dramatically increased, thanks to AI-powered tools. However, this velocity has inadvertently led to a "productivity paradox" in some instances, where rapid shipping of features does not correlate with improved outcomes. Companies initially attempted to build quickly and release into the wild, hoping something would "stick." This approach, however, has often resulted in frustrated customers, eroded trust, high churn rates, and, critically, a lack of tangible improvement in business metrics.
This realization has brought the importance of early-stage experimentation — figuring out what to build — into sharp focus. The critical skill is no longer merely building fast, but intelligently discovering solutions that genuinely deliver desired outcomes. Analysts suggest that companies embracing robust pre-build experimentation frameworks can reduce development waste by an estimated 20-30% and accelerate market validation by up to 50%. The next few years are anticipated to witness experimentation solidify its position as the core competency for effective product discovery and for successful product companies seeking sustainable innovation.
Experimentation Across the Product Lifecycle
Effective product organizations leverage different types of experiments across three distinct phases of product work:
-
Creating New Products (Zero to One): This phase involves the broadest range of risks. Beyond the typical product discovery risks of value (do customers want it?), usability (can they use it?), viability (can the business sustain it?), and feasibility (can we build it?), new product creation demands rigorous assessment of cost structure, monetization strategy, and go-to-market execution. The ultimate goal here is achieving product-market fit. Experiments in this stage are predominantly qualitative due to the nascent customer base and lack of scale. Rapid prototyping and testing before significant investment are paramount, focusing on deep insights that can lead to foundational innovations or strategic pivots. Data indicates that companies employing rigorous "zero-to-one" experimentation significantly increase their chances of achieving product-market fit, with some studies showing a 3x higher success rate compared to those that rush to market.
-
Improving Existing Products – Innovation: Once a product is established, the focus shifts to sustained business growth. This involves creating new value through novel capabilities and capturing existing value more effectively. Continuous product discovery is key, expanding the range of problems solved for customers, enhancing existing solutions, or entering adjacent markets. This work combines both quantitative and qualitative experiments. With an existing customer base, live product data can inform A/B tests, while qualitative methods continue to unearth deeper user insights. This blend ensures that innovations are both impactful and well-received. Industry benchmarks suggest that continuous innovation, driven by balanced experimentation, can lead to a 10-15% annual growth in user engagement and revenue for mature products.
-
Improving Existing Products – Optimization: This phase focuses on maximizing the potential of the current product. It involves reducing user friction, improving clarity, and optimizing elements like pricing. These are typically quantitative, data-driven experiments, such as A/B tests on button placements, copy changes, or pricing tiers. While individual improvements may seem small, their cumulative effect across thousands or millions of users can be substantial. For example, a minor UI tweak resulting in a 0.5% conversion rate increase can translate into millions of dollars in additional revenue for a large-scale platform. This stage relies heavily on robust A/B testing platforms and sophisticated data analysis.
For all three types of product work, the overarching objective remains the same: maximizing learning. Quantitative testing aims to establish causality and quantify outcomes, while qualitative testing seeks to understand the "why" behind user behavior, uncovering new opportunities and addressing unmet needs.
The Role of AI in Modern Experimentation
AI is not merely a tool but a transformative agent across the entire product discovery lifecycle. In the realm of new product creation and improvements, AI aids in analyzing vast datasets of user feedback, synthesizing qualitative insights, and even generating live-data prototypes. It streamlines the preparation and administration of tests, and critically, assists in evaluating results with greater speed and accuracy. However, the core strategic decisions—what to build and why—remain firmly in the human domain.
For optimization work, AI’s impact is even more pronounced. It automates numerous tasks, from generating diverse variants for testing, running power calculations, and estimating sample sizes, to checking instrumentation, monitoring key metrics, and detecting anomalies. This automation significantly reduces the manual burden on data scientists and product teams, allowing them to focus on higher-level analysis and strategy.
Furthermore, when developing or enhancing AI products themselves (which are inherently non-deterministic), an additional layer of complexity emerges. Product discovery for AI often includes an "eval" process, where many judgments about product quality are subjective. AI products also alter the calculus for quantitative tests, as these can be more expensive, time-consuming, and their conclusions potentially less durable due to the dynamic nature of AI models. Therefore, leveraging "evals" to assess quality earlier in the cycle and reserving A/B tests for establishing tangible value becomes crucial.
Organizational Imperatives: Towards a Unified Experimentation Competency
Currently, many organizations operate with fragmented experimentation capabilities. Typically, a user research group (often within product design) handles qualitative experiments, a data analysis/data science group (often within product management) manages quantitative analysis, and a CRO/optimization group (potentially part of marketing or a product growth team) focuses on conversion rate optimization. This siloed approach, however, often leads to a lack of synergy and hinders the comprehensive learning required for true innovation.
The keynote strongly advocated for a broader, more holistic definition of experimentation, demanding a unified "experimentation competency" within organizations. This unification addresses two critical dimensions:
-
Integrating Experimentation Types: The cultural bias towards either qualitative or quantitative methods in many companies limits their learning potential. The most effective organizations understand that both are essential and synergistic, where the insights from one amplify the other (1 + 1 > 2). AI further blurs this line by enabling qualitative experiments at a scale previously unimaginable, allowing for more rigorous analysis of user feedback. The "eval" process for AI products is a prime example where the rigor of experimentation professionals can significantly enhance broader product discovery. As Product Discovery Coach Tom Chi eloquently puts it: "Learning why it worked is way more important than whether it worked. Why it worked is an enduring property… it comes from inside the person you’re trying to serve." Embracing both "why" and "what" is fundamental to deep learning and sustained innovation.
-
Rethinking Experimentation Ownership: The question of who runs experiments often pits "agency models" against "autonomous/democratized models." In an agency model, product teams submit requests to a centralized experimentation team, receiving findings after the fact. In autonomous models, product teams run their own experiments. After decades of observation, a "hybrid" or "enablement model" has emerged as the most effective. This model empowers product teams to own their experimentation, fostering a sense of responsibility for outcomes, while providing coaching and assistance from expert experimentation professionals.
The hybrid model recognizes that while product teams are best positioned to internalize learnings, they may lack the deep expertise in experimental design and interpretation. The experimentation team, operating with an "enablement charter," focuses on equipping product teams to conduct experiments intelligently and effectively. This includes providing services like experiment design support, managing test subjects, and reporting, alongside ongoing coaching. A significant benefit for experimentation professionals in this model is their integration directly into product teams, ensuring they are "in the room where it happens" – influencing crucial decisions rather than merely presenting post-hoc findings.
The Future: Experimentation as a Core Strategic Pillar
As AI continues to drive down the cost and time associated with building and deploying product capabilities, the strategic imperative shifts entirely to what an organization chooses to build. Experimentation is no longer a peripheral function but the central competency for product organizations striving to discover truly valuable solutions. This necessitates a move from a narrow focus on optimization, often measured by the volume of A/B tests, to a broad strategic function measured by business and customer impact.
This evolution demands that companies embrace the full spectrum of qualitative and quantitative experimentation, moving beyond merely proving outcomes to actively influencing them. It means expanding experimentation’s remit from pure optimization to driving groundbreaking innovation, and from merely determining "which version won" to deeply understanding "what we should build and why." The future calls for experimentation professionals with a profound grasp of diverse techniques, capable of assisting product teams not just in validating solutions, but in shaping products that generate transformative outcomes. The Circus Austin Conference served as a powerful testament to this impending shift, urging organizations to cultivate a holistic experimentation competency to navigate the complexities and opportunities of the AI-driven product landscape.
