Sun. Nov 22nd, 2026

AI and Experimentation Drive a New Era in Product Development: Circus Austin Conference Highlights Expanded Role

Austin, TX – The future of product development, deeply intertwined with the accelerating pace of artificial intelligence, hinges on a fundamental shift in how organizations approach experimentation. This was the central theme at the recent Circus Austin Experimentation Conference, where industry leaders underscored that experimentation is rapidly transitioning from a specialized function—often confined to optimization—to a foundational, company-wide competency essential for consistent innovation and sustainable business outcomes. The evolving landscape demands a more holistic and integrated approach, particularly within the modern "product operating model," which prioritizes solving user problems over merely delivering features.

Historical Context: The Evolution of Product Experimentation

The concept of product experimentation is not new, but its scope and methodologies have undergone significant transformations over the decades. Early forms of product discovery, dating back to the 1980s, often involved qualitative research—crafting prototypes and testing them directly with a small set of customers. These early experiments aimed to understand user needs, identify pain points, and validate preliminary design concepts before committing extensive resources to development. The insights gained were primarily qualitative, focusing on "why" users reacted in certain ways, providing crucial directional guidance.

A pivotal moment arrived with the dawn of the internet era in the late 1990s, ushering in the capability for large-scale quantitative experimentation. Marty Cagan, a prominent voice in product management, recounted his experience at Netscape Communications in 1997. Faced with conflicting opinions between an executive and a product designer regarding product interface, the company’s design leader, Hugh Dubberly (formerly of Apple), introduced the revolutionary idea of A/B testing. With millions of daily active users, Netscape possessed the unprecedented data volume required to empirically compare different design treatments and determine which performed better based on measurable metrics. This era was famously encapsulated by Netscape CEO Jim Barksdale’s dictum: "If we have data, let’s look at data. If all we have are opinions, let’s go with mine." The internet democratized data collection, empowering product teams to make data-informed decisions and overcome hierarchical objections, directly attributing product changes to business outcomes.

This shift laid the groundwork for the modern product operating model, which contrasts sharply with the older "project model." In the project model, stakeholders dictate roadmaps of features to be built, focusing on output. Conversely, the product model empowers teams to solve defined problems and innovate solutions that deliver specific business results or "outcomes." While optimization work—fine-tuning existing products—could still occur within the project model, the broader, more strategic role of experimentation as a driver of innovation truly flourishes within the outcome-driven product model. As Amazon founder Jeff Bezos famously stated, "If you want to innovate, you simply must experiment."

The AI Imperative: Redefining Product Development

The advent of Artificial Intelligence marks another profound inflection point, dramatically altering the calculus of product development. AI has significantly reduced the time and cost associated with building and deploying new capabilities. In the past year, many companies have leveraged AI to ship more features and products at an unprecedented pace. However, this acceleration has revealed a critical challenge: the "AI productivity paradox." Despite faster development cycles, many organizations have not seen a corresponding improvement in desired business outcomes.

This paradox highlights a crucial vulnerability: the ease of building does not equate to building the right thing. Rapid releases without thorough validation have led to high failure rates, resulting in frustrated and confused customers, erosion of trust, and increased churn. For instance, recent industry reports suggest that upwards of 70% of new product launches fail to meet their objectives, a figure that remains stubbornly high even with accelerated development. The core problem is not the inability to build, but the struggle to accurately "figure out what to build."

In this environment, experimentation has never been more critical. The stakes are higher; the cost of building has decreased, but the consequences of building the wrong thing—in terms of market perception, customer loyalty, and wasted resources—remain substantial. This dynamic is driving a renewed and intensified focus on experimentation as the primary means to discover solutions that genuinely deliver necessary outcomes. Industry analysts predict that the coming years will see experimentation emerge as the core competency for effective product discovery and, consequently, for successful product companies. This necessitates an expansion of experimentation’s role beyond mere optimization, extending into innovation, moving from simply proving outcomes to actively influencing them, and shifting from post-launch testing to driving pre-development strategic decisions.

Experimentation’s Broader Mandate: Beyond Optimization to Innovation

Top-tier product organizations understand that different types of product work demand tailored experimental approaches. These can be broadly categorized into three areas:

  1. Creating New Products ("Zero to One"): This involves bringing entirely new products or services to market. Here, the risks are broadest, encompassing not only value (do customers want it?), usability (can they use it?), viability (can the business sustain it?), and feasibility (can we build it?), but also critical strategic elements like cost structure, monetization models, and go-to-market strategies. The ultimate goal is achieving product/market fit. Experiments in this phase are often qualitative, given the nascent customer base and traffic. They prioritize rapid learning through extensive prototyping, allowing teams to test assumptions and pivot directions before significant investment in full-scale development. For example, a fintech startup might run concept tests with potential users to gauge interest in a novel budgeting tool, iterating on prototypes based on feedback to refine its value proposition.

  2. Improving Existing Products – Innovation: Once a product is established, the focus shifts to sustained growth and value creation. This involves continuous product discovery: expanding the range of problems solved for customers or the company, enhancing solutions to existing problems, or extending offerings to adjacent markets. This work is characterized by rapid learning, leveraging both quantitative and qualitative experiments. Teams might use A/B tests to validate new features or conduct user interviews to uncover unmet needs within existing user segments. A social media platform, for instance, might experiment with new content formats to increase user engagement or explore new communication features based on ethnographic studies.

  3. Improving Existing Products – Optimization: This type of work aims to maximize the potential of existing products by refining user flows, increasing clarity, or optimizing pricing strategies. These experiments are predominantly quantitative and data-driven. While individual improvements might seem small, their cumulative effect across thousands or millions of users can be substantial. Examples include testing different button placements to improve conversion rates, optimizing onboarding flows to reduce drop-off, or A/B testing pricing tiers to maximize revenue. The goal is to capture more of the value already created by the product.

Across all three types, the overarching objective is to maximize learning. Quantitative tests establish causality and quantify outcomes, answering "what" worked. Qualitative tests delve deeper, explaining "why" customers behave as they do, identifying underlying motivations, frustrations, and opportunities for innovation.

AI’s Transformative Impact on Experimentation Mechanics

AI is not just influencing why we experiment, but also how we do it. It offers substantial assistance at virtually every stage of product discovery:

  • Discovery and Prototyping: AI-powered tools can analyze vast datasets of user feedback, market trends, and competitive intelligence to identify unmet needs and potential problem areas. Generative AI can rapidly create diverse prototypes, from wireframes to functional mock-ups, significantly accelerating the ideation phase. This allows teams to test a wider array of solutions more quickly.
  • Test Preparation and Administration: AI can automate tedious tasks such as generating experiment variants, calculating power analyses, estimating optimal sample sizes, and ensuring correct instrumentation for data collection. This reduces human error and speeds up the setup process for both qualitative and quantitative studies. For instance, AI can help segment user groups for targeted testing or even simulate user behavior to pre-validate experiment designs.
  • Evaluation and Analysis: AI algorithms can monitor metrics in real-time, detect anomalies, and even provide preliminary interpretations of experiment results. For qualitative studies, AI can transcribe and synthesize insights from interviews and user tests, identifying recurring themes and sentiment patterns at scale. This drastically reduces the manual effort involved in processing large volumes of qualitative data.

However, building or improving AI-driven (non-deterministic) products introduces an additional layer of complexity. Product discovery for AI products must incorporate specialized "eval" processes, where human judgment is often required to assess subjective qualities like fairness, relevance, or helpfulness of AI outputs. Quantitative A/B tests for AI products can be more expensive and time-consuming, and their conclusions may be less durable as AI models continuously evolve. Therefore, a strategic approach involves using "evals" to assess quality and ethical considerations earlier in the development cycle, reserving more resource-intensive A/B tests for establishing tangible business value.

Addressing Organizational Silos: Towards a Holistic Experimentation Competency

Despite the clear benefits of a comprehensive approach, many organizations face structural challenges. Typically, experimentation responsibilities are fragmented across multiple, often siloed, groups: a user research team (part of product design), a data analysis/data science team (part of product management), and a CRO (Conversion Rate Optimization) or marketing team. This fragmentation often leads to a lack of synergy and a limited, tactical view of experimentation.

The need for a unified "experimentation competency" necessitates addressing two critical dimensions:

  1. What Types of Experiments? Integrating Qualitative and Quantitative: Many companies exhibit a cultural bias, favoring either qualitative or quantitative methods. However, the most effective organizations recognize that both are indispensable and create a synergistic effect where "1 + 1 > 2." Qualitative insights explain the "why," uncovering underlying motivations and unforeseen opportunities, while quantitative data validates "what" works and quantifies its impact. AI is beginning to blur the lines between these two, enabling qualitative experiments to be conducted at a much larger scale, approaching the statistical power traditionally associated with quantitative methods. Moreover, the rigor applied by experimentation professionals in defining, executing, and evaluating tests can significantly enhance broader product discovery efforts, including the subjective "eval" processes for AI products. As Product Discovery Coach Tom Chi eloquently states, "Learning why it worked is way more important than whether it worked. Why it worked is an enduring property. That it worked is an episodic property. The reason why it worked is enduring because it comes from inside the person you’re trying to serve." Embracing both qualitative and quantitative approaches is paramount for a truly robust experimentation strategy.

  2. Who Runs Experiments? Organizational Models and Empowerment: The question of who executes experiments often sparks organizational debate. Three main models emerge:

    • The Agency Model: Experimentation teams function as a service, receiving requests from product teams, conducting the tests, and presenting findings. While this centralizes expertise, it can disempower product teams, leading to less ownership and adoption of learnings. Product teams might view findings as external reports rather than integral to their development process.
    • The Autonomous/Democratized Model: Product teams are fully responsible for designing, running, and interpreting their own experiments. This fosters strong ownership but often suffers from a lack of specialized expertise within individual teams, potentially leading to flawed experiment design, incorrect statistical analysis, or misinterpretation of results.
    • The Hybrid/Enablement Model: This model is increasingly advocated as the most effective. Product teams own their experiments and are accountable for outcomes, but they receive continuous coaching and assistance from dedicated experimentation professionals. The experimentation team operates with an "enablement charter," empowering product teams to conduct as much of the experimentation process as possible, while ensuring intelligence, rigor, and effectiveness. This team provides essential services—such as experiment design support, managing participant recruitment, and sophisticated reporting tools—alongside ongoing mentorship. A significant benefit of this hybrid approach is that experimentation professionals, often sidelined in the past, become integral members of product teams, "in the room where it happens," directly influencing strategic decisions and product direction. This integration elevates their role from service providers to strategic partners.

The Future of Product Experimentation: A Core Competency

As AI continues to make the "building" aspect of product development faster and cheaper, the strategic imperative shifts entirely to "what to build." Experimentation, therefore, is rapidly becoming the core competency for product organizations dedicated to discovering solutions truly worth developing. This paradigm embraces "building to learn," where prototypes are systematically tested against critical product risks—value, usability, viability, and feasibility—before full-scale development.

For organizations that have historically confined experimentation to niche optimization tasks, this represents a profound transformation. It means moving beyond a data science or CRO team measured solely by the number of A/B tests conducted per quarter, to a company-wide core competency measured by tangible business and customer impact. This broader mandate requires:

  • Embracing both qualitative and quantitative experimentation: Recognizing that each provides unique, complementary insights.
  • Experimenting for both value creation and value capture: Driving innovation for new capabilities while optimizing existing ones.
  • Expanding experimentation beyond optimization to innovation: Moving from incremental improvements to discovering breakthrough solutions.
  • Shifting from merely proving outcomes to actively influencing outcomes: Using experimentation as a proactive tool for strategic decision-making.
  • Moving from simply determining "which version won" to understanding "what we should build and why."

Ultimately, the future demands experimentation professionals with deep, comprehensive knowledge of the full spectrum of qualitative and quantitative techniques, capable of supporting product teams across all types of product work. Their role will evolve from simply validating results to actively assisting in the creation of products that successfully drive desired business outcomes. This profound shift is not merely an operational adjustment; it is a strategic imperative for any company aiming to thrive and innovate in the AI-driven economy.

Leave a Reply

Your email address will not be published. Required fields are marked *