a-b-testing_skill

Overview

This skill teaches the science and practice of A/B testing to build validated learning into product development. It focuses on experiment design, statistical rigor, feature flagging, and practical analysis so teams learn quickly and reduce the cost of being wrong. The approach treats every feature as a hypothesis and every launch as an opportunity to learn.

How this skill works

The skill inspects experiment proposals for clear hypotheses, required sample sizes, and single-variable treatment design. It checks for feature-flag readiness, randomization integrity, and data collection plans, then evaluates results using appropriate statistical methods and practical significance criteria. Risk checks flag common failure modes like underpowered tests, multiple uncorrected comparisons, and instrumentation gaps.

When to use it

• You plan a product change and need a hypothesis-driven experiment
• You need help calculating sample size, power, or test duration
• You want to implement feature flags and rollouts safely
• You are unsure whether a negative or insignificant result is actionable
• You need guidance on interpreting p-values, uplift, and practical significance

Best practices

• Define a clear hypothesis and primary metric before launching the test
• Power your test: compute required sample size and avoid underpowered experiments
• Change only one primary variable per experiment to isolate effects
• Use feature flags to control exposure and enable safe rollbacks
• Prioritize practical significance and learning over chasing p-values
• Document instrumentation, segment definitions, and stopping rules in advance

Example use cases

• Compare two onboarding flows to measure first-week retention uplift
• Test a pricing layout change to see its impact on conversion rate
• Validate an algorithm tweak by running a treatment in production with a feature flag
• Run a holdout group experiment to estimate the causal effect of a growth campaign
• Audit existing experiments for power, instrumentation gaps, and flawed analysis