experiment_skill

Overview

This skill designs and analyzes rigorous A/B and multivariate experiments to validate product hypotheses with statistical confidence. It produces hypothesis documents, calculates sample sizes, implements feature flags for safe rollouts, and generates experiment reports with clear recommendations. The focus is on reproducible design, correct inference, and actionable outcomes.

How this skill works

I start by turning a product question into a falsifiable hypothesis with a primary metric and success criteria. I then design variants, run power calculations (baseline, MDE, significance, power), and recommend duration and randomization. For execution I outline feature-flag patterns and exposure tracking; after data collection I run appropriate statistical tests (Z-test, Welch’s t-test, chi-square, or Bayesian methods), apply sequential testing when needed, and produce a results report with confidence intervals and recommended actions.

When to use it

• Validating a new feature or UX change before rollout
• Estimating required traffic and duration for reliable results
• Implementing feature flags for gradual or targeted rollouts
• Analyzing experiment outcomes to decide ship/iterate/rollback
• Designing multivariate or factorial tests to optimize multiple factors

Best practices

• Pre-register primary metric and success criteria before starting
• Calculate sample size and required duration; plan for 80%+ power and 5% significance
• Use deterministic userId hashing for randomization and avoid overlapping experiments
• Limit primary metrics to one and correct for multiple comparisons when needed
• Avoid peeking—use alpha spending rules for valid early stopping

Example use cases

• A/B test signup CTA copy with sample size, rollout flag, and final statistical analysis
• Multivariate test pricing page layout with factorial design and interaction reporting
• Power calculation for predicted 10% lift on checkout conversion and test duration estimate
• Feature-flag rollout plan: deterministic allocation, incremental exposure, and cleanup checklist
• Post-experiment report summarizing confidence intervals, learnings, and launch recommendation