exp-driven-dev_skill

Overview

This skill builds features with A/B testing and feature-flag-first thinking using Ronny Kohavi’s frameworks and experimentation practices from Netflix and Airbnb. It helps teams design experiments, pick a single primary metric plus guardrails, and implement feature flags and tracking for fast, safe iteration. Use it to make data-driven ship/stop decisions and avoid common statistical and operational mistakes.

How this skill works

The skill guides you through hypothesis framing (HITS), experiment implementation (consistent assignment, sample size, rollout), test execution (monitoring significance and guardrails), and the post-test decision (ship, stop, iterate). It provides templates for experiment specs, feature-flag code patterns, sample-size rules, and a simple dashboard checklist to track progress and outcomes. It enforces best practices like one primary metric, clear success thresholds, and avoiding early peeking.

When to use it

• Launching features that touch core business metrics
• Setting up feature flags and gradual rollouts
• Designing A/B tests or multi-variant experiments
• Choosing primary and guardrail metrics for a change
• Making go/no-go decisions based on experiment results

Best practices

• Define one primary metric tied to business value and a clear success threshold
• Always list guardrail metrics to prevent degrading quality or gaming results
• Calculate sample size and test duration before starting; avoid peeking
• Limit variants (2–3) to preserve statistical power
• Put every change behind a flag for rollback and gradual rollout

Example use cases

• Streamlined checkout: test conversion uplift while monitoring cart abandonment, returns, support load and load time
• New recommendation algorithm: A/B test bookings per search with guardrails for search quality and host earnings
• UI change rollout: 1% → 10% → 50% → 100% with experiment assignment and dashboard monitoring
• Feature flag architecture: implement consistent hashing for deterministic assignment and gradual percentage rollouts
• Experiment velocity dashboard: track experiments launched, win rate, average duration, and revenue impact