growth-experiment_skill

Overview

This skill is an Experimentation Agent that designs, runs, and analyzes A/B tests to validate growth hypotheses. It focuses on rigorous experiment setup, correct sample-size planning, and clear statistical interpretation to inform product and marketing decisions. The goal is to turn ideas into reliable, data-driven outcomes.

How this skill works

The agent translates a business hypothesis into a randomized experimental design, defines primary and guardrail metrics, and calculates required sample sizes and test duration. During the experiment it collects outcome data, applies appropriate statistical tests and adjustments, and produces an interpretation with confidence levels and recommended actions. Deliverables include a clear experiment plan, analyzed results, and a decision recommendation.

When to use it

• Validating product or feature changes before full rollout
• Comparing variations of messaging, UI, or pricing
• Estimating impact of marketing campaigns on conversion or retention
• Prioritizing growth ideas with empirical evidence
• Checking for unintended negative effects using guardrail metrics

Best practices

• Define a single primary metric and a small set of guardrail metrics up front
• Pre-calculate sample size and minimum detectable effect to set realistic expectations
• Randomize assignment and ensure consistent instrumentation across variations
• Avoid peeking at results; use pre-specified stopping rules or correction methods
• Document hypotheses, segment definitions, and decision criteria before starting

Example use cases

• Test two onboarding flows to reduce time-to-first-action and measure retention lift
• Compare two pricing page layouts to maximize trial-to-paid conversion
• Validate a new email subject line for improved open and click rates
• Assess effect of a UI tweak on key engagement metrics across user segments
• Run holdout experiments to measure incremental lift from a new feature