senior-data-scientist_skill

Overview

This skill codifies a rigorous data science workflow for turning ambiguous or open-ended questions into measurable metrics, experiments, and models. It guides hypothesis framing, metric selection, experiment design, validation, and clear reporting of assumptions and caveats. Use it to ensure analyses and models drive actionable decisions rather than ambiguous interpretations.

How this skill works

The skill walks you through translating an ask into a decision that will change behavior or product choices, then defining primary metrics, guardrails, and relevant segments. It helps choose the right method (descriptive analysis, A/B test, causal inference, or predictive modeling), perform validation checks (leakage, baseline, error analysis, robustness), and produce concise experiment or model reports with explicit limitations. Optional tooling supports quick CSV profiling to accelerate initial data assessment.

When to use it

• Framing ambiguous stakeholder questions into testable hypotheses
• Selecting a primary metric and guardrail metrics for an experiment
• Designing and sizing A/B tests or causal studies
• Building predictive models with a clear decision context
• Performing error analysis and robustness checks before deployment
• Writing concise experiment or model reports with assumptions and next steps

Best practices

• Start by specifying the decision that depends on the result; align metric choice to that decision
• Keep one primary metric and a small set of guardrails to avoid metric fishing
• Pre-register analysis plans for experiments to reduce bias and p-hacking
• Run leakage and sanity checks early, and validate baselines before interpreting results
• Segment results thoughtfully and report heterogeneity rather than only averages
• Document assumptions, data limitations, and plausible alternative explanations in every report

Example use cases

• Turn a vague request like “improve engagement” into a specific testable hypothesis and metric set
• Design and analyze an A/B test to evaluate a new feature while protecting revenue metrics as guardrails
• Choose between causal and predictive approaches when measuring product changes or forecasting outcomes
• Perform error analysis on a churn prediction model and communicate operational tradeoffs
• Produce a short experiment report that summarizes methods, results, caveats, and recommended next steps