flaky-detect_skill

Overview

This skill identifies flaky tests by analyzing CI history, test reruns, and test-code patterns to surface intermittent failures that reduce CI reliability. It produces prioritized reports with pass rates, root-cause categories, and concrete remediation or quarantine actions. Use it to improve release stability, speed up debugging, and track flaky-test fixes over time.

How this skill works

The skill parses CI run history (GitHub Actions, other CI logs) to compute pass/fail rates per test and flags tests with inconsistent outcomes. It scans test source code for common flaky patterns (timing, async misuse, random values, environment or network access). It also supports automated re-run campaigns to verify flakiness, then produces a ranked report with recommendations and quarantine suggestions.

When to use it

• Intermittent test failures reported by developers or CI
• CI stability audits before releases or post-incident
• When triaging failing pipelines to separate flaky tests from real regressions
• To build a prioritized remediation plan for test reliability
• When adding test quarantine or tracking to CI workflows

Best practices

• Require a minimum number of runs (e.g., >=5) before classifying a test as flaky
• Combine CI-history metrics with code-pattern scans and reruns for higher confidence
• Prioritize fixes by impact: timing and async issues first, then environment/resource problems
• Quarantine only when a fix is blocked; track quarantined tests and set unquarantine targets
• Use mocking for time and network and add isolation for stateful resources

Example use cases

• Run a 30-day CI history analysis to find top flaky tests and their pass rates
• Scan the test suite to list files that use Date.now, Math.random, or real network calls
• Automate reruns of a suspicious test 10–20 times to confirm intermittent behavior
• Generate a quarantine manifest and CI exclude list for temporary release stabilization
• Produce a remediation plan mapping each flaky test to recommended fixes and PRs