deflake-e2e-recent-commits_skill

This skill automatically deflakes recent E2E tests by aggregating flaky results from main CI and PRs, then proposes fixes and reports.

TypeScript

19.7k

GitHub Stars

1

Bundled Files

2 months ago

Catalog Refreshed

4 months ago

First Indexed

Readme & install

Copy the install command, review bundled files from the catalogue, and read any extended description pulled from the listing source.

Installation

Preview and clipboard use veilstrat where the catalogue uses aiagentskills.

npx veilstrat add skill dyad-sh/dyad --skill deflake-e2e-recent-commits

SKILL.md7.3 KB

Overview

This skill automatically finds flaky end-to-end tests from recent CI runs on the main branch and from recent PRs authored by wwwillchen/wwwillchen-bot, then attempts to deflake them. It collects Playwright results artifacts and bot comments, ranks flaky tests by frequency, and runs a controlled deflaking workflow. The goal is targeted, repeatable fixes to tests without changing application code.

How this skill works

It scans recent completed CI workflow runs on main (configurable commit count) and downloads Playwright html-report artifacts to extract flaky tests from results.json. It also scans recent open PR comments posted by the Playwright bot to extract reported flaky tests. Tests are deduplicated and ranked by occurrence, then each spec file is run repeatedly to confirm flakiness before applying test-only fixes and verifying them.

When to use it

After a series of CI failures where tests intermittently fail on main
When Playwright bot comments in PRs report flaky tests
As part of a regular maintenance pass to reduce CI flakiness
Before a release to improve E2E stability and reduce noise
When you need a prioritized list of highest-impact flaky tests

Best practices

Always use the TodoWrite tool to track progress and mark steps in_progress/completed
Use PLAYWRIGHT_RETRIES=0 when reproducing flakiness to avoid masking issues
Do not change application code; only modify tests and snapshot baselines
Run npm install and npm run build before running E2E tests and after any app-code changes
Limit fix attempts to two per spec file; revert and move on if unresolved

Example use cases

Scan the last 10 commits on main, gather flaky tests, and create PRs with test fixes
Investigate flaky tests reported in recent PR comments by wwwillchen-bot and deflake them
Run targeted deflake workflow for the top 5 most frequently flaky spec files
Automate a weekly deflake sweep to keep CI noise low
Confirm suspected flaky tests by running a spec 10 times with retries disabled

FAQ

It scans 10 recent commits by default, but the commit count is configurable via an argument.

What counts as a flaky test?

A test whose final run status is passed but has at least one prior run with failed, timedOut, or interrupted in the Playwright results.

Can this change application code to fix flakiness?

No. The workflow only edits test files and snapshot baselines; changing application code is explicitly disallowed.