testpilot — Self-Repairing QA Agent
Turns plain-English QA stories into Playwright tests that diagnose their own failures — and refuse to weaken themselves when the product is what broke.
Highlights
- Eight-type failure taxonomy separates safe UI drift (selector, copy, flake) from real breakage — only drift is ever auto-repaired; regressions are flagged for a human, never patched over.
- Repair guardrails: never removes an assertion, never changes the expected business outcome, never edits outside the generated-tests directory. Vision can veto a repair — never authorize one.
- Verified end-to-end: 113/113 tests across 22 suites, plus a three-scenario demo where a “Sign in” → “Log in” copy change is safely repaired and a broken login is refused as PRODUCT_REGRESSION.
- Live dashboard streams every pipeline stage over SSE with expandable evidence (screenshots, diffs, verdicts); stories ingest from GitHub issues and Jira via MCP; approved repairs ship as reviewable PR bundles — never silent merges.
The killer feature: the repair it refuses to make
Any agent can rewrite a failing test until it goes green. That’s the failure mode — a QA agent that “fixes” tests by deleting the assertions that caught a real bug. testpilot draws the line explicitly: when the UI merely drifted (a button’s label changed, a selector moved), it proposes a safe repair; when the product actually broke (login no longer reaches the dashboard), it refuses to touch the test and flags it for a human.
Failure taxonomy
Every failure is classified into one of eight types. Only the first three are ever eligible for auto-repair:
| Diagnosis | Action |
|---|---|
| SELECTOR_DRIFT | Safe repair proposed |
| UI_COPY_CHANGE | Safe repair proposed |
| TIMING_OR_FLAKE | Safe repair proposed |
| PRODUCT_REGRESSION | Refused — flagged for a human |
| APP_UNAVAILABLE | Refused — flagged for a human |
| NETWORK_OR_API_FAILURE | Refused — flagged for a human |
| AUTH_OR_TEST_DATA_FAILURE | Refused — flagged for a human |
| UNKNOWN | Refused — flagged for a human |
Vision-assisted diagnosis follows one rule: vision can veto a repair, never authorize one. A screenshot can make the system more cautious; it can never upgrade a regression into an auto-fix.
A verified run
The bundled demo runs three scenarios end to end against a live app: a clean pass, a “Sign in” → “Log in” copy change, and a genuinely broken login. The copy change gets a repair that widens the brittle selector to a role locator matching both labels — every assertion preserved. The broken login is diagnosed PRODUCT_REGRESSION and the test is left untouched.
--- tests/generated/login.spec.ts
+++ tests/generated/login.spec.ts (repaired)
@@
- await page.getByRole('button', { name: 'Sign in' }).click();
+ await page.getByRole('button', { name: /^(Sign in|Log in)$/ }).click();
await expect(page).toHaveURL(/\/dashboard/);
await expect(page.getByText('Welcome, Demo User')).toBeVisible();The action changes; the assertions don’t. That asymmetry is the whole safety model: a repair may adapt how the test drives the app, never what the test demands of it.
Pipeline
| Stage | What happens |
|---|---|
| spec | Plain-English story parsed into a structured test spec |
| observe | Live page state captured — DOM, screenshots, network |
| generate | Playwright test grounded in the observed page |
| run | Test executed; failure artifacts collected |
| diagnose | Failure classified into the eight-type taxonomy |
| repair | Safe repair proposed and re-validated — or refused |
| report | Verdict, evidence, and diff bundled for review |
Every stage streams live to a Grand Canyon-themed dashboard over SSE, with expandable evidence per stage. Stories ingest from manual upload, GitHub issues, or Jira tickets via MCP connectors, and approved repairs ship as reviewable PR bundles or real GitHub PRs — never silent merges.
Receipts
- 113/113 tests passing across 22 suites (unit + integration), covering the classifier, repair validator, connectors, and pipeline end to end.
- Deterministic mock mode is the default — the full demo reproduces bit-for-bit with no API key, so the safety behavior itself is testable in CI.
- CI-enforced perf budget on the dashboard: a script boots the built UI, drives a full run, and fails the build on any budget breach.
Clone it and run npm run testpilot -- demo --mode mock — the three-scenario demo runs end to end in about a minute, no API key required.