When a healer hides a regression: what Playwright's agent did with real bugs

A test that repairs itself sounds like an obvious win. A selector changes, the test fixes itself, nobody gets paged. The harder question is what happens when the test was right and the product broke, because a repair then has nothing to repair and every reason to make the red go away.

Playwright now ships a healer agent, so I ran it on real bugs to see. I build Regresco, which repairs selectors too, and ours got this wrong until this week. That part is further down.

The setup

A small shop page and one test, in the form Playwright's generator writes: enter a discount code, check the total is 90.00, place the order, check for the thank-you message. Then I changed one thing on the site for each situation.

The agent ran as shipped: Playwright 1.64.1, added with npx playwright init-agents --loop=claude, on the model its definition asks for (Sonnet), with the prompt from their documentation, “Fix the test”. Each situation got three runs on a fresh copy of the project, 15 in all. The site's source was in the project, as it would be in a real repository.

What it did

It fixed the renamed button properly, and it never rewrote an expected value to match a bug. I expected it might change 90.00 to 95.00, since its instructions list “fixing assertions and expected values”, and it didn't once. All 12 notes on the bugs were accurate, and 11 named the code behind the bug:

// FIXME: app bug. site/app.js computes `discounts[code] / 2`, so SAVE10 gives 5% off
// (Total: 95.00 EUR) instead of the 10% off (90.00 EUR) required by specs/checkout.md.
// Remove fixme once the app applies the full discount.
test.fixme('discount code lowers the total, and the order goes through', async ({ page }) => {

As a way to find out what's wrong, that's very good. The question is where that note goes next.

Where the regression goes

Before the healer, npx playwright test said 1 failed and exited with code 1. After it, the same command says 1 skipped and exits with code 0. CI is green, and whatever waits on CI goes ahead. The bug report is now a comment in a diff.

Playwright documents this. Their documentation gives the healer's output as “a passing test, or a skipped test if the healer believes that functionality is broken”. Its instructions say to mark the test “so that it is skipped during the execution”, and also: “Do not ask user questions, you are not interactive tool, do the most reasonable thing possible to pass the test.”

A skipped test also stops checking everything else it covers. Mine had a second half, placing the order. With the test marked, I broke the order endpoint as well, and ran the suite again: still 1 skipped, still exit code 0. Two bugs, one green run.

If a person reads that diff, none of this is a problem: they see the note and fix the bug. The risk is the setup where nobody does, such as an agent told to get CI passing, a pull request merged because the checks are green, or a healer run in a loop overnight.

How to keep it from hiding anything

• Make a new skip fail the pull request. Playwright's forbidOnly option catches a stray test.only, and I found no option like it for test.fixme. A small step in CI does the job. On the healer's change it found the new mark, and on an untouched copy it stayed quiet:

  if git diff origin/main -- tests \
    | grep -E '^\+.*test\.(fixme|skip)\('
  then
    echo "A test was just marked skipped."
    echo "Read the note and fix the bug."
    exit 1
  fi

• Treat test.fixme from an agent as a bug report. Open the issue before you merge, and put the note in it.
• Keep one run the repair can't reach. The thing that edits your tests shouldn't be the only thing telling you whether the product works.

How Regresco handles this

Regresco repairs selectors during a cloud run, with nobody watching, so the limits on a repair have to be strict:

• A check is never replaced. If it doesn't hold, the step fails.
• A click or a fill only follows an element with exactly the same name, and only if the page has one such element. Otherwise the step fails.
• A step that worked with a replacement says so on the run page and in the results an agent reads.
• A failing step stays a failure, with a label and a screenshot. Nothing turns it into a skip.

When I had a coding agent cover our own product over MCP, 24 tests and 706 steps, I added five checks written to be false, including a wrong price on the pricing page. All five fail. The same limit on what a repair may touch is why a flow won't follow a renamed button: if “Apply” becomes “Redeem”, the step fails and asks you to look, which is more work than what Playwright's healer does for that case. That's the trade we chose.

One caveat. If you connect a coding agent to Regresco, it can edit a flow too. A token can carry the rights to read results and start runs without the right to change flows, and that's the token I'd give an agent whose job is to get things green. The MCP guide lists the rights.

Should you use the healer?

Yes. It fixed the outdated test and correctly diagnosed every bug I gave it, in under a minute each. Read what it writes, and don't let a skip through unnoticed. I wrote up how the agents and Regresco fit together in a separate comparison, including what happened when I imported the tests they generate.