Playwright Test Agents and Regresco: who writes the test, and who keeps running it

Playwright now comes with agents that write its tests. If you've tried them, you may be wondering why anyone would still pay for a tool like Regresco. I build Regresco, so read this knowing that. To answer it, I took the tests Playwright's agents generated for their own example, imported them into Regresco and ran them.

The agents write and repair tests in your editor, while someone is working. Regresco runs tests on a schedule, when nobody is. Those are different jobs, and you can use both.

What Playwright Test Agents are

Three agent definitions, in Playwright since version 1.56. You add them to a project with npx playwright init-agents --loop=claude (the other options are vscode, codex and opencode), and your coding tool can call them:

• The planner explores your app and writes a test plan in Markdown.
• The generator turns the plan into Playwright test files, checking selectors and assertions on the live page as it goes.
• The healer runs the tests and repairs the ones that fail.

They're free, and the result is ordinary Playwright code in your repository: fixtures, network mocks, three browser engines, everything Playwright can do. You pay for the coding tool and the model.

They also stop when you stop. Nothing runs until someone starts an agent, and running the tests every night, keeping their history and telling someone what failed is yours to set up.

What Regresco does

Regresco keeps flows in your account and runs them in cloud Chromium on a schedule, from a CI webhook, or when your coding agent asks. Each run is compared with the ones before it. A failed step gets a label (likely regression, broken selector or flaky), a screenshot and the browser's console errors, and you get an email.

The limits are real. It runs Chromium only. A flow has ten step types and no custom JavaScript, so it can't use fixtures or network mocks, can't double-click, and can't check that something is gone or what a field holds.

Side by side

We ran their tests in Regresco

Playwright's repository has an example project, examples/todomvc, with tests in the form their generator writes. I pasted all 23 into Regresco's import page, one file at a time, in a project pointed at their demo site, and ran the project once.

• 18 of 23 passed on the first run, with no selector replaced.
• The other 5 failed, all for one reason: they start editing a todo with dblclick(). A flow has no double-click step, so the import listed that line as skipped, and the next step found no edit box.

Passing doesn't mean the flow checks everything its test did. From the 23 files, 187 lines became steps and 29 stayed behind:

• 21 checks a flow can't make: toHaveValue (8), .not.toBeVisible (6), .not.toBeChecked (2), toHaveCount (2), and one each of toBeFocused, toMatchAriaSnapshot and toHaveURL with a regular expression.
• 8 actions: the 5 double clicks, 2 page.keyboard.press calls with nothing typed before them, and a fill('').

Another 4 checks, all toBeChecked, came in as “the element is visible”. Only 7 of the 23 tests came through whole. “Should not add empty todo” passes in Regresco with one click and nothing else, because everything it checks is on that list. The import page shows those lines under each test before you import, so read them: if a test's whole point is on the list, keep that test in your repository.

Their tests open no page, because a fixture does that for them, and fixtures aren't imported. Each flow starts at the project's staging URL instead, and the page says so.

The list of what the importer reads is in the docs.

How to use both

1. Let the planner and the generator draft tests for your critical paths. Read them, and keep them in your repository.
2. Run them in CI on pull requests, in the browsers you care about. That's where the full Playwright API earns its keep.
3. Paste the ones that matter most into Regresco, point the project at staging and schedule a daily run. From then on nobody has to remember to run them, and a failure arrives as an email with a label and a screenshot.
4. When a selector changes, let the healer repair the code in your editor. Regresco's own repair is narrower: a click or fill whose selector stopped matching follows the one element on the page with exactly the same name.
5. When the healer marks a test test.fixme, treat it as a bug report. I ran it on real bugs, and that's what it did every time: the test stops running, and the suite goes green.

If your coding agent is Claude Code, you can skip the file: it can save and run flows in Regresco directly, over MCP. Here's how that works.

When you don't need Regresco

You have a Playwright suite, CI runs it every night, and someone reads the reports. Or you need Firefox and WebKit, or things a flow can't do, like network mocks. Then Test Agents and managed Playwright runners are the closer fit, and I'd use them.

When it helps

Nobody on the team owns the test suite, and generated tests would sit in a folder and go stale. You want someone outside engineering to read a flow and a failure. You want history, labels and alerts without building them.

Questions