AI Agent for QA Testing: How Agentic Testing Works

Ask four vendors what an AI agent for QA testing is and you will hear about four different products: a coding assistant that writes Playwright tests, a recorder with smarter locators, a service that turns plain-language steps into a browser test, and a model that clicks through your app on every run with no fixed script at all. They share a label and fail in very different ways. This guide explains what an agent is, how agentic testing differs from its neighbours, what agents can and cannot do today, how to evaluate one, and where a person has to stay in the loop.

For the broader argument about where agents belong in a quality strategy, see our essay AI Agents for Software Testing: What Works and What Doesn't.

What an AI QA agent actually is

Anthropic's engineering team draws a useful line in Building Effective Agents. Workflows are "systems where LLMs and tools are orchestrated through predefined code paths". Agents are systems where models "dynamically direct their own processes and tool usage".

In testing, an agent works in a loop. It reads the state of the application, chooses an action, performs it in a real browser, observes the result and decides what to do next, until it reaches the goal or gets stuck. Microsoft's Playwright MCP server shows the plumbing: it lets a language model drive a browser through structured accessibility snapshots of the page.

For a buyer, "is it an agent?" is the wrong question. The useful one is when does the model make decisions?

• When the test is written. The model drafts a test, which is stored and replayed.
• While the test runs. The model chooses steps during every execution.
• After the run. The model reads the evidence from a failure and explains or repairs it.

Most products mix all three. Where a tool sits decides how repeatable its results are, what each run costs and what you review.

Why the difference matters for your test results

A test is only useful if red means something is wrong and green means it isn't. In Eradicating Non-Determinism in Tests, Martin Fowler calls non-deterministic tests "useless": when one fails, you cannot tell whether a bug caused it.

Language models are not deterministic by default. Anthropic's glossary notes that even with temperature set to 0, "the results will not be fully deterministic and identical inputs may produce different outputs across API calls." A model that chooses steps at run time adds a new source of variation to every run, on top of the timing and data problems we covered in Flaky Tests: Why End-to-End Suites Break.

That matters more as teams ship more machine-written code. Google Cloud's summary of the 2024 DORA report says rising AI adoption was accompanied by an estimated 7.2% reduction in delivery stability. Verification needs less noise, not more.

Agentic testing compared with the tools it gets confused with

Five approaches are regularly sold as AI testing. Separate them by who decides the steps, and when.

Approach	Who decides the steps	When a model is involved	What you review	Same result run to run?
Record and playback	A person, by clicking	Not at all	The recording	Yes, until the UI changes
AI-assisted script writing	A model, as code	While the test is written	Test code, in a pull request	Yes
Self-healing add-on	A script; the tool repairs locators	When an element isn't found	Repairs, if the tool shows them	Mostly
Plain-language scenarios	A person, in plain language; a model matches them to the page	When the scenario is created, and for some repairs	A readable scenario before it runs	Largely, once confirmed
Run-time agent	A model, during every run	At every step	Evidence after the run	No guarantee

Record and playback is the oldest form: Playwright's codegen, for example, records your actions as test code. Fowler's TestPyramid note calls record-playback tools "almost always a bad idea for any kind of automation" because they make tests hard to change.

AI-assisted script writing keeps the test as code. Playwright documents three test agents: a planner that "explores the app and produces a Markdown test plan", a generator that turns it into test files, and a healer that repairs failing tests. What runs in CI is ordinary code.

Self-healing add-ons find a moved element by other attributes or a model's judgement. That fixes where to click, not whether the test still checks the right thing.

Plain-language scenarios keep the steps in your words. The model's judgement goes into matching each step to the page and turning the steps into a scenario with checks, which a person reads before anything runs.

Run-time agents are agentic testing in the full sense: the agent works out the path to a goal on every run. Valuable for exploration, fragile as a release gate.

What an AI agent for test automation can and cannot do today

Agents are genuinely good at the mechanical parts of testing:

• Drafting a scenario from a description against the live page, with varied test data.
• Re-finding an element that moved after a redesign, using more than one signal.
• Explaining a failure from screenshots, console and network logs, so whoever looks first starts with a hypothesis.
• Trying unusual paths in exploratory sessions and reporting leads for a person to confirm.
• Stopping to ask. Anthropic notes that agents "can then pause for human feedback at checkpoints or when encountering blockers". A good testing agent asks instead of guessing.

They cannot do the following, however well you prompt them:

• Decide what correct means. Whether a discount should stack with a running promotion is a product decision. Testers call this the oracle problem, and an agent's only sources are what you tell it and what the app does today.
• Guarantee the same decisions on every run, for the reasons above.
• Reliably ignore instructions hidden in your pages. OWASP's LLM01:2025 Prompt Injection describes indirect injection, which occurs "when an LLM accepts input from external sources, such as websites or files." Anthropic's computer use documentation warns that instructions on webpages or in images "might override your instructions or cause Claude to make mistakes". A testing agent reads every page of your app, including text your users wrote.
• Get past a CAPTCHA, and it shouldn't try. Use test keys in staging.
• Run cheaply. Every decision is a model call, repeated on every run that improvises.

Common mistakes teams make with agentic testing tools

Letting the agent choose what to test. An agent can list everything a user could do on a page, but not which three journeys carry your revenue. Coverage built from what it noticed is coverage of what was easy to find.

Letting the agent write the expected result. When an agent derives its assertions from the app's current behaviour, it records that behaviour, bugs included. A checkout that shows the wrong total today produces a test that asserts the wrong total from then on.

Treating "the agent finished" as "the test passed". Reaching the last screen is not a check. A run passes only if explicit assertions held: an order number on the page, the right amount, an email in the inbox.

Trusting fluent output. NIST's Generative AI Profile (NIST AI 600-1) lists confabulation, "the production of confidently stated but erroneous or false content", as a core risk, and warns about automation bias, "excessive deference to automated systems". A scenario that reads well can still check the wrong thing.

How to evaluate an AI agent for QA testing

Questions to ask a vendor

Each question exposes a design decision that a demo hides.

1. When does the model decide? While writing, while running, after a failure? Is a model called on a run where nothing changed?
2. What is stored and versioned? Can you read the exact steps and assertions that will run?
3. How are repairs handled? Applied automatically, proposed for approval, or opened as a pull request?
4. What happens when the agent is unsure? Does it guess, fail or ask a person?
5. What is the agent allowed to do? Can you keep it away from delete, refund and cancel?
6. What leaves your environment? Which model providers receive page content and screenshots, for how long, and is it used for training?
7. How is cost metered? What does a failing run cost?
8. What do you keep if you leave? Code, exported scenarios or nothing?

How to run a trial that tells you something

A demo is designed to succeed. A two-week trial on your own staging environment is designed to find the edges:

1. Pick five real journeys, including one behind a sign-in, one that sends an email and one that has broken before.
2. Write each expected result before the trial starts. This is your scoring sheet.
3. Plant known bugs in a staging branch: a wrong total, a disabled button, a missing confirmation email.
4. Re-run unchanged code several times, then rename a label or move a button and run again.
5. Log every human intervention: edits, re-runs and questions answered.

Then score the results:

Measure	What it tells you
Planted bugs caught	Whether a green result can be trusted
Failures on unchanged code	Whether a red result can be trusted
Time to a scenario you trust, review included	The real authoring cost
Repairs made, and how many you could see	Whether healing is auditable
Cost per run, failed runs included	What the tool costs at your volume

A longer hands-on checklist is in End-to-End Testing with AI.

Is there a best AI agent for QA testing?

Not in the abstract. The best fit depends on who will maintain the tests. Grouped by team, from each vendor's own pages as of October 2026:

• Engineers who want tests as code in their repository. Playwright's test agents come with the open-source framework. Checksum writes Playwright tests into your repository and, when a flow changes, "opens a PR for your review". Momentic keeps plain-English steps in YAML in your repo, and its permanent healing "proposes a test-file repair for you to review".
• Testers and product people who want readable tests without code. Rainforest QA, for example, regenerates failed steps and, if the retry passes, "automatically updates your test to the new, passing state", with a history you can restore from. That is the opposite choice to a proposal-first tool.
• Teams that want someone else to own the suite. QA Wolf combines agents with QA engineers, produces Playwright and Appium code, and offers a managed tier.

Our comparison page puts these and other tools side by side, with a source for every claim.

The risks of agentic testing, and the review practices that control them

Risk	What it looks like	Review practice that controls it
Non-determinism	Same code, different result	Review and freeze the scenario before it gates anything
False passes	Green run, broken feature	Assertions on outcomes; plant a bug and confirm the test fails
Hallucinated assertions	A check on text that never appears, or on a current bug	Read every assertion against the expected result you wrote
Silent repairs	A healed step clicks the wrong button	Every repair recorded and reviewable
Destructive actions	Deleted data, emails to real people	Staging, test identities, least privilege, approval for irreversible steps
Prompt injection	Page text changes what the agent does	Test data you control; an allowlist of domains
Data exposure	Screenshots and form data sent to a model provider	Synthetic data; check retention and training terms

These practices match published guidance. OWASP's LLM06:2025 Excessive Agency recommends "human-in-the-loop control to require a human to approve high-impact actions before they are taken". OpenAI's practical guide to building agents names two triggers for human intervention: exceeding failure thresholds, and high-risk actions that are "sensitive, irreversible, or have high stakes". In testing terms, the agent may draft and run, but a person approves anything irreversible and anything it could not finish.

There is also a cheaper control. Anthropic recommends "finding the simplest solution possible, and only increasing complexity when needed", and warns that autonomy brings "higher costs, and the potential for compounding errors". If a reviewed scenario replayed nightly catches the bugs you care about, improvising every step adds cost and variance, not coverage.

Where people stay in charge

Agents change who does the mechanical work, not who is accountable for quality.

Decision	Agent	Person
Which journeys matter	Can suggest	Decides
What the correct outcome is	Cannot know	Writes it down
Steps, test data, locators	Drafts	Reviews
Repairs after UI changes	Proposes	Approves or audits
Why a run failed	Explains	Judges: bug, intended change or noise
Whether to release	Not its call	Decides

The QA role moves towards the right-hand column. Our guide on writing prompts for test generation covers how to write a test's steps and expected result well, and How AI Is Changing the Role of QA Engineers looks at the wider shift.

Where BuniOD fits

BuniOD is an AI QA agent for web applications built on the review-first model described in this guide. You connect your app by URL and write each check as a plain-language prompt: numbered steps with the labels on the page, the values to use and "Confirm …" checks, plus the expected result. BuniOD works through the steps in a real browser, matches each one to an element on the live page and turns the prompt into a scenario with assertions. A person reviews and confirms it before it runs, editing input values or regenerating it from an updated prompt if needed. When the agent can't tell what to do, it raises an agent question for a person instead of guessing.

Confirmed scenarios run in headless Chromium, on demand or once a day at a time you choose. When the UI changes, element lookup falls back from the stored locator to an element fingerprint and then to model-based matching. Each run keeps screenshots, console and network logs, and a plain-language explanation of any failure. Test identities with their own inbox let a scenario check the emails your app sends.

The limits matter for the evaluation above: BuniOD runs desktop Chromium only, schedules one run a day and has no CI/CD integration, so it does not run on every pull request. The how it works page shows the full flow.

Conclusion

An AI agent for QA testing is worth using when you know which decisions it makes and which stay with your team. Prefer designs that turn the model's judgement into something you can read before it counts. Evaluate it on your own application with planted bugs and repeated runs, not on a demo. Keep people in charge of what to test, what correct means, irreversible actions and release decisions. Done that way, agentic testing removes the repetitive work without removing the accountability.