Agentic / Self-Healing Test Framework (Tier 4 LLM-Assisted Re-Binding)

Problem class — when this pattern applies

Locator brittleness is not a universal problem — it is a mature-framework problem. A greenfield Playwright suite with disciplined data-testid coverage and a weekly release cadence will barely register locator drift in its first year. The pattern I am describing here targets a specific failure mode that emerges in frameworks that have been running long enough to accumulate several cycles of UI change: a growing share of test failures that, on investigation, turn out to be locator drift rather than real application regressions. The flake noise is high enough that engineers stop trusting the CI gate, start skipping investigation, and the suite begins to rot.

The teams this pattern fits share a few specific properties. Locator drift must be the dominant flake category — not timing, not data, not environment instability. If the dominant cause is anything else, Tier 4 is solving the wrong problem. The team must already have observability over failures: a test dashboard that surfaces failure reasons, flake rate per test, and a history of which tests break across which releases. Without that baseline the guardrails have nothing to anchor to. The team must also have an AI-aware engineering culture — not deep ML expertise, but engineers who are comfortable reasoning about non-deterministic outputs and holding a reviewer-in-loop discipline. And the economics have to work: Tier 4 invokes an LLM in the failure path, which carries a per-call cost; that cost needs to be weighed against the engineer time currently absorbed by manual locator repair.

This is an opt-in layer on top of a mature framework, not a replacement for the foundational work. Teams that have not yet invested in resilient locator strategy, semantic selectors, and page object hygiene should do that first. Tier 4 is the residual handler for the drift that survives all of that.

---

Architecture / design decisions

Self-healing is not one thing. The mistake I see most often in how this topic is discussed is conflating Tier 4 LLM-assisted re-binding with the concept of self-healing itself, as if the only options are "no self-healing" and "the LLM figures it out." The architecture I use frames healing as a five-tier continuum where each tier is a progressively more expensive intervention, invoked only when the tiers before it are exhausted.

Tier 1 is the foundation: resilient locator strategy. Every selector written with semantic intent — data-testid attributes placed by the development team, ARIA role queries, visible-text anchors — is a locator that survives the bulk of cosmetic UI changes. Tier 1 is not a self-healing mechanism; it is the reason most tests do not need healing at all. If Tier 1 is well-implemented, the subsequent tiers are invoked rarely. Tier 2 is Playwright's built-in auto-wait and retry: the framework waits for elements to be actionable before interacting, which handles transient-miss failures caused by animation, lazy rendering, and network-induced timing variance. Tier 3 is heuristic re-binding: a rule-based fallback table defined at authoring time — if data-testid="submit-button" is not found, try role=button, name=Submit, then button:has-text("Submit"). This is deterministic, cheap to reason about, and handles the predictable renames that follow a UI component library upgrade.

Tier 4 invokes an LLM only when Tiers 1–3 have been exhausted and the element is still not found. At that point the framework captures a filtered DOM snapshot (PII redacted, irrelevant subtrees pruned), the original failing locator, the test's intent description, the current page URL, and the healing history for this test. That context is sent to the LLM, which returns a structured JSON response: a proposed locator, a confidence score, a rationale, and the DOM evidence it used. The response is gated on a confidence threshold before being applied. High-confidence proposals are applied immediately with an audit event emitted. Mid-confidence proposals are applied but flagged for review. Low-confidence proposals cause the test to fail loudly — the LLM is not confident enough, and the failure is routed to human triage. The full Tier 4 system — including the Promptfoo eval harness, the promotion-to-source pipeline, and the governance surface — was built and proven end-to-end as part of the AgentQE Continuum, an R&D project that is the origin of this architecture.

The reason Tier 4 is opt-in rather than default-on is fundamental to the architecture. Self-healing is non-deterministic. Accepting it across the entire suite — without explicit per-test opt-in — means accepting silent locator changes everywhere, including in tests where a loud failure is the correct and necessary outcome. For critical-path tests: login flows, payment flows, regulated submission forms, any test where a false positive carries real-world consequence — Tier 4 must be explicitly disabled. The self-healing tag is a deliberate declaration that this test is tolerant of locator drift and the traceability discipline surrounding it is in place.

The intent description is the grounding contract for Tier 4. It is a short human-authored string attached to each locator interaction: "Submit form button on the registration page." This is what prevents the LLM from proposing a plausible-looking but semantically wrong locator — one that points to a different button that happens to match the DOM shape. The LLM must ground its proposal in the intent, not just the structural similarity. Tests that do not have intent descriptions cannot participate in Tier 4 healing; the framework enforces this as a precondition.

The audit trail is non-negotiable in this architecture. Every healing event — whether it results in a test pass or a subsequent failure — is logged with the original locator, the proposed locator, the confidence score, the LLM rationale, the test identifier, the run identifier, and the timestamp. The DOM snapshot is stored compressed alongside the log entry. This is not optional overhead. The audit trail is what makes Tier 4 defensible: every healed locator can be reconstructed, reviewed, and challenged. Without it, self-healing is invisible technical debt accumulating in the test suite.

---

Architecture at a glance

Loading diagram…

The framework, not the LLM, is the source of truth. The LLM's output is treated as a structured proposal that is gated, audited, and reversible. Every confidence band has a deterministic destination — there is no path where the LLM silently changes test behaviour.

---

Guardrails (the non-negotiables)

The guardrails are not bolt-on safety theatre. They are the reason Tier 4 is safe to deploy at all. Without them, self-healing is a mechanism that turns real regressions into passing tests and obscures test design rot behind a layer of automated repairs.

Per-test heal cap. Each test carries a maximum number of Tier 4 healings it will accept before the framework stops healing it and flags it for human review. The default is three healings in a rolling month-window. A test that heals more than three times in a month is not a locator-drift problem — it is a test design problem or an indicator of sustained UI churn in that area. The cap forces the signal to surface rather than allowing the healing loop to run indefinitely.

Repeated-heal human-review flag. When a test crosses the per-test heal cap, it is automatically added to a review queue surfaced in the quality dashboard. The review queue shows: test name, number of healings, original locators, proposed locators, confidence scores, and rationale. A QE must clear the flag — either by redesigning the test, accepting the healing as permanent and updating the source, or tagging the test for deletion. The queue is never silently cleared by the system.

Daily quality dashboard. A daily report surfaces healings per test, healings per page area, healing-to-subsequent-re-fail rate (healings that fixed the test versus healings that led to a failure on the next run), and confidence score distribution. The dashboard is always-on. If Tier 4 is working correctly, the healing rate should trend toward zero over time as healed locators are promoted to source and the source stabilises. A flat or rising healing rate is a signal that the framework is masking drift rather than resolving it.

Intent description as grounding contract. A test step that does not have an intent description cannot invoke Tier 4. The framework enforces this as a build-time check. This is a guardrail against low-context healing: the LLM must have a human-authored statement of what the step is trying to do, not just a locator string and a DOM snapshot.

Opt-in via explicit tag — not default-on. Tier 4 healing is enabled per test via an explicit @self-heal tag. Critical-path tests — login, payment, regulated submission flows — must never carry this tag. The convention is enforced by a CI lint step that fails the build if a test tagged @critical-path also carries @self-heal. Default-off prevents silent recovery from becoming the default mode for the entire suite. For a worked example of how these guardrails — locator discipline, confidence-threshold tuning, and the quality dashboard — were applied in a production client engagement, see the AI-Augmented Playwright Test Pipeline case study.

Promptfoo eval harness. A Promptfoo probe suite runs against the Tier 4 prompt with intentionally broken locators and known-good DOM changes to verify healing is correct, and with adversarial probes — DOM changes where the element has been removed entirely, not just renamed — to verify Tier 4 rejects and fails loud. The eval gates any change to the Tier 4 prompt or the confidence thresholds. This is the same audit-trail discipline applied to grader snapshots in an LLM-as-Judge pipeline: the resolved prompt is frozen at version creation, and regressions against the golden set block deployment.

---

Code snippets

1. Tiered fallback fixture (TypeScript + Playwright)

// fixtures/self-heal.fixture.ts
import { test as base, type Locator, type Page } from '@playwright/test';

type HealContext = { intentDescription: string; fallbacks: string[] };

export const test = base.extend<{ healingLocator: (primary: string, ctx: HealContext) => Promise<Locator> }>({
  healingLocator: async ({ page }, use) => {
    const resolve = async (primary: string, ctx: HealContext): Promise<Locator> => {
      // Tier 1: primary locator
      const loc = page.locator(primary);
      if (await loc.count() > 0) return loc;
      // Tier 2: heuristic fallbacks
      for (const fallback of ctx.fallbacks) {
        const fb = page.locator(fallback);
        if (await fb.count() > 0) { await emitFallbackEvent(primary, fallback, ctx); return fb; }
      }
      // Tier 3: session refresh + one retry handled by Playwright auto-wait upstream
      // Tier 4: LLM re-binding (opt-in only — guarded by @self-heal tag at test level)
      return invokeTier4(page, primary, ctx);
    };
    await use(resolve);
  },
});

async function emitFallbackEvent(original: string, used: string, ctx: HealContext) {
  // Write to audit log; omitted for brevity
}

async function invokeTier4(page: Page, original: string, ctx: HealContext): Promise<Locator> {
  // DOM snapshot, LLM call, threshold gate, audit emit — see full implementation
  throw new Error(`Tier 4 stub: implement LLM re-binding call here`);
}

2. Tier 4 LLM prompt template (truncated for illustration)

You are a test automation engineer. A Playwright test step has failed because the
following locator was not found in the current DOM.

ORIGINAL LOCATOR: {{original_locator}}
TEST INTENT: {{intent_description}}

CURRENT DOM SNAPSHOT (filtered, no PII):
{{dom_snapshot}}

STRUCTURAL DIFF vs LAST-KNOWN-GOOD DOM:
{{dom_diff}}

PRIOR HEALING HISTORY FOR THIS TEST:
{{healing_history}}

Return a JSON object with these fields:
- proposed_locator: a valid Playwright selector string
- confidence: float 0.0–1.0
- rationale: why this locator matches the stated intent
- dom_evidence: array of DOM fragments used as evidence

Ground your proposal in the TEST INTENT, not just structural similarity.
If you cannot identify a confident match, return confidence below 0.7.

3. Audit log entry (TypeScript type)

type HealAuditEntry = {
  testId: string;
  runId: string;
  timestamp: string;          // ISO 8601
  originalLocator: string;
  proposedLocator: string;
  confidence: number;
  rationale: string;
  domEvidence: string[];
  outcome: 'applied' | 'applied_flagged' | 'rejected';
  subsequentResult?: 'pass' | 'fail';  // populated after test re-run
  domSnapshotRef: string;     // path to compressed snapshot artefact
};

---

When I'd brief this

This pattern is right for teams that have already shipped a mature Playwright or Selenium framework, have observability in place over failure categories, and have confirmed that locator drift — not timing, data, or environment instability — is the dominant flake driver. The investment only makes sense when the engineer-hours absorbed by manual locator repair are measurably redirected by automation, and where the audit discipline and daily-dashboard practice are things the team will actually maintain.

It also requires an AI-aware engineering culture: not deep ML expertise, but engineers who are comfortable reasoning about non-deterministic outputs, understand why opt-in matters, and will hold the reviewer-in-loop discipline for promoted healings rather than treating the dashboard as noise. Teams that are not yet at that cultural point will get more value from tightening Tiers 1–3 first.

The pattern is wrong for black-box vendor tools that already provide self-healing (Testim, Mabl) — if the vendor abstraction fits, use it rather than building Tier 4. It is also wrong for critical-path tests where a loud failure is the intended outcome and for applications undergoing structural architectural change, where wholesale DOM redesigns require test redesign, not healing.

---