AgentQE Continuum — Event-Driven, Self-Healing QA Validation System

Project context

Every client engagement I take on in AI-augmented testing runs into the same friction: CI failures land, engineers triage manually, and the gap between "test failed" and "root cause understood" consumes disproportionate engineering time. I wanted to prove out a complete architectural answer to that problem — not as a deliverable for a client, but as something I owned end-to-end and could iterate on without external constraints.

AgentQE Continuum is that answer. I designed and built it as a personal engineering R&D project: an event-driven pipeline that connects to a GitHub Actions webhook, processes Playwright test failures through a LangGraph analysis pipeline, applies a confidence-scored decision engine, and (in Phase 3) generates and verifies autonomous fixes. Every architectural pattern in the system — the LangGraph node topology, the deterministic fallback strategy, the pg-boss job queue design, the multi-provider LLM tier routing — is something I now carry into client engagements with production evidence behind it rather than theory.

The project spans 6 work streams and 87 planned tasks across the docs. It is ongoing.

The win

"I designed and built a full event-to-decision pipeline in TypeScript — GitHub webhook to LangGraph to confidence-scored output — that handles the complete CI failure lifecycle: HMAC-verified ingestion, Playwright JSON parsing with flake detection, pgvector RAG retrieval of historical failure patterns, LangGraph 7-node analysis graph with AI clustering and deterministic Levenshtein fallback, a decision engine that emits escalate / report / annotate_pr with explainable confidence scores, Slack and Jira notification on escalation, and a self-healing sub-graph (Phase 3) that classifies failure type, generates a ts-morph AST fix, triggers a CI verification run, and merges only if no previously-passing test regresses. The key architectural contribution is that every decision is explainable without AI involvement — the system never produces a black box output."

High-level architecture

Loading diagram…

The pipeline is fully asynchronous: the webhook endpoint returns immediately and the analysis runs via pg-boss. Every AI node has a deterministic fallback path, and every decision carries a degraded boolean so downstream consumers always know whether an LLM was involved.

What was built (phase-by-phase)

Architectural patterns proven (the portable lessons)

LangGraph as orchestration with deterministic fallback at every node. The pipeline is designed so that every AI node has a deterministic path that fires when the LLM is unavailable. clusterFailures falls back to Levenshtein distance clustering (70% similarity threshold); generateSummary falls back to a template. The circuit breaker — 3 consecutive failures open for 30 seconds — coordinates the switch. No event is dropped; every decision carries a degraded boolean. This pattern is directly portable to any client pipeline where LLM availability cannot be guaranteed. The Tier 4 LLM re-binding sub-graph in particular has been generalised into a reusable reference architecture — the Agentic Self-Healing Test Framework pattern doc covers the tiered locator recovery model, confidence gating, and audit trail design that originated here.

Confidence-scored decision engine with explicit explainability. The decision engine computes confidence from four weighted signals (error consistency, stack-trace origin match, historical pattern match, degraded mode penalty) and maps the result to a human-readable action with supporting evidence. Engineers can inspect exactly why the system chose to escalate rather than report. This is more trustworthy than an LLM emitting a string decision with no audit trail, and it is the pattern I bring to client triage tooling.

pg-boss for resilient job queuing with PR-level locking. Choosing a PostgreSQL-native queue eliminated a Redis dependency and unified the operational surface. pg-boss handles job deduplication, dead-letter queues, cron scheduling, and retry with backoff. PR-level advisory locks in the event processor prevent race conditions when multiple webhooks arrive for the same pull request concurrently — a problem that emerges in high-velocity repos and is invisible until it bites in production.

Prompt versioning with the database as the source of truth. Prompts are versioned and managed via the /admin/prompts dashboard, not committed as static strings. The re-analyse button on the run detail view re-runs the pipeline with the current prompt version against already-ingested signals — this is how prompt regression testing works in a live system. Human overrides feed back into prompt tuning via the learning layer.

Multi-provider LLM tier routing with a single abstraction boundary. llm-client.ts exposes a single call surface; the provider (Anthropic, OpenAI, Groq, Together, Fireworks, OpenRouter, Ollama, or any OpenAI-compatible backend), model, and temperature are resolved at startup from env vars per tier (high / medium / low). Switching the clustering model from Claude Sonnet to a self-hosted Llama 3.3 on Groq is one env var change with zero code changes. I now apply this abstraction to any client system where LLM cost or vendor lock-in is a concern.

Stack rationale

TypeScript-only monorepo (no Python). The entire system — Express API, LangGraph pipeline, Next.js dashboard — runs on Node.js with strict TypeScript. A single language, a single type system, and a single pnpm workspace means end-to-end type safety without a Python service boundary. The LangGraph TypeScript SDK (@langchain/langgraph) is a first-class API, not a wrapper — this was a deliberate bet that the TS ecosystem for AI pipelines had matured enough to go language-pure. Many of the same architectural bets — TypeScript-native framework, LLM-driven triage, confidence-scored decisions — were applied in a production energy-sector client engagement; the AI-Augmented Playwright Test Pipeline case study shows how these patterns translate under the constraints of a large enterprise QA function.

pgvector over a dedicated vector database. Storing 1024-dimensional Voyage AI embeddings in the same PostgreSQL instance that holds events, clusters, and decisions keeps the operational stack simple: one database, one backup policy, one connection pool. PGVectorStore from @langchain/community handles upsert and HNSW index queries. The retrieval problem — find failure patterns similar to this new failure — is vector similarity search, not graph traversal, so a dedicated vector store adds complexity without benefit.

LangGraph over LangChain expression chains. LangGraph's graph-based execution model maps directly to the pipeline topology (ingest → validate → cluster → summarise → decide). Built-in state management (PipelineState) eliminates custom plumbing for passing context between nodes. Parallel node execution (clusterFailures + analyseDiff concurrently) is a one-line change. LangGraph's PostgreSQL checkpointer is used specifically for the fix sub-graph, where the graph needs to suspend mid-execution while waiting for a CI webhook — a checkpoint-based design that LangChain chains cannot express.

pg-boss over Redis-based queues. Redis adds an operational dependency and a separate persistence boundary. pg-boss runs on the existing PostgreSQL instance, supports advisory locks (critical for PR-level deduplication), and provides dead-letter queues and cron scheduling without a separate scheduler service. The scheduler worker role must never be scaled past one replica — pg-boss does not deduplicate cron-job inserts by handler, so two schedulers fire every job twice. This is a known and documented constraint, not a deficiency.

Current status

Phases 1 through 4 are complete. Phase 4 — the Next.js dashboard — shipped on 2026-05-19. The pipeline is fully operational: webhook ingestion, LangGraph analysis, confidence-scored decisions, PR comments, Slack alerts, Jira tickets, RAG retrieval, and the observability stack are all live.

Phase 3 — the autonomous fix generation engine (app/src/fix/ directory, ts-morph AST operations, verification gate, autonomy level progression) — is planned and architecturally specified but not yet implemented. The fix sub-graph structure, guardrails (test files only, max 50 lines across 5 files, 120-second timeout), and the four autonomy levels (L0 shadow through L3 auto-merge) are fully designed in the architecture document. Implementation is the next phase of active development.

The patterns in this system are what I now bring to client engagements. AgentQE Continuum is where they were proven.