#Agent Quality Kit

An LLM agent harness with versioned prompts, runtime guardrails, automated evals and a CI quality gate — the full loop that makes agent behaviour predictable, measurable and safe to ship.

The example agent reviews code diffs and returns structured findings. The agent is the pretext; the platform around it is the project.

Runs offline, with no API key: the default provider is a deterministic mock, so make eval works on your laptop and in CI at zero cost and with zero variance. Switching to the real API is one environment variable.

---

#Why this exists

Calling an LLM is easy. Everything after that is the actual engineering problem:

The thesis: the model is the only non-deterministic component in the system. Anything you can move out of the prompt and into the harness becomes a guarantee instead of a probability. What's left — judgement — is measured by evals and protected by gates.

Source code and identifiers are in English; inline comments are in Brazilian Portuguese, which is the author's working language.

---

#Running it

pip install pyyaml          # the only dependency in mock mode

make test                   # 57 deterministic harness tests (<1s, no API key)
make run                    # run the agent on one diff: cost, tokens, trace
make eval                   # full eval suite
make eval PROMPT=1.0.0      # same suite, previous prompt version
make gate                   # compare against the baseline, fail on regression
make trace                  # summary of the last trace

Against the real API:

pip install anthropic
export AQK_PROVIDER=anthropic ANTHROPIC_API_KEY=...
make eval

---

#What the commands show

#make eval — stratified measurement, with uncertainty and cost

  EVAL  code_review@1.1.0 (hash f469c963aa5c) · model claude-opus-5 · provider mock
  suite=full  k=1  dataset=63fe5faadf9b
  --------------------------------------------------------------------
  overall pass rate    92.3%  (12/13)  CI95 [66.7%, 98.6%]
  critical cases      100.0%  (n=5)  ← gate threshold: 100%
  --------------------------------------------------------------------
  by category:
    correctness      100.0% (3/3) ████████████████████
    false-positive     0.0% (0/1)
    noise            100.0% (2/2) ████████████████████
    performance      100.0% (2/2) ████████████████████
    security         100.0% (5/5) ████████████████████
  --------------------------------------------------------------------
  total $0.0628 · mean/case $0.0048 · tokens 11,358 · cache hit 69.0%

Three deliberate decisions are visible in that report:

1. A confidence interval next to the rate. With 13 cases the interval is wide — and saying so is the honest thing to do. It is what stops anyone from celebrating a 3-point "improvement" that fits entirely inside the noise.
2. Per-category metrics, not just the average. A global average hides localized regressions; the gate looks slice by slice.
3. The suite is deliberately not all green. Case cr-013 documents a known false positive in the detector (a parameterized query flagged as SQL injection). A suite that is always green measures nothing. The gate demands no regression, not perfection.

#make gate — a subjective standard turned into an objective, blocking check

Simulating a pull request that reverts the prompt to the previous version:

  GATE  code_review@1.0.0 vs baseline 1.1.0
  --------------------------------------------------------------------
  metric                    baseline     current       delta
  overall pass rate            92.3%       38.5%     ▼ 53.8%
  critical                    100.0%        0.0%    ▼ 100.0%
    security                  100.0%        0.0%    ▼ 100.0%
    performance               100.0%        0.0%    ▼ 100.0%
  mean cost/case             $0.0048     $0.0037   ▼ $0.0011
  --------------------------------------------------------------------
  ❌ GATE BLOCKED
     [critical_must_pass] critical cases failed: ['cr-001', ...]. Threshold is 100%, no margin.
     [category_regression:security] security: 100.0% → 0.0% (cases: [...])

Note that cost went down and the gate blocked anyway. Cheaper and worse is not an optimization.

Gate rules (evals/gate.py):

---

#Architecture

                      ┌──────────────── HARNESS (deterministic) ─────────────────┐
   diff ──► IN-GUARD ─┤  context assembly  →  loop  →  budget  →  telemetry      ├─► OUT-GUARD ──► findings
            (secrets, │        ▲                │                                │   (schema,
             size)    │        │                ▼                                │    secrets,
                      │        │         ACTION-GUARD → tool                     │    grounding)
                      │        └── RESULT-GUARD ◄─┘                              │
                      └──────────────────────────────────────────────────────────┘
                                            │
                 prompts/ (versioned)       │       traces/ (OTel GenAI attrs)
                                            ▼
                              evals/ ──► gate ──► CI blocks the merge

---

#Engineering decisions (and their trade-offs)

A manual loop instead of a framework. The SDK ships a tool runner and that is the right way to start. The loop here is manual because a platform needs things the runner does not expose: its own budget accounting, guardrails in the middle of the loop, a compaction policy, record/replay for testing, and telemetry carrying the organization's own attributes. Buy first, build when the platform matures.

Context split into a stable prefix and volatile content. Rules and tool schemas live in system (cacheable); the diff, which changes per case, goes in the user message. Prompt caching is a prefix match — one different byte at the front invalidates everything after it. That is also why tool ordering is deterministic (tools.py), with a test enforcing it.

The input guardrail masks the secret's value but preserves the shape of the code. api_key = "prod_live_..." becomes api_key = "REDACTED_SECRET_VALUE". The secret never leaves the building, and downstream analysis can still see that a hardcoded credential is there. Masking in a way that destroys the syntax would blind the review.

Anything a tool returns is data, never instruction. It comes back wrapped in <dados_nao_confiaveis>, and if it contains an injection pattern the run drops privilege: mutating tools then require human approval. The defence does not rely on detecting the injection — it relies on leaving the attack with no channel to act through.

Tool errors come back as structured observations. file not found. Nearby files: [...] lets the model correct itself on the next turn. An exception would kill the agent. This is the cheapest large quality lever in agent work.

LLM-as-judge only for the residue. The ladder is: structural assertion → real execution (run_checks) → judge. The judge (evals/judge.py) uses a binary rubric, asks for evidence before the verdict, and declares its calibration (kappa=0.78 against 60 human labels). Below KAPPA_FLOOR = 0.60 it refuses to be used as a gate — an uncalibrated judge is automated opinion.

Template delimiter is {{@var}}, not {{var}}. The injected content is source code, and Blade/Handlebars/Jinja use {{ }}. The collision broke every review of a .blade.php file — a bug the eval suite itself surfaced.

---

#Statistics the runner uses

• Wilson interval instead of the normal approximation: better behaved at small n.
• McNemar's test (scorers.mcnemar_p) for paired comparison between versions. Because both sides run on the same cases, only the discordant ones carry information: ~100 paired cases detect what would need ~900 per arm in an unpaired comparison.
• k samples per case (--k 3): the system is stochastic, so the criterion is a pass rate over k, not pass/fail from a single run.

---

#Layout

prompts/code_review/1.0.0.yaml   prompt + model + params + changelog + owner
prompts/code_review/1.1.0.yaml   version with security rules (the A/B in the demo)
prompts/judge/1.0.0.yaml         binary judge rubric, calibration declared

src/aqk/                         the harness
evals/dataset/code_review.yaml   13 cases with category, severity and PROVENANCE
evals/fixtures/                  the diffs
evals/{runner,scorers,judge,gate}.py
evals/baseline.json              the current ruler (promoted only in a reviewed PR)
tests/                           57 deterministic tests
.github/workflows/ai-quality.yml smoke on PRs, full on merge, plus a nightly cron

Every dataset case carries a provenance field — the incident that produced it. That is what stops someone deleting an "odd looking" case six months later. Every incident becomes a permanent case.

CI also runs on a cron, not only on pull requests: in LLM systems a good share of regressions come from outside your commit (a model update, a changed index, an altered tool).

---

#What this project deliberately is not

• It is not a good vulnerability detector. tools.RULES is plain regex and it gets things wrong — case cr-013 documents exactly that. The object of study is the quality pipeline, not state-of-the-art static analysis.
• There is no RAG. The context fits in the window here; adding retrieval would be complexity without a problem.
• Numbers in mock mode are simulated. Token counts are estimated; the path the cost travels (per call → trace → gate) is the production one.
• No session persistence or resume. The loop is short by the nature of the task.

#License

MIT