Course: 2B — Securing & Attacking Harnesses and LLMs Module: B12 — Harness Security Assessments as a Service Duration: 60 minutes Level: Senior Engineer and above Prerequisites: B0–B11 complete. This is the capstone methodology module — it synthesizes everything you built and attacked into a repeatable, scoped, deliverable engagement.
B0 built the legal control plane. B1 mapped the surface. B2–B8 built the defenses. B9 gave you the checklist. B10 gave you the chains. B11 gave you the governance layer. None of that is a service until you can scope it, price it, deliver it, and retest it. This module turns the techniques into a profession. The buyer is a CISO or an AI security lead, and they will measure you on the artifact you hand them, not on the techniques you ran.
After completing this module, you will be able to:
Every prior module gave you a technique, a control, or a framework. None of them gave you an engagement. A CISO who hires you does not want a technique — they want a defensible answer to "is my agent safe to ship, and if not, what is the residual risk and what do I fix first?" That answer is an artifact, produced by a method, bounded by a scope, priced as a service. This module is the operational layer that turns B0–B11 into that artifact.
The temptation at the capstone is to treat the service as "run B9's checklist, hand over the output." That is half an engagement. A real agent security assessment has six phases, and only two of them are the testing that B9 and B10 supply. Scoping is where the legal control plane (B0) gets written into the contract; reporting is where the scored output (B9) becomes a deliverable a board can read; retesting is where the residual-risk discipline (B0.2) proves a control actually moved the needle. Skip any phase and the engagement is either illegal (no scope), unsellable (no report), or unverifiable (no retest).
The buyer matters. From the course spec: this module's buyer is the CISO or AI security lead. That buyer reads frameworks, not exploits. They will ask, before anything else, "which control framework does this map to?" (B9 OWASP, B10 Microsoft, B11 NIST AI RMF), "what is the residual risk in numbers?" (B9's measured rows), and "can my auditor read this?" (B11's audit trail). The methodology in this module is calibrated to that buyer. A beautiful exploit chain that does not map to a control row will not move budget. A measured residual that maps to a NIST AI RMF Measure function will.
Three sub-sections, twenty minutes each:
The backbone of the engagement, and the front-end work that determines whether the testing is legal, scoped, and priced correctly.
The methodology adapts the pentest standards — PTES (Penetration Testing Execution Standard) and NIST SP 800-115 ("Technical Guide to Information Security Testing and Assessment") — to the agentic surface. The phases are unchanged in name; what changes is the content of each, because the target is an agent, not a network. Every phase ties to a prior module.
Phase 1 — Scoping. What systems, what surfaces, what model versions, and — the AI-specific step — what provider authorizations. This is where B0's legal control plane gets written into the engagement contract. The output is the SOW and the JSON scope file (from B0's lab) that separates deployer-controlled from provider-controlled surfaces and carries the provider_authorization field per technique. A scoping conversation that does not produce a scope file with that separation is incomplete.
Phase 2 — Reconnaissance. Map the agent's surfaces. B1's threat model is the scaffold: the loop, the tools, the memory, the provider, the identity, the sandbox, the inter-agent edges. The deliverable is a surface map — every input boundary, every tool, every persisted state, every external call, every sub-agent. This is not "port scan the agent"; it is "enumerate the attack surface as B1 defines it." Reconnaissance is also where B10's capability-disclosure mode (Mode 7) runs first: the agent's own description of what it can do is the cheapest surface map you will ever get.
Phase 3 — Vulnerability discovery. Run B9's OWASP checklist (ASI01–ASI10, each with its attack procedure and test) and construct B10's attack chains (the seven failure modes, plus the zero-click HITL bypass chain). Discovery is two-track: the checklist track (B9) tests each control deterministically or measures it probabilistically; the chain track (B10) constructs multi-step compound-intent attacks that slip between controls. Neither alone is sufficient — B9 finds missing controls; B10 finds gaps between controls that exist.
Phase 4 — Exploitation / validation. Confirm findings with B0's minimum-proof discipline. A candidate finding is not a finding until it has been reproduced with the minimum-evidence field set (model version, exact prompt, sampling params, success rate over N attempts, scope reference). Validation is where most engagements lose rigor: a single successful jailbreak reported as a finding is an anecdote; the same jailbreak at 60% over 100 attempts is a finding. This is also where the dual-use discipline bites — a validated jailbreak is now a dual-use artifact and the RoE's disclosure clauses take over.
Phase 5 — Reporting. The deliverable. Covered in B12.2. The output of B9's checklist executor (the scored table with 8 PASS/FAIL + 2 MEASURED rows) plus B10's chain findings, formatted into the engagement report structure.
Phase 6 — Retesting. After remediation, re-run the discovery and validation phases and measure residual risk before/after. Covered in B12.3. The retest is the proof the controls moved the needle — and the recurring-revenue layer of the service, because every release re-runs it.
The six phases are a loop, not a pipeline. A retest feeds back into scoping for the next engagement. A finding that escapes the checklist routes a new row into B9 for the next release. The methodology is a cycle the client buys into across releases, not a one-shot audit.
Scoping an agent assessment is harder than scoping a network pentest because the surface is less legible. A network has IP addresses and ports; an agent has tools, memory, prompts, model versions, and inter-agent edges, most of which are not documented anywhere the client can point you to. The scoping conversation must enumerate four things:
gpt-model-v3 is meaningless if the client ships v4 next week. The scope file (from B0) carries model_versions_in_scope with pinned checkpoints, and every finding records the version it was tested against. A scope with "the model" and no version is a scope that cannot be retested.Pricing an agent assessment is driven by three factors, weighted into an effort estimate:
The honest pricing rule: do not sell a depth you cannot deliver with rigor. A "chain-depth" engagement priced at checklist-depth hours is an engagement that will ship a B10 chain that has not been validated end-to-end. Underpriced assessments produce under-rigored reports, and under-rigored reports are how an agent ships with a real vulnerability the assessment said was fixed.
The SOW is where B0's clauses get written into the engagement contract. A traditional pentest SOW covers scope, duration, and deliverables. An AI red-team SOW must additionally carry the clauses B0.1 specifies — these are not optional, and an SOW without them is an SOW that fails at the moment a serious finding appears:
The lab for this module has you write a sample SOW with all seven clauses for a realistic client. That SOW is the artifact the client's counsel signs; it is also the artifact the assessment practice reuses across engagements, parameterized by client.
The deliverable. This is the scored artifact B9's checklist executor produces, formatted into a structure a CISO can read and a regulator can audit.
The engagement report has five sections. Every framework the buyer reads (NIST AI RMF, EU AI Act conformity, ISO 42001) expects roughly this shape; the agent-specific content is what fills it.
One page. The overall residual-risk posture in plain language, the count of findings by severity, the top three risks in priority order, and a single-sentence ship recommendation calibrated to the buyer's risk appetite. The executive summary is the only section most of the buyer's organization will read; it must stand alone. The ship recommendation is never "secure" — it is "ship with characterized residuals at X% and Y%, conditional on remediating the two Critical findings" or "do not ship; the compound chain in Finding F-04 reaches lateral movement." B9's discipline carries: the summary reports the measured residuals, not a pass/fail.
The body of the report. One entry per finding, each with the same field set so findings are comparable and the table is machine-readable. The fields:
| Field | Content |
|---|---|
| Finding ID | F-01, F-02, ... (stable across retests) |
| Title | One-line description |
| Severity | Critical / High / Medium / Low (CVSS-like, calibrated to agent impact) |
| Taxonomy reference | OWASP ASI row (B9) and/or Microsoft failure mode (B10) — e.g., "ASI01 Goal Hijacking; Microsoft Mode 2 (goal hijack drift)" |
| Attack procedure | The concrete steps: the injection vector, the tool reached, the impact. For B10 chains, the multi-step sequence with each step's approval gate. |
| Evidence | B0's minimum-evidence field set: model version, exact prompt (or a redacted reference if dual-use), sampling params, success rate over N attempts, timestamp UTC, scope reference. |
| Residual risk | The measured rate after any in-engagement mitigation (e.g., "injection success 60% over 100 attempts; residual 4% after L4 taint gate enabled mid-engagement"). For PASS/FAIL rows, the binary result. |
| Remediation | The control to add or fix, routed to the module that builds it (B2–B8), with an effort estimate. |
Every finding maps to a taxonomy reference. A finding with no OWASP row and no Microsoft mode is a finding that has not been classified — and classification is what makes it actionable for the client's governance layer (B11). The taxonomy reference is the bridge from "we found a bug" to "here is the control framework row it satisfies."
A two-dimensional view: the controls that exist (rows) against the surfaces they cover (columns), with each cell marked Present / Absent / Partial / Mismeasured. This is the gap analysis. A control that is "Present" on the system-prompt surface but "Absent" on the retrieval-store surface is a finding (ASI04 not defended on retrieval). A control that is "Mismeasured" — present but never tested — is a B11 audit-trail gap. The control matrix is what a regulator reads to ask "did you actually verify these controls, or just document them?"
The findings, prioritized by severity and dependency, sequenced into a release plan. Critical findings first; findings that unblock other findings next (e.g., the principal-binding fix that makes the tool-contract fixes testable). Each remediation item routes to the module that builds the control (B2–B8), with an owner and an estimated effort. The roadmap is the artifact the client's engineering team works from between the report and the retest.
The reproducibility layer. The methodology (the six phases), the scope (the SOW and scope file), the model versions tested, the tool versions used, and an index to the evidence store (with the B0 data-class classification redacted as appropriate). This appendix is what makes the report auditable: a third party can reconstruct what was tested, against what, with what tools, and verify the findings reproduce.
This is the structural connection that makes B12 a synthesis module rather than a new surface. B9's checklist executor, when run to scope-completion, produces a table with one row per ASI risk — eight PASS/FAIL and two MEASURED. That table is the findings section's backbone. B12 does not re-test; it packages. The B9 rows become findings (F-01 through F-10, roughly); the B10 chains become the additional findings (F-11+) that the checklist alone would miss; the control matrix is the B9 risk-to-module mapping (ASI → B2/B3/B4/B5/B7/B8) read as a coverage view; the remediation roadmap is the B9/B10 findings routed to their building modules.
The buyer sees one report. The methodology underneath is B9 + B10, governed by B0, evidenced by B11's audit trail. That is the capstone: every prior module contributes a layer, and the report is where they stack into a single deliverable.
The lab builds a report generator that takes the JSON output of B9's checklist executor and produces the structured engagement report. The generator is opinionated: it enforces the field set per finding, it refuses to emit a finding without a taxonomy reference, and it computes the executive-summary residual posture from the measured rows. The generator is what makes the practice repeatable — the same input produces the same report structure across engagements, so a client can compare release N to release N+1 and see whether the residuals moved.
from __future__ import annotations
from dataclasses import dataclass, field
from typing import Literal, Optional
Severity = Literal["Critical", "High", "Medium", "Low", "Info"]
ResultType = Literal["PASS", "FAIL", "MEASURED", "N/A"]
@dataclass
class Finding:
"""A single assessment finding. The field set enforces B0's minimum-evidence
discipline and the taxonomy-reference requirement (no orphan findings)."""
finding_id: str
title: str
severity: Severity
taxonomy_ref: str # required: OWASP ASI row and/or Microsoft mode
attack_procedure: str
model_version: str # pinned checkpoint — a finding without it is untestable
success_rate: Optional[str] # "60% over 100 attempts" for MEASURED; None for PASS/FAIL
sampling_params: dict # temperature, top_p, etc. — reproducibility
scope_reference: str # the SOW/scope-file clause that authorized this
result: ResultType
residual_risk: str # measured residual after any in-engagement mitigation
remediation_module: str # which of B2-B8 builds the fix
timestamp_utc: str
def validate(self) -> list[str]:
"""Return a list of validation errors. A finding that fails validation
is a finding that should not ship in the report."""
errors: list[str] = []
if not self.taxonomy_ref:
errors.append(f"{self.finding_id}: missing taxonomy reference (OWASP/Microsoft)")
if not self.model_version:
errors.append(f"{self.finding_id}: missing pinned model version")
if self.result == "MEASURED" and not self.success_rate:
errors.append(f"{self.finding_id}: MEASURED result requires a success rate")
if self.severity in ("Critical", "High") and not self.remediation_module:
errors.append(f"{self.finding_id}: {self.severity} finding has no remediation route")
return errors
@dataclass
class ControlCell:
control: str # e.g., "taint gate (B2)"
surface: str # e.g., "retrieval store"
status: Literal["Present", "Absent", "Partial", "Mismeasured"]
note: str = ""
@dataclass
class EngagementReport:
client: str
engagement_id: str
scope_version: str
findings: list[Finding] = field(default_factory=list)
control_matrix: list[ControlCell] = field(default_factory=list)
def add_b9_checklist(self, checklist_rows: list[dict]) -> None:
"""Ingest the JSON output of B9's checklist executor. Each row becomes
a Finding with its taxonomy reference set to the ASI row."""
for row in checklist_rows:
severity = self._severity_for(row)
self.findings.append(Finding(
finding_id=row["id"],
title=row["risk_name"],
severity=severity,
taxonomy_ref=f"OWASP {row['id']}",
attack_procedure=row.get("attack_procedure", ""),
model_version=row.get("model_version", ""),
success_rate=row.get("measured_rate"),
sampling_params=row.get("sampling_params", {}),
scope_reference=row.get("scope_reference", ""),
result=row["result"],
residual_risk=row.get("residual_risk", ""),
remediation_module=row.get("defense_module", ""),
timestamp_utc=row.get("timestamp_utc", ""),
))
def executive_summary(self) -> dict:
"""Compute the overall residual posture from the measured rows.
Never returns 'secure' — returns the characterized residuals."""
by_severity: dict[str, int] = {}
measured_residuals: list[dict] = []
for f in self.findings:
by_severity[f.severity] = by_severity.get(f.severity, 0) + 1
if f.result == "MEASURED":
measured_residuals.append(
{"finding": f.finding_id, "residual": f.residual_risk}
)
ship_blockers = [f.finding_id for f in self.findings if f.severity == "Critical"]
recommendation = (
f"Do not ship; remediate Critical findings {ship_blockers} first."
if ship_blockers
else "Ship with characterized residuals; see measured rows."
)
return {
"findings_by_severity": by_severity,
"measured_residuals": measured_residuals,
"ship_recommendation": recommendation,
}
def validate_all(self) -> list[str]:
"""Every finding must pass validation before the report ships."""
errors: list[str] = []
for f in self.findings:
errors.extend(f.validate())
return errors
@staticmethod
def _severity_for(row: dict) -> Severity:
"""Map a B9 checklist row to a severity. FAIL on a high-impact row is
Critical/High; MEASURED with a high residual is High; PASS is Info."""
if row["result"] == "FAIL":
return "Critical" if row.get("high_impact") else "High"
if row["result"] == "MEASURED":
rate = row.get("measured_rate", "")
# naive parse: "60% over 100 attempts" -> 60
try:
pct = int(rate.split("%")[0])
except (ValueError, IndexError):
pct = 0
return "High" if pct >= 20 else "Medium"
return "Info"
The generator is deliberately strict. A finding that lacks a taxonomy reference does not ship. A MEASURED finding without a success rate does not ship. A Critical finding without a remediation route does not ship. The strictness is the point — a report generator that emits whatever it is given produces the "10/10 PASS" lie B9 refuses. The generator enforces the honesty at the output layer.
Retesting is where the residual-risk discipline proves a control moved the needle. Packaging is where the methodology becomes a repeatable practice.
This is the B0.2 principle made operational, and it is the single most common point of failure in AI assessment engagements. A client who has remediated a prompt-injection finding will ask: "is it fixed?" The answer is never "yes." The answer is "the injection success rate moved from 60% over 100 attempts to 4% over 100 attempts under the same harness, with the same sampling parameters, against the same pinned model version. The residual is characterized at 4%."
The retest protocol is a strict before/after comparison:
The retest is recurring revenue because residuals drift. Every model-version bump, every new tool added, every new MCP integration can move the residuals. A mature assessment practice sells a retest with every release, tracks the residual trend over time, and flags a regression before it reaches production. The trend line — injection success rate over the last six releases — is the single most valuable long-term artifact the service produces.
A one-off assessment is a project. A repeatable assessment practice is a service. The difference is packaging: the methodology, tooling, and templates that make the engagement run the same way across clients and releases. Four layers:
The packaging is the difference between an assessment that produces an artifact and an assessment practice that produces comparable artifacts. A CISO who buys the service across six releases wants to see the residual trend move; that is only possible if every engagement ran the same methodology, the same tooling, against the same report template. Drift in any layer breaks the comparability, and comparability is the long-term value.
This module does not introduce a new technique. It introduces the operational layer that makes B0–B11 a profession. The legal control plane (B0) becomes the SOW. The threat model (B1) becomes the reconnaissance phase. The defenses (B2–B8) become the controls the report assesses. The checklist (B9) and the chains (B10) become the discovery phase. The governance layer (B11) becomes the framework the report maps to and the audit trail it evidences. The residual-risk discipline (B0.2) becomes the retest. None of that is new; all of it is synthesized.
The student who has completed B0–B12 can, at the end, do four things: build a hardened agent harness (the Capstone B1 that follows), attack it with the full methodology, defend it with the full control set, and — the contribution of this module — deliver the assessment of it as a scoped, priced, reported, retestable service. That last capability is what turns the course's techniques into a practice a CISO will buy.
Half an engagement. The checklist is two of six phases (discovery + part of validation). Scoping is where the legal plane gets set; reporting is where the output becomes a deliverable; retesting is where the residual gets measured. Cure: run all six phases, with the checklist as the discovery-phase backbone.
"The model" is not a scope. A finding against gpt-model-v3 is meaningless against v4. Cure: pin every model version in the scope file; record the version in every finding; refuse to retest against a different version without reporting the version delta.
The deployer says "jailbreak everything"; the provider's ToS forbids it; the deployer cannot waive the provider's terms. Cure: the B0 provider-authorization check runs during scoping, not during testing. Surfaces that fail all three conditions are out of scope until the gap closes.
An anecdote, not a finding. Cure: B0's minimum-evidence discipline — success rate over N attempts, sampling params, model version. "60% over 100 attempts" is a finding; "1/1" is an anecdote.
Binary retests lie. Cure: residual-risk measurement — before/after success rate under the same harness, same sampling, same pinned version. The verdict is Resolved / Improved / Unchanged / Regressed, never "fixed."
An unclassified finding is not actionable for the client's governance layer. Cure: every finding maps to an OWASP ASI row (B9) and/or a Microsoft failure mode (B10). The report generator enforces this.
An underpriced chain-depth engagement ships an unvalidated chain. Cure: price the depth you can deliver with rigor; quote chain depth as a premium engagement for high-stakes agents.
An SOW that fails at the moment a serious finding appears. Cure: the seven-clause SOW template (from B12.1); the client's counsel signs it before testing begins.
| Term | Definition |
|---|---|
| Six-phase methodology | Scoping → Reconnaissance → Discovery → Exploitation/Validation → Reporting → Retesting; the PTES/NIST SP 800-115 pentest methodology adapted to the agentic surface |
| Scoping | Phase 1; enumerates surfaces (B1), model versions, provider authorizations (B0), and exclusions; produces the SOW and the scope file |
| Provider authorization | The B0 control: per provider-surface technique, one of ToS-perits / waiver-on-file / self-hosted must hold; verified at scoping, not at testing |
| Dual-use clause | The SOW clause that resolves B0.2's dilemma: report-to-provider-only, 180-day model-level embargo, recipe suppression by default |
| Checklist depth vs. chain depth | Two sellable depths: B9's ten rows (checklist) vs. B10's compound-intent chains (chain, premium) |
| Engagement report | The five-section deliverable: executive summary, findings, control matrix, remediation roadmap, methodology/scope/evidence appendix |
| Finding | A report entry with the field set: taxonomy reference, attack procedure, evidence (B0 minimum-evidence), residual risk, remediation route |
| Control matrix | The controls × surfaces grid marking Present / Absent / Partial / Mismeasured — the gap analysis a regulator reads |
| Residual risk | The measured rate after mitigation (e.g., "injection 4% over 100 attempts"); never binary "fixed" |
| Retest | Phase 6; same harness, same sampling, same pinned version; produces a before/after delta report |
| Service packaging | The four layers — methodology, tooling, templates, evidence — that make the assessment repeatable and comparable across releases |
See 07-lab-spec.md. Two deliverables: (1) the assessment report generator — a Python module that takes the JSON output of B9's checklist executor and produces the structured engagement report (executive summary, findings table with the required field set, control matrix, remediation roadmap), with strict validation that refuses to ship unclassified or incomplete findings; (2) a sample SOW for a realistic client incorporating all seven of B0/B12's clauses (systems in scope, provider authorization, techniques permitted/prohibited, dual-use and disclosure, DMCA waiver, data handling, residual-risk measurement protocol). Runnable, type-hinted, no GPU. The report generator is the capstone engineering artifact — it is where the B9 output becomes the B12 deliverable.
www.pentest-standard.org.genai.owasp.org/resource/owasp-top-10-for-agentic-applications-for-2026/; the B9 checklist that supplies the discovery-phase backbone and the taxonomy reference for every finding.# Module B12 — Harness Security Assessments as a Service
**Course**: 2B — Securing & Attacking Harnesses and LLMs
**Module**: B12 — Harness Security Assessments as a Service
**Duration**: 60 minutes
**Level**: Senior Engineer and above
**Prerequisites**: B0–B11 complete. This is the capstone methodology module — it synthesizes everything you built and attacked into a repeatable, scoped, deliverable engagement.
> *B0 built the legal control plane. B1 mapped the surface. B2–B8 built the defenses. B9 gave you the checklist. B10 gave you the chains. B11 gave you the governance layer. None of that is a service until you can scope it, price it, deliver it, and retest it. This module turns the techniques into a profession. The buyer is a CISO or an AI security lead, and they will measure you on the artifact you hand them, not on the techniques you ran.*
---
## Learning Objectives
After completing this module, you will be able to:
1. Run a six-phase agent security assessment — Scoping, Reconnaissance, Vulnerability Discovery, Exploitation/Validation, Reporting, Retesting — as a repeatable professional engagement, with each phase tied to the module that supplies its content (B0 authorization, B1 surfaces, B9 checklist, B10 chains, B0 minimum-proof, B11 governance).
2. Scope and price an agent assessment: enumerate the surfaces, pin the model versions, verify provider authorizations, and weight the cost drivers (number of agents, surface complexity, depth of testing) into a defensible fee.
3. Write an AI-specific Statement of Work that incorporates B0's dual-use clause and provider-authorization clause, the techniques permitted/prohibited, the data-handling rules, and the residual-risk measurement protocol.
4. Produce the engagement report — the scored artifact B9's checklist executor produces — with an executive summary, a findings table (each tagged to the OWASP/Microsoft taxonomy reference, the attack procedure, the evidence, the residual risk), a control matrix, and a remediation roadmap.
5. Retest a remediated agent using residual-risk measurement rather than binary "fixed/unfixed," and produce a before/after retest report that a regulator or auditor can read.
6. Package the service — methodology, tooling, templates — into a repeatable assessment practice that produces comparable results across engagements and releases.
---
## Why this module exists
Every prior module gave you a technique, a control, or a framework. None of them gave you an *engagement*. A CISO who hires you does not want a technique — they want a defensible answer to "is my agent safe to ship, and if not, what is the residual risk and what do I fix first?" That answer is an *artifact*, produced by a *method*, bounded by a *scope*, priced as a *service*. This module is the operational layer that turns B0–B11 into that artifact.
The temptation at the capstone is to treat the service as "run B9's checklist, hand over the output." That is half an engagement. A real agent security assessment has six phases, and only two of them are the testing that B9 and B10 supply. Scoping is where the legal control plane (B0) gets written into the contract; reporting is where the scored output (B9) becomes a deliverable a board can read; retesting is where the residual-risk discipline (B0.2) proves a control actually moved the needle. Skip any phase and the engagement is either illegal (no scope), unsellable (no report), or unverifiable (no retest).
The buyer matters. From the course spec: this module's buyer is the **CISO or AI security lead**. That buyer reads frameworks, not exploits. They will ask, before anything else, "which control framework does this map to?" (B9 OWASP, B10 Microsoft, B11 NIST AI RMF), "what is the residual risk in numbers?" (B9's measured rows), and "can my auditor read this?" (B11's audit trail). The methodology in this module is calibrated to that buyer. A beautiful exploit chain that does not map to a control row will not move budget. A measured residual that maps to a NIST AI RMF Measure function will.
Three sub-sections, twenty minutes each:
- **B12.1 — The Six-Phase Assessment Methodology + Scoping & Pricing.** The backbone: Scoping → Reconnaissance → Discovery → Exploitation/Validation → Reporting → Retesting. Then the scoping deep-dive: surfaces, model versions, provider authorizations, and how those factor into price and the SOW (with B0's dual-use and provider-authorization clauses).
- **B12.2 — The Engagement Report.** The deliverable. Executive summary, findings (each with taxonomy reference, attack procedure, evidence, residual risk), control matrix, remediation roadmap. This is the same scored artifact B9's checklist executor produces, formatted for a CISO.
- **B12.3 — Retesting and Packaging the Service.** Retesting as residual-risk measurement (not binary fixed). Then packaging: the methodology, tooling, and templates that make an assessment practice repeatable across clients and releases.
---
# B12.1 — The Six-Phase Methodology + Scoping & Pricing
*The backbone of the engagement, and the front-end work that determines whether the testing is legal, scoped, and priced correctly.*
## The six phases
The methodology adapts the pentest standards — PTES (Penetration Testing Execution Standard) and NIST SP 800-115 ("Technical Guide to Information Security Testing and Assessment") — to the agentic surface. The phases are unchanged in *name*; what changes is the *content* of each, because the target is an agent, not a network. Every phase ties to a prior module.
**Phase 1 — Scoping.** What systems, what surfaces, what model versions, and — the AI-specific step — what provider authorizations. This is where B0's legal control plane gets written into the engagement contract. The output is the SOW and the JSON scope file (from B0's lab) that separates deployer-controlled from provider-controlled surfaces and carries the `provider_authorization` field per technique. A scoping conversation that does not produce a scope file with that separation is incomplete.
**Phase 2 — Reconnaissance.** Map the agent's surfaces. B1's threat model is the scaffold: the loop, the tools, the memory, the provider, the identity, the sandbox, the inter-agent edges. The deliverable is a surface map — every input boundary, every tool, every persisted state, every external call, every sub-agent. This is not "port scan the agent"; it is "enumerate the attack surface as B1 defines it." Reconnaissance is also where B10's capability-disclosure mode (Mode 7) runs first: the agent's own description of what it can do is the cheapest surface map you will ever get.
**Phase 3 — Vulnerability discovery.** Run B9's OWASP checklist (ASI01–ASI10, each with its attack procedure and test) and construct B10's attack chains (the seven failure modes, plus the zero-click HITL bypass chain). Discovery is two-track: the *checklist track* (B9) tests each control deterministically or measures it probabilistically; the *chain track* (B10) constructs multi-step compound-intent attacks that slip *between* controls. Neither alone is sufficient — B9 finds missing controls; B10 finds gaps between controls that exist.
**Phase 4 — Exploitation / validation.** Confirm findings with B0's minimum-proof discipline. A candidate finding is not a finding until it has been reproduced with the minimum-evidence field set (model version, exact prompt, sampling params, success rate over N attempts, scope reference). Validation is where most engagements lose rigor: a single successful jailbreak reported as a finding is an anecdote; the same jailbreak at 60% over 100 attempts is a finding. This is also where the dual-use discipline bites — a validated jailbreak is now a dual-use artifact and the RoE's disclosure clauses take over.
**Phase 5 — Reporting.** The deliverable. Covered in B12.2. The output of B9's checklist executor (the scored table with 8 PASS/FAIL + 2 MEASURED rows) plus B10's chain findings, formatted into the engagement report structure.
**Phase 6 — Retesting.** After remediation, re-run the discovery and validation phases and measure residual risk before/after. Covered in B12.3. The retest is the proof the controls moved the needle — and the recurring-revenue layer of the service, because every release re-runs it.
The six phases are a loop, not a pipeline. A retest feeds back into scoping for the next engagement. A finding that escapes the checklist routes a new row into B9 for the next release. The methodology is a cycle the client buys into across releases, not a one-shot audit.
## Scoping the assessment
Scoping an agent assessment is harder than scoping a network pentest because the surface is less legible. A network has IP addresses and ports; an agent has tools, memory, prompts, model versions, and inter-agent edges, most of which are not documented anywhere the client can point you to. The scoping conversation must enumerate four things:
1. **The surfaces in scope.** Use B1's threat model as the enumeration template: every input boundary (user, retrieved context, tool output, sub-agent message), every tool (with its arguments and validators), every persisted state (memory, retrieval store, session), every external call (model provider, MCP servers, downstream systems), every identity the agent acts as, every sandbox boundary, every inter-agent edge. The client will under-report; B1's template is how you make sure nothing is silent.
2. **The model versions in scope.** Pin them. A finding against `gpt-model-v3` is meaningless if the client ships `v4` next week. The scope file (from B0) carries `model_versions_in_scope` with pinned checkpoints, and every finding records the version it was tested against. A scope with "the model" and no version is a scope that cannot be retested.
3. **The provider authorizations.** B0's load-bearing point: the deployer cannot authorize what the provider forbids. For each provider-controlled surface (weight reads, jailbreak attempts, system-prompt extraction), verify one of B0's three conditions — provider ToS permits it, a provider waiver/preview enrollment is on file, or the model is self-hosted and the deployer owns it. Surfaces that fail all three are *out of scope* until the authorization gap is closed. This is the most common scoping mistake: a client who says "yes, jailbreak everything" without checking their provider's terms.
4. **The exclusions.** What is *not* in scope. Production data with real PII, shared infrastructure the test could degrade (B0's third-party-harm risk), techniques the provider ToS forbids, surfaces the client is decommissioning. Exclusions are written into the SOW so scope drift has a documented boundary.
## Pricing the assessment
Pricing an agent assessment is driven by three factors, weighted into an effort estimate:
- **Number of agents (and sub-agents).** Each agent is a separate surface with its own loop, tools, and memory. A single agent is one unit; a multi-agent system (orchestrator + N sub-agents) multiplies the surface roughly linearly with a chain-complexity premium, because B10's inter-agent trust attacks only exist in multi-agent systems.
- **Surface complexity.** A retrieval-augmented agent with a code-execution tool, a memory store, and three MCP integrations is an order of magnitude more surface than a stateless chatbot. Complexity is measured by counting B1's surface elements: number of tools, presence of persistent memory, presence of retrieval, presence of code execution, number of external integrations, number of inter-agent edges.
- **Depth of testing.** Two depths are sellable: *checklist depth* (B9's ten rows run deterministically/measured against each agent) and *chain depth* (B10's compound-intent chains constructed against the highest-value surfaces). Checklist depth is the baseline engagement; chain depth is the premium engagement for high-stakes agents (financial, healthcare, autonomous). A client who buys only checklist depth gets a report that says "no missing controls"; a client who buys chain depth gets a report that says "and here is the compound chain that slips between them."
The honest pricing rule: do not sell a depth you cannot deliver with rigor. A "chain-depth" engagement priced at checklist-depth hours is an engagement that will ship a B10 chain that has not been validated end-to-end. Underpriced assessments produce under-rigored reports, and under-rigored reports are how an agent ships with a real vulnerability the assessment said was fixed.
## The Statement of Work
The SOW is where B0's clauses get written into the engagement contract. A traditional pentest SOW covers scope, duration, and deliverables. An AI red-team SOW must *additionally* carry the clauses B0.1 specifies — these are not optional, and an SOW without them is an SOW that fails at the moment a serious finding appears:
- **Systems in scope** (the four-point enumeration above, pinned versions included).
- **Provider authorization / ToS compliance** — an explicit statement of how the engagement complies with each model provider's terms, OR a copy of the provider-issued waiver/preview enrollment. This is the clause that closes B0's most common legal gap.
- **Techniques permitted / prohibited** — prompt injection (direct, indirect, multi-step), memory poisoning, tool/MCP abuse, sandbox escape attempts, weight-extraction probes, jailbreak attempts. Each either permitted, permitted-with-minimum-proof, or prohibited. Weight extraction is never "permitted"; it is at most "permitted-with-minimum-proof" (path + hash + byte count, never the file).
- **Dual-use and disclosure clause** — how jailbreak and misuse-enabling findings will be handled: report-to-provider-only, coordinated disclosure timeline (180 days for model-level, 90 for harness-level), publication embargo, pre-publication review. This clause is the resolution of B0.2's dual-use dilemma, made contractual.
- **DMCA / anti-circumvention waiver** — if the test requires bypassing a technical access control on a model, an explicit waiver of the provider's § 1201 rights for the engagement scope and duration.
- **Data handling** — what the red-team may capture, how it is stored, the retention classes (B0's Public / Provider-Only / Restricted / Destroy-on-Report), and the destruction timeline.
- **Residual-risk measurement protocol** — the engagement reports measured residuals (injection success rate before/after), not binary "fixed/unfixed." This clause prevents the most common retest failure: a client who demands "show me it's fixed" and a tester who obliges with a single re-run.
The lab for this module has you write a sample SOW with all seven clauses for a realistic client. That SOW is the artifact the client's counsel signs; it is also the artifact the assessment practice reuses across engagements, parameterized by client.
---
# B12.2 — The Engagement Report
*The deliverable. This is the scored artifact B9's checklist executor produces, formatted into a structure a CISO can read and a regulator can audit.*
## The report structure
The engagement report has five sections. Every framework the buyer reads (NIST AI RMF, EU AI Act conformity, ISO 42001) expects roughly this shape; the agent-specific content is what fills it.
### 1. Executive summary
One page. The overall residual-risk posture in plain language, the count of findings by severity, the top three risks in priority order, and a single-sentence ship recommendation calibrated to the buyer's risk appetite. The executive summary is the only section most of the buyer's organization will read; it must stand alone. The ship recommendation is never "secure" — it is "ship with characterized residuals at X% and Y%, conditional on remediating the two Critical findings" or "do not ship; the compound chain in Finding F-04 reaches lateral movement." B9's discipline carries: the summary reports the measured residuals, not a pass/fail.
### 2. Findings
The body of the report. One entry per finding, each with the same field set so findings are comparable and the table is machine-readable. The fields:
| Field | Content |
| --- | --- |
| **Finding ID** | F-01, F-02, ... (stable across retests) |
| **Title** | One-line description |
| **Severity** | Critical / High / Medium / Low (CVSS-like, calibrated to agent impact) |
| **Taxonomy reference** | OWASP ASI row (B9) and/or Microsoft failure mode (B10) — e.g., "ASI01 Goal Hijacking; Microsoft Mode 2 (goal hijack drift)" |
| **Attack procedure** | The concrete steps: the injection vector, the tool reached, the impact. For B10 chains, the multi-step sequence with each step's approval gate. |
| **Evidence** | B0's minimum-evidence field set: model version, exact prompt (or a redacted reference if dual-use), sampling params, success rate over N attempts, timestamp UTC, scope reference. |
| **Residual risk** | The measured rate after any in-engagement mitigation (e.g., "injection success 60% over 100 attempts; residual 4% after L4 taint gate enabled mid-engagement"). For PASS/FAIL rows, the binary result. |
| **Remediation** | The control to add or fix, routed to the module that builds it (B2–B8), with an effort estimate. |
Every finding maps to a taxonomy reference. A finding with no OWASP row and no Microsoft mode is a finding that has not been classified — and classification is what makes it actionable for the client's governance layer (B11). The taxonomy reference is the bridge from "we found a bug" to "here is the control framework row it satisfies."
### 3. Control matrix
A two-dimensional view: the controls that *exist* (rows) against the surfaces they cover (columns), with each cell marked Present / Absent / Partial / Mismeasured. This is the gap analysis. A control that is "Present" on the system-prompt surface but "Absent" on the retrieval-store surface is a finding (ASI04 not defended on retrieval). A control that is "Mismeasured" — present but never tested — is a B11 audit-trail gap. The control matrix is what a regulator reads to ask "did you actually verify these controls, or just document them?"
### 4. Remediation roadmap
The findings, prioritized by severity and dependency, sequenced into a release plan. Critical findings first; findings that unblock other findings next (e.g., the principal-binding fix that makes the tool-contract fixes testable). Each remediation item routes to the module that builds the control (B2–B8), with an owner and an estimated effort. The roadmap is the artifact the client's engineering team works from between the report and the retest.
### 5. Appendix: methodology, scope, evidence index
The reproducibility layer. The methodology (the six phases), the scope (the SOW and scope file), the model versions tested, the tool versions used, and an index to the evidence store (with the B0 data-class classification redacted as appropriate). This appendix is what makes the report auditable: a third party can reconstruct what was tested, against what, with what tools, and verify the findings reproduce.
## The report is the scored artifact B9 produces
This is the structural connection that makes B12 a synthesis module rather than a new surface. B9's checklist executor, when run to scope-completion, produces a table with one row per ASI risk — eight PASS/FAIL and two MEASURED. That table *is* the findings section's backbone. B12 does not re-test; it *packages*. The B9 rows become findings (F-01 through F-10, roughly); the B10 chains become the additional findings (F-11+) that the checklist alone would miss; the control matrix is the B9 risk-to-module mapping (ASI → B2/B3/B4/B5/B7/B8) read as a coverage view; the remediation roadmap is the B9/B10 findings routed to their building modules.
The buyer sees one report. The methodology underneath is B9 + B10, governed by B0, evidenced by B11's audit trail. That is the capstone: every prior module contributes a layer, and the report is where they stack into a single deliverable.
## The report generator
The lab builds a report generator that takes the JSON output of B9's checklist executor and produces the structured engagement report. The generator is opinionated: it enforces the field set per finding, it refuses to emit a finding without a taxonomy reference, and it computes the executive-summary residual posture from the measured rows. The generator is what makes the practice repeatable — the same input produces the same report structure across engagements, so a client can compare release N to release N+1 and see whether the residuals moved.
```python
from __future__ import annotations
from dataclasses import dataclass, field
from typing import Literal, Optional
Severity = Literal["Critical", "High", "Medium", "Low", "Info"]
ResultType = Literal["PASS", "FAIL", "MEASURED", "N/A"]
@dataclass
class Finding:
"""A single assessment finding. The field set enforces B0's minimum-evidence
discipline and the taxonomy-reference requirement (no orphan findings)."""
finding_id: str
title: str
severity: Severity
taxonomy_ref: str # required: OWASP ASI row and/or Microsoft mode
attack_procedure: str
model_version: str # pinned checkpoint — a finding without it is untestable
success_rate: Optional[str] # "60% over 100 attempts" for MEASURED; None for PASS/FAIL
sampling_params: dict # temperature, top_p, etc. — reproducibility
scope_reference: str # the SOW/scope-file clause that authorized this
result: ResultType
residual_risk: str # measured residual after any in-engagement mitigation
remediation_module: str # which of B2-B8 builds the fix
timestamp_utc: str
def validate(self) -> list[str]:
"""Return a list of validation errors. A finding that fails validation
is a finding that should not ship in the report."""
errors: list[str] = []
if not self.taxonomy_ref:
errors.append(f"{self.finding_id}: missing taxonomy reference (OWASP/Microsoft)")
if not self.model_version:
errors.append(f"{self.finding_id}: missing pinned model version")
if self.result == "MEASURED" and not self.success_rate:
errors.append(f"{self.finding_id}: MEASURED result requires a success rate")
if self.severity in ("Critical", "High") and not self.remediation_module:
errors.append(f"{self.finding_id}: {self.severity} finding has no remediation route")
return errors
@dataclass
class ControlCell:
control: str # e.g., "taint gate (B2)"
surface: str # e.g., "retrieval store"
status: Literal["Present", "Absent", "Partial", "Mismeasured"]
note: str = ""
@dataclass
class EngagementReport:
client: str
engagement_id: str
scope_version: str
findings: list[Finding] = field(default_factory=list)
control_matrix: list[ControlCell] = field(default_factory=list)
def add_b9_checklist(self, checklist_rows: list[dict]) -> None:
"""Ingest the JSON output of B9's checklist executor. Each row becomes
a Finding with its taxonomy reference set to the ASI row."""
for row in checklist_rows:
severity = self._severity_for(row)
self.findings.append(Finding(
finding_id=row["id"],
title=row["risk_name"],
severity=severity,
taxonomy_ref=f"OWASP {row['id']}",
attack_procedure=row.get("attack_procedure", ""),
model_version=row.get("model_version", ""),
success_rate=row.get("measured_rate"),
sampling_params=row.get("sampling_params", {}),
scope_reference=row.get("scope_reference", ""),
result=row["result"],
residual_risk=row.get("residual_risk", ""),
remediation_module=row.get("defense_module", ""),
timestamp_utc=row.get("timestamp_utc", ""),
))
def executive_summary(self) -> dict:
"""Compute the overall residual posture from the measured rows.
Never returns 'secure' — returns the characterized residuals."""
by_severity: dict[str, int] = {}
measured_residuals: list[dict] = []
for f in self.findings:
by_severity[f.severity] = by_severity.get(f.severity, 0) + 1
if f.result == "MEASURED":
measured_residuals.append(
{"finding": f.finding_id, "residual": f.residual_risk}
)
ship_blockers = [f.finding_id for f in self.findings if f.severity == "Critical"]
recommendation = (
f"Do not ship; remediate Critical findings {ship_blockers} first."
if ship_blockers
else "Ship with characterized residuals; see measured rows."
)
return {
"findings_by_severity": by_severity,
"measured_residuals": measured_residuals,
"ship_recommendation": recommendation,
}
def validate_all(self) -> list[str]:
"""Every finding must pass validation before the report ships."""
errors: list[str] = []
for f in self.findings:
errors.extend(f.validate())
return errors
@staticmethod
def _severity_for(row: dict) -> Severity:
"""Map a B9 checklist row to a severity. FAIL on a high-impact row is
Critical/High; MEASURED with a high residual is High; PASS is Info."""
if row["result"] == "FAIL":
return "Critical" if row.get("high_impact") else "High"
if row["result"] == "MEASURED":
rate = row.get("measured_rate", "")
# naive parse: "60% over 100 attempts" -> 60
try:
pct = int(rate.split("%")[0])
except (ValueError, IndexError):
pct = 0
return "High" if pct >= 20 else "Medium"
return "Info"
```
The generator is deliberately strict. A finding that lacks a taxonomy reference does not ship. A MEASURED finding without a success rate does not ship. A Critical finding without a remediation route does not ship. The strictness is the point — a report generator that emits whatever it is given produces the "10/10 PASS" lie B9 refuses. The generator enforces the honesty at the output layer.
---
# B12.3 — Retesting and Packaging the Service
*Retesting is where the residual-risk discipline proves a control moved the needle. Packaging is where the methodology becomes a repeatable practice.*
## Retesting: residual risk, not binary "fixed"
This is the B0.2 principle made operational, and it is the single most common point of failure in AI assessment engagements. A client who has remediated a prompt-injection finding will ask: "is it fixed?" The answer is never "yes." The answer is "the injection success rate moved from 60% over 100 attempts to 4% over 100 attempts under the same harness, with the same sampling parameters, against the same pinned model version. The residual is characterized at 4%."
The retest protocol is a strict before/after comparison:
1. **Pin the test harness.** The retest uses the *same* B9/B10 test cases, the *same* InjecAgent-style measurement protocol, the *same* sampling parameters. A retest that changes the test set is not a retest — it is a new assessment.
2. **Pin the model version.** If the client has bumped the model version between the original test and the retest, the before/after is invalid — the residuals moved because of the version bump, not the remediation. The retest either runs against the original pinned version (to isolate the remediation effect) or reports both versions separately (to show the combined effect of remediation + version).
3. **Re-run discovery.** A remediation that closes the original finding may open a new one. B10's chains are especially prone to this: patching one step of a chain may reroute the chain through a different gap. The retest re-runs the B9 checklist (did any PASS regress to FAIL?) and re-attempts the B10 chains (did the compound still complete through a different path?).
4. **Measure the residual.** Report the before/after success rate for every MEASURED finding, and the PASS/FAIL status for every deterministic finding. A deterministic finding that regressed from PASS to FAIL is a regression bug in the remediation.
5. **Produce the retest report.** A delta document against the original report: each finding with its original residual, its post-remediation residual, and a verdict (Resolved / Improved / Unchanged / Regressed). The retest report is the artifact a regulator reads to verify the client actually fixed what they said they fixed.
The retest is recurring revenue because residuals drift. Every model-version bump, every new tool added, every new MCP integration can move the residuals. A mature assessment practice sells a retest with every release, tracks the residual trend over time, and flags a regression before it reaches production. The trend line — injection success rate over the last six releases — is the single most valuable long-term artifact the service produces.
## Packaging the service
A one-off assessment is a project. A repeatable assessment practice is a service. The difference is packaging: the methodology, tooling, and templates that make the engagement run the same way across clients and releases. Four layers:
- **Methodology layer.** The six-phase process, documented, with the phase-to-module mapping (Phase 1 → B0 scope, Phase 2 → B1 recon, Phase 3 → B9 + B10 discovery, Phase 4 → B0 minimum-proof, Phase 5 → report, Phase 6 → retest). This is the playbook. A new assessor on the team reads it and runs an engagement the same way the senior assessor does.
- **Tooling layer.** B9's checklist executor (the scored table generator), B10's chain-construction harness, the report generator (from this module's lab), the scope-file validator (from B0's lab), the evidence classifier (from B0's lab). The tools are the same across engagements; the inputs (scope, target agent) change. Tooling is what makes the practice *scalable* — a senior assessor can run three concurrent engagements because the tooling handles the deterministic work.
- **Template layer.** The SOW template (with B0's seven clauses), the scope-file template (with the deployer/provider surface separation), the report template (with the five sections), the retest-report template (the delta format), the CVD timeline template (from B0's lab). Templates are what make the practice *consistent* — every engagement produces the same artifact shape, so a client can compare releases.
- **Evidence layer.** B0's evidence store with the four retention classes, the audit trail from B11, the AI BOM from B11. The evidence layer is what makes the practice *defensible* — every finding traces to scoped, authorized, minimum-proof evidence that survives a regulator's review.
The packaging is the difference between an assessment that produces an artifact and an assessment practice that produces *comparable* artifacts. A CISO who buys the service across six releases wants to see the residual trend move; that is only possible if every engagement ran the same methodology, the same tooling, against the same report template. Drift in any layer breaks the comparability, and comparability is the long-term value.
## The capstone synthesis
This module does not introduce a new technique. It introduces the operational layer that makes B0–B11 a profession. The legal control plane (B0) becomes the SOW. The threat model (B1) becomes the reconnaissance phase. The defenses (B2–B8) become the controls the report assesses. The checklist (B9) and the chains (B10) become the discovery phase. The governance layer (B11) becomes the framework the report maps to and the audit trail it evidences. The residual-risk discipline (B0.2) becomes the retest. None of that is new; all of it is synthesized.
The student who has completed B0–B12 can, at the end, do four things: build a hardened agent harness (the Capstone B1 that follows), attack it with the full methodology, defend it with the full control set, and — the contribution of this module — *deliver the assessment of it as a scoped, priced, reported, retestable service*. That last capability is what turns the course's techniques into a practice a CISO will buy.
---
## Anti-Patterns
### "Run B9's checklist and hand over the output"
Half an engagement. The checklist is two of six phases (discovery + part of validation). Scoping is where the legal plane gets set; reporting is where the output becomes a deliverable; retesting is where the residual gets measured. Cure: run all six phases, with the checklist as the discovery-phase backbone.
### Scoping without pinning the model version
"The model" is not a scope. A finding against `gpt-model-v3` is meaningless against `v4`. Cure: pin every model version in the scope file; record the version in every finding; refuse to retest against a different version without reporting the version delta.
### Scoping without verifying provider authorization
The deployer says "jailbreak everything"; the provider's ToS forbids it; the deployer cannot waive the provider's terms. Cure: the B0 provider-authorization check runs during scoping, not during testing. Surfaces that fail all three conditions are out of scope until the gap closes.
### Reporting a single successful jailbreak as a finding
An anecdote, not a finding. Cure: B0's minimum-evidence discipline — success rate over N attempts, sampling params, model version. "60% over 100 attempts" is a finding; "1/1" is an anecdote.
### Accepting "is it fixed?" as the retest question
Binary retests lie. Cure: residual-risk measurement — before/after success rate under the same harness, same sampling, same pinned version. The verdict is Resolved / Improved / Unchanged / Regressed, never "fixed."
### A report with a finding that has no taxonomy reference
An unclassified finding is not actionable for the client's governance layer. Cure: every finding maps to an OWASP ASI row (B9) and/or a Microsoft failure mode (B10). The report generator enforces this.
### Selling chain depth at checklist-depth hours
An underpriced chain-depth engagement ships an unvalidated chain. Cure: price the depth you can deliver with rigor; quote chain depth as a premium engagement for high-stakes agents.
### An SOW without the dual-use and provider-authorization clauses
An SOW that fails at the moment a serious finding appears. Cure: the seven-clause SOW template (from B12.1); the client's counsel signs it before testing begins.
---
## Key Terms
| Term | Definition |
| --- | --- |
| **Six-phase methodology** | Scoping → Reconnaissance → Discovery → Exploitation/Validation → Reporting → Retesting; the PTES/NIST SP 800-115 pentest methodology adapted to the agentic surface |
| **Scoping** | Phase 1; enumerates surfaces (B1), model versions, provider authorizations (B0), and exclusions; produces the SOW and the scope file |
| **Provider authorization** | The B0 control: per provider-surface technique, one of ToS-perits / waiver-on-file / self-hosted must hold; verified at scoping, not at testing |
| **Dual-use clause** | The SOW clause that resolves B0.2's dilemma: report-to-provider-only, 180-day model-level embargo, recipe suppression by default |
| **Checklist depth vs. chain depth** | Two sellable depths: B9's ten rows (checklist) vs. B10's compound-intent chains (chain, premium) |
| **Engagement report** | The five-section deliverable: executive summary, findings, control matrix, remediation roadmap, methodology/scope/evidence appendix |
| **Finding** | A report entry with the field set: taxonomy reference, attack procedure, evidence (B0 minimum-evidence), residual risk, remediation route |
| **Control matrix** | The controls × surfaces grid marking Present / Absent / Partial / Mismeasured — the gap analysis a regulator reads |
| **Residual risk** | The measured rate after mitigation (e.g., "injection 4% over 100 attempts"); never binary "fixed" |
| **Retest** | Phase 6; same harness, same sampling, same pinned version; produces a before/after delta report |
| **Service packaging** | The four layers — methodology, tooling, templates, evidence — that make the assessment repeatable and comparable across releases |
---
## Lab Exercise
See `07-lab-spec.md`. Two deliverables: (1) the assessment report generator — a Python module that takes the JSON output of B9's checklist executor and produces the structured engagement report (executive summary, findings table with the required field set, control matrix, remediation roadmap), with strict validation that refuses to ship unclassified or incomplete findings; (2) a sample SOW for a realistic client incorporating all seven of B0/B12's clauses (systems in scope, provider authorization, techniques permitted/prohibited, dual-use and disclosure, DMCA waiver, data handling, residual-risk measurement protocol). Runnable, type-hinted, no GPU. The report generator is the capstone engineering artifact — it is where the B9 output becomes the B12 deliverable.
---
## References
1. **Advanced Course S13** — source for the six-phase assessment methodology (lines 707–734): Scoping, Reconnaissance, Vulnerability Discovery, Exploitation/Validation, Reporting, Retesting. The backbone this module adapts to the agentic surface.
2. **PTES (Penetration Testing Execution Standard)** — the traditional pentest methodology whose phase structure the agent assessment inherits; `www.pentest-standard.org`.
3. **NIST SP 800-115** — "Technical Guide to Information Security Testing and Assessment," the US federal baseline for security testing methodology; the government-buyer-readable methodology reference.
4. **OWASP Top 10 for Agentic Applications (2026)** — `genai.owasp.org/resource/owasp-top-10-for-agentic-applications-for-2026/`; the B9 checklist that supplies the discovery-phase backbone and the taxonomy reference for every finding.
5. **Microsoft AI Red Team — Failure Mode Taxonomy v2.0** — the B10 red-team framework that supplies the chain track and the seven failure modes; the offense complement to OWASP's defense.
6. **Course 1, Module 11** — the canonical OWASP Agentic Top 10 numbering students encounter first; reconciled against the OWASP primary source.
7. **Module B0** — the legal control plane: the authorization chain, the provider-authorization field, the dual-use disclosure principles, the minimum-evidence discipline, the seven-clause SOW. The scope phase and the retest protocol both run on B0.
8. **Module B1** — the threat model of agentic systems; the reconnaissance phase's enumeration template (loop, tools, memory, provider, identity, sandbox, inter-agent edges).
9. **Module B9** — the OWASP checklist executor whose scored output (8 PASS/FAIL + 2 MEASURED) is the findings table's backbone; the report generator consumes its JSON.
10. **Module B10** — the Microsoft taxonomy attack chains that supply the additional findings the checklist alone misses; the chain-depth engagement.
11. **Module B11** — the governance layer (NIST AI RMF, AI BOM, audit trail, policy-as-code) the report maps to and evidences; the framework the CISO buyer reads.
12. **NIST AI RMF (AI 100-1)** — the Govern/Map/Measure/Manage framework the report's structure implicitly satisfies; the de facto US governance standard.
13. **ISO/IEC 42001** — the certifiable AI management system standard; the international-buyer-readable framework reference.
14. **EU AI Act (Regulation 2024/1689)** — Art. 12 (logging/audit), Art. 14 (human oversight), Annex IV (technical documentation); the conformity obligations a high-risk-system report must evidence.