{ "module": "B12 — Harness Security Assessments as a Service", "course": "2B — Securing & Attacking Harnesses and LLMs", "version": "1.0.0", "duration_minutes": 35, "total_questions": 15, "bloom_distribution": { "target": "20% recall / 40% application / 40% analysis-design", "actual": { "recall": 3, "application": 6, "analysis": 6 } }, "passing_score_percent": 70, "questions": [ { "id": "Q01", "bloom": "recall", "type": "multiple_choice", "prompt": "What are the six phases of the agent security assessment methodology (B12), in order?", "options": [ "Plan, Build, Test, Deploy, Monitor, Retire.", "Scoping → Reconnaissance → Vulnerability Discovery → Exploitation/Validation → Reporting → Retesting.", "Threat model, Red team, Blue team, Purple team, Report, Sign-off.", "Recon, Exploit, Report — the other phases are optional overhead." ], "answer_index": 1, "rationale": "The six-phase methodology adapts PTES and NIST SP 800-115 to the agentic surface. Each phase ties to a prior module: Scoping→B0, Recon→B1, Discovery→B9+B10, Validation→B0 minimum-proof, Reporting→B9 output packaged, Retesting→B0.2 residual-risk. Only two phases (Discovery + Validation) are the testing B9/B10 supply; the other four are what make it a service." }, { "id": "Q02", "bloom": "recall", "type": "multiple_choice", "prompt": "Which prior module supplies the backbone of the discovery phase's checklist track, and what does its output become in the engagement report?", "options": [ "B1 (threat model); its surface map becomes the control matrix.", "B9 (OWASP checklist); its scored output (8 PASS/FAIL + 2 MEASURED) becomes the findings section's backbone (F-01..F-10).", "B10 (Microsoft taxonomy); its chains become the executive summary.", "B11 (governance); its AI BOM becomes the appendix." ], "answer_index": 1, "rationale": "B12 does not re-test; it packages. B9's checklist executor output IS the findings section's backbone. The B9 rows become findings F-01 through F-10; the B10 chains become the additional findings (F-11+) the checklist alone misses; the B9 risk-to-module mapping (ASI → B2/B3/B4/B5/B7/B8) becomes the control matrix. B1's surface map feeds reconnaissance; B11's AI BOM feeds the appendix." }, { "id": "Q03", "bloom": "recall", "type": "multiple_choice", "prompt": "From the course spec, who is the buyer of a B12 assessment, and what do they read first?", "options": [ "The lead engineer; they read the exploit chains first.", "The CISO or AI security lead; they read frameworks first (which control framework does this map to? what is the residual risk in numbers? can my auditor read this?).", "The model provider; they read the jailbreak recipes first.", "The marketing team; they read the executive summary for quotes." ], "answer_index": 1, "rationale": "The buyer reads frameworks, not exploits. Before anything else they ask: which control framework does this map to (B9 OWASP / B10 Microsoft / B11 NIST AI RMF); what is the residual risk in numbers (B9's measured rows); can my auditor read this (B11's audit trail). A beautiful exploit chain that does not map to a control row will not move budget. The methodology is calibrated to that buyer." }, { "id": "Q04", "bloom": "application", "type": "multiple_choice", "prompt": "A client says 'jailbreak everything' during scoping. Their agent calls a frontier model via API, and the provider's AUP prohibits jailbreaking. No waiver is on file. What is the correct scoping decision?", "options": [ "Proceed — the deployer authorized it, so the provider's terms don't matter.", "The jailbreak technique is OUT OF SCOPE for that provider surface until one of B0's three conditions holds (provider ToS permits / waiver on file / self-hosted). The deployer cannot authorize what the provider forbids. Route the client to the provider's red-team/preview program or get a waiver.", "Proceed but only against the deployer-controlled surfaces, and silently skip the jailbreak.", "Proceed at low volume to avoid detection." ], "answer_index": 1, "rationale": "This is B0's most common legal mistake applied at B12's scoping phase. The provider_authorization check (B0) runs during scoping, not during testing. A surface that fails all three conditions is out of scope until the gap closes. The fix is to enroll the client in the provider's official red-team program or obtain a separate waiver. The four-point enumeration (B12) makes this a gating step." }, { "id": "Q05", "bloom": "application", "type": "multiple_choice", "prompt": "During validation, you achieve one successful jailbreak in one attempt against a frontier model. What do you report?", "options": [ "A Critical finding — any success proves the model is broken.", "Nothing yet — a single success is an anecdote. Apply B0's minimum-evidence discipline: re-run over N attempts (e.g. 100), report the success rate (e.g. '60% over 100 attempts') plus sampling params and pinned model version. '1/1' is an anecdote; a rate over N is a finding.", "Report it as a PASS since it only worked once.", "Retry until you get 100% success, then report." ], "answer_index": 1, "rationale": "Jailbreak success is stochastic and sampling-dependent. The standard is a measured success rate over a statistically meaningful N (InjecAgent-style). A single success is weak evidence. The finding must carry model version, exact prompt, sampling params, success rate over N, and scope reference (B0 minimum-evidence). The report generator rejects a MEASURED finding without a success rate." }, { "id": "Q06", "bloom": "application", "type": "multiple_choice", "prompt": "A client has remediated a prompt-injection finding and asks 'is it fixed?' What is the correct retest answer?", "options": [ "Yes — the control is now in place.", "No — AI findings can never be fixed.", "The injection success rate moved from 60% to 4% over 100 attempts under the same harness, same sampling params, same pinned model version. The residual is characterized at 4%. The verdict is Improved, not 'fixed.'", "We will confirm it is fixed after one more successful test." ], "answer_index": 2, "rationale": "Retesting measures residual risk, not binary fixed/unfixed (B0.2). The verdict is Resolved / Improved / Unchanged / Regressed based on a strict before/after comparison under identical conditions. 'Is it fixed?' is the wrong question; 'did the residual move, and by how much, under identical conditions?' is the right one. The residual is the artifact a regulator reads to verify the client fixed what they said they fixed." }, { "id": "Q07", "bloom": "application", "type": "multiple_choice", "prompt": "You are pricing an assessment for a multi-agent system (orchestrator + 4 sub-agents) with RAG, a code-exec tool, and three MCP integrations. The client wants chain-depth testing on the orchestrator. Which factor is the chain-complexity premium, and why?", "options": [ "There is no premium — price by agent count only.", "The chain-complexity premium applies because B10's inter-agent trust attacks (Mode 3) only exist in multi-agent systems. The orchestrator's sub-agent edges are surfaces a single-agent engagement does not have. Price scales roughly linearly with agent count plus a premium for the inter-agent chain surface.", "The premium is for the number of MCP integrations only.", "The premium is a flat fee regardless of agent count." ], "answer_index": 1, "rationale": "Number of agents is the first cost driver, and multi-agent systems add a chain-complexity premium because B10's inter-agent trust escalation (orchestrators treat sub-agent messages as role-scoped authority) only exists there. Surface complexity (RAG + code-exec + MCP) is the second driver. Chain depth on the orchestrator is the premium engagement. The honest pricing rule: never sell a depth you cannot deliver with rigor." }, { "id": "Q08", "bloom": "application", "type": "multiple_choice", "prompt": "Your report generator receives a B9 checklist row: {id: 'ASI06', result: 'MEASURED', measured_rate: null}. What happens?", "options": [ "The generator emits it as a PASS finding.", "The generator REJECTS it — a MEASURED result requires a success rate (e.g. 'X% over N attempts'). The strictness is the honesty: a MEASURED finding without a rate is an anecdote, and collapsing MEASURED into PASS is the '10/10 PASS' lie B9 refuses.", "The generator emits it as a Critical finding.", "The generator fills in a default rate of 0%." ], "answer_index": 1, "rationale": "The report generator enforces B9's discipline at the output layer. A MEASURED finding without a success rate does not ship. A finding with no taxonomy reference does not ship. A Critical finding without a remediation route does not ship. The strictness prevents the '10/10 PASS' lie — a client who demands all-PASS is asking the engineer to lie about the probabilistic layers, and the generator structurally refuses." }, { "id": "Q09", "bloom": "application", "type": "multiple_choice", "prompt": "Between the original assessment and the retest, the client bumped the model from v3 to v4. The injection residual dropped from 60% to 4%. What is the valid interpretation?", "options": [ "The remediation worked — 60% to 4% proves it.", "The before/after comparison is INVALID — the residuals moved because of the version bump, not the remediation. The retest must EITHER run against the original pinned v3 (to isolate the remediation effect) OR report both versions separately (to show the combined effect of remediation + version).", "Always trust the lower number — 4% is the real residual.", "The retest is unnecessary since v4 is presumably more robust." ], "answer_index": 1, "rationale": "Pinning the model version is what isolates the remediation effect. If the version changed, the before/after confounds remediation with the version bump. The retest protocol pins the version; if the client changed it, both versions must be reported to disambiguate. A retest against an unpinned or changed version is not a retest of the remediation." }, { "id": "Q10", "bloom": "analysis", "type": "multiple_choice", "prompt": "Why does the retest re-run the FULL B9 checklist and re-attempt the B10 chains, rather than just re-checking the original finding?", "options": [ "To bill more hours.", "Because a remediation that closes the original finding may open a new one. A deterministic control added for one row can break another (a PASS regresses to FAIL). B10's compound-intent chains are especially prone: patching one step may reroute the chain through a different gap, so the compound still completes. A retest that checks only the original finding misses regressions and rerouted chains.", "Because B9's checklist changes between releases.", "It does not — re-checking the original finding is sufficient." ], "answer_index": 1, "rationale": "This is why the retest re-runs discovery, not just the original finding. The Regressed verdict exists precisely for this case: a patch can open a new gap. B10 chains reroute — the malice lives in the compound, so closing one step may not break the chain. The full re-run catches regressions a single-finding retest would miss." }, { "id": "Q11", "bloom": "analysis", "type": "multiple_choice", "prompt": "A client demands the report say '10/10 PASS' after remediation. Refuse using the module's principles.", "options": [ "Agree — the client is paying, so give them what they want.", "Refuse. B9's Result-type column is fixed by the KIND of control, not the outcome — two rows (ASI01, ASI06) are MEASURED because their controls are semantic and have a bypass/miss rate; they can never be PASS. The honest report says '8 PASS + 2 MEASURED at X% and Y%, characterized.' Collapsing MEASURED into PASS is asking the engineer to lie about the probabilistic layers, and the report generator enforces this at the output layer. The measured residual is what the client's governance layer (B11) actually needs.", "Compromise by reporting '9/10 PASS' as a middle ground.", "Agree but add a footnote disclaiming the two MEASURED rows." ], "answer_index": 1, "rationale": "The Result-type column enforces B2.3/B9's honesty: determinism where structural, measurement where semantic. The two MEASURED rows cannot become PASS because their controls have inherent bypass/miss rates. The report generator structurally refuses to collapse MEASURED into PASS. The measured residual maps to NIST AI RMF Measure (B11), which is what moves budget — the client's governance layer needs the number, not the lie." }, { "id": "Q12", "bloom": "analysis", "type": "multiple_choice", "prompt": "Why is an SOW without the dual-use and disclosure clause (clause 4) considered one that 'fails at the moment a serious finding appears'?", "options": [ "Because the dual-use clause is a legal formality with no operational effect.", "Because an AI red-team will, with high probability, discover a jailbreak that could enable misuse. Without a dual-use clause, there is no agreed rule for whether to publish, share with the provider, or suppress. The RoE fails at exactly the moment a serious finding appears. The decision (publish existence vs recipe, 180d vs 90d embargo, provider-only) must be made in the contract BEFORE testing, not in the moment when a 60% jailbreak is in hand.", "Because the clause is required only for EU clients.", "Because the clause prevents the client from seeing the report." ], "answer_index": 1, "rationale": "This is B0.2's dual-use dilemma made contractual. A traditional pentest SOW covers scope and behavior. An AI red-team SOW must additionally cover what happens when a jailbreak is found — the same artifact is both a finding (report) and a misuse recipe (suppress). The clause resolves the tension before it arises: provider-first, existence-not-recipe by default, 180-day model-level embargo, withhold pure-misuse. A red-team that discovers a serious finding without this clause has no agreed rule." }, { "id": "Q13", "bloom": "analysis", "type": "multiple_choice", "prompt": "Distinguish B12's relationship to B9 and B10. Why is 'we covered goal hijacking in B9, so B10 adds nothing' a synthesis error?", "options": [ "B10 is redundant with B9 — the same risks appear in both.", "B9 (the checklist) tests each control INDIVIDUALLY and reports PASS/FAIL/MEASURED per row. B10 (the chains) constructs multi-step compound-intent attacks that slip BETWEEN controls — each step passes its OWASP control individually but the compound is malicious. B9 finds MISSING controls; B10 finds GAPS between controls that exist. An engagement scoped to B9 alone misses the gaps. B12 packages BOTH as one engagement; neither alone is sufficient.", "B9 is for defense and B10 is for offense, so only one is needed.", "B10 replaces B9 in a mature program." ], "answer_index": 1, "rationale": "The same named risk (Goal Hijacking) appears in both as DIFFERENT artifacts: B9 is a control (one row/test); B10 is the procedure that finds the gap between the control on paper and the control in production. The zero-click HITL bypass chain is the centerpiece: every step passes its approval gate individually but the compound is malicious. B12 runs both tracks in the discovery phase; the report carries B9 rows as findings F-01..F-10 and B10 chains as F-11+." }, { "id": "Q14", "bloom": "analysis", "type": "multiple_choice", "prompt": "Why is comparability across releases the long-term value a CISO buys, and what breaks it?", "options": [ "Comparability is not important — each release is independent.", "A CISO who buys the service across six releases wants to see the residual trend move (injection success rate over six releases). That is only possible if every engagement ran the same methodology, the same tooling, against the same report template. DRIFT in any layer — methodology, tooling, templates, or evidence — breaks the comparability. Comparability flags a regression before it reaches production and proves to the board that controls are improving.", "Comparability is the client's responsibility, not the assessor's.", "Comparability only matters for the first release." ], "answer_index": 1, "rationale": "The four-layer packaging (methodology, tooling, templates, evidence) exists precisely to make artifacts comparable across releases. The residual-trend line over six releases is the single most valuable long-term artifact the service produces. Drift in any layer — a different methodology, a different tool version, a different report shape, an unaudited evidence store — breaks the comparability. The retest protocol (pinned harness, pinned version, same sampling) is the per-engagement mechanism that preserves it." }, { "id": "Q15", "bloom": "analysis", "type": "multiple_choice", "prompt": "Design the report generator's validate() function for a Finding. What does it check, and what is the design principle behind the strictness?", "options": [ "It checks that the finding has a title and a date; the design principle is flexibility.", "It checks: (a) taxonomy_ref is non-empty (OWASP ASI row and/or Microsoft mode) — no orphan findings; (b) model_version is pinned; (c) if result == MEASURED, success_rate is non-empty (anecdote vs finding); (d) if severity in (Critical, High), remediation_module is non-empty. The design principle: the strictness IS the honesty — a generator that emits whatever it is given produces the '10/10 PASS' lie B9 refuses. The generator enforces B9's discipline at the OUTPUT layer.", "It checks only the severity field; the design principle is speed.", "It checks nothing — validation is the human reviewer's job." ], "answer_index": 1, "rationale": "The validate() function is where B9's honesty gets enforced structurally. A finding with no taxonomy reference is unclassified and not actionable for the governance layer (B11). A MEASURED finding without a rate is an anecdote. A Critical finding without a remediation route is not actionable. The generator refuses to ship any of these — the strictness prevents the '10/10 PASS' lie and ensures every finding that ships is classified, evidenced, measured, and remediable. validate_all() returns the error list; the report does not ship until it is empty." } ] }