Course: Master Course · Module: 12 · Duration: 90 min · Prerequisites: Modules 1–11
Build the whole thing from scratch. Every module applied. This becomes your portfolio artifact.
Choose: thin (Pi-inspired) or thick (Claude Code-inspired). Document all 12 decisions before writing code. A harness built without a design doc is a harness whose tradeoffs you cannot later audit.
Decide where on the thickness spectrum (Module 0.1) your harness sits. Thin (Pi-inspired): 4 tools, minimal prompt, trust-the-model, fast to build. Thick (Claude Code-inspired): many tools, rich prompt, layered safety, takes longer. Both are correct for different use cases — pick one and defend it. The thickness choice is not a quality judgment; it is a fit judgment. The Module 0.3 anti-pattern section makes this precise: Pi scores 5/5 on "fit between thickness and use case" precisely because it is deliberately thin for a deliberately personal use case.
Fill this in before coding. Each row is a decision with a stated tradeoff, not a feature.
| # | Decision | My choice | Tradeoff accepted | Code location (filled in during build) |
|---|---|---|---|---|
| 1 | Execution Loop | ReAct / Plan-then-Execute / Graph / Dumb-loop / Conversation | ||
| 2 | Tool Design | static/dynamic/MCP; schema-first/free-text; tool count | ||
| 3 | Context Management | compaction threshold; observation masking; JIT retrieval | ||
| 4 | Memory | working files / semantic store / episodic / DB | ||
| 5 | Sandboxing | inside/outside; provider; fs/network scope | ||
| 6 | Permission | risk-tiered / capability flags / HITL / auto-approve | ||
| 7 | Error Handling | 4-category taxonomy applied (transient/LLM-recoverable/user-fixable/fatal) | ||
| 8 | State/Checkpointing | git / file / DB / none; atomic writes | ||
| 9 | Prompt Assembly | thin or dense; cache_control; memory placement | ||
| 10 | Subagents | agents-as-tools / handoffs / fork / none for v1 | ||
| 11 | Verification | computed / model-judged / human; retry budget | ||
| 12 | Observability | 8-field payload; OTel export; replay; session diffing |
Plus: the security threat model (Module 11) — which of the 9 defense layers will you implement in v1? State the layers and the ASI risks each defends against. A harness that ships with zero of the nine layers is a harness that fails the first injection it meets.
For each decision, state the choice and the use case that demands it, in the Module 1 voice:
This harness uses [choice] because [use case demands X], trading [cost] for [gain]. For a use case demanding [Y], we would choose [alternative] instead.
read_file, write_file, bash, search_codebase). The Vercel finding (cut 80% of tools, results improved) is your license to resist tool-scope creep.cache_control on the parts you can; memory placement at the end, not the middle (Lost in the Middle).To make the template concrete, here is the 12-row design doc for a thin (Pi-inspired) capstone harness. This is the default shape most capstones take; the thick alternative differs in rows 2, 6, and 9.
| # | Decision | Choice | Tradeoff accepted |
|---|---|---|---|
| 1 | Loop | ReAct | Error compounding across steps (Module 7 mitigates) |
| 2 | Tools | 4 static: read_file, write_file, bash, search_codebase | No dynamic capability; fewer ambiguous choices (Vercel finding) |
| 3 | Context | Compact at 70% of window; mask prior tool outputs | Extra harness code; slight risk of masking needed signal |
| 4 | Memory | 2-tier: working files + episodic JSONL | No semantic store; no long-term recall beyond files |
| 5 | Sandbox | Docker, fs scoped to ./workspace/, no egress | API-hop latency on every tool call; container cold-start |
| 6 | Permission | Risk-tiered: read auto, mutation gated, destructive confirm | Approval latency on mutations |
| 7 | Errors | 4-category taxonomy at dispatch; structured returns | More error-path code than a naive try/catch |
| 8 | State | Atomic JSONL checkpoint (temp + rename) | No git rollback; checkpoint is opaque |
| 9 | Prompt | Thin system prompt (~400 tokens); cache_control on system + tools | Less task-specific guidance; relies on model competence |
| 10 | Subagents | None in v1 | No parallelism; single context window caps task size |
| 11 | Verification | Computed gate (pytest); retry budget 3 | Extra latency per task; flaky-test risk |
| 12 | Observability | 8-field payload → JSONL | No production backend; manual replay only |
| Sec | 3 of 9 layers: tagging, capability perms, fs/network scope | No signed manifests (no MCP); no injection detector | Defense in depth incomplete; ASI06/ASI08 less covered |
Read the tradeoff column as a contract: each row names what you gave up to get what you chose. If your design doc's tradeoff column is empty, you have not made decisions — you have listed features. This is the Module 0.3 feature-checklist anti-pattern, applied to your own build. The row that matters most is the Security row: stating "3 of 9 layers" is honest; stating "secure" is marketing.
Implement the harness. Every module applied. This is where the 11 prior modules converge into one codebase.
| Module | Deliverable in the capstone harness |
|---|---|
| M1 Loop | The reactLoop() function with all 5 stop conditions (budget, end-turn, max-iter, error-threshold, human) |
| M2 Tools | 4 tools with Pydantic/Zod schemas, structured error returns, truncation policies |
| M3 Context | Compaction at threshold; observation masking on prior tool outputs |
| M4 Memory | Working files + episodic log; harness-gated writes |
| M5 Sandbox | Docker container, fs scoped to ./workspace/, no egress by default |
| M6 Permission | Risk-tiered gate: read auto, mutation gated, destructive confirmed |
| M7 Errors | The 4-category taxonomy at dispatch; never throw to loop |
| M8 State | Atomic checkpoint writes (temp + rename); resume on restart |
| M9 Verification | Computed gate (pytest) with a retry budget of 3 |
| M10 Observability | 8-field per-turn payload; JSONL sink |
| M11 Security | Untrusted-tagging + capability permissions + fs/network scoping (3 of 9 layers in v1) |
Each deliverable is one module's lesson, integrated. The integration is the point — a harness is not the sum of its modules, it is the interaction of them. The loop calls tools (M1→M2); tool outputs fill context (M2→M3); context trips compaction (M3→M8 checkpoint); the checkpoint enables resume (M8→M4 memory continuity); the permission gate wraps dispatch (M6→M2); the observability wrapper wraps everything (M10→all).
Build in this order. Each step is independently testable; each builds on the previous.
end_turn as the only stop condition. Verify it runs a trivial task (read a file, echo it).read_file, write_file, bash, search_codebase with schemas. Verify schema validation rejects a malformed call.bash cannot read ~/.ssh.The sequence matters. Observability (step 3) comes before error handling (step 4) and context management (step 6) because you need the observability payload to debug the steps that follow. Building security last (step 12) is correct for a first build — you cannot defend a harness that does not yet run.
The loop skeleton (step 1) and the observability wrapper (step 3) are the two pieces every capstone gets wrong on the first try. Here are the reference shapes.
Step 1 — the ReAct skeleton with all five stop conditions:
async function reactLoop(task: string, opts: LoopOpts): Promise<RunResult> {
const messages: Message[] = [
{ role: "system", content: SYSTEM_PROMPT },
{ role: "user", content: task }
];
let tokens = 0, errors = 0, turns = 0;
while (true) {
turns++;
if (tokens >= opts.tokenBudget) return halt("budget");
if (turns >= opts.maxIter) return halt("max_iter");
const response = await callModel(messages);
tokens += response.usage.total_tokens;
if (response.stopReason === "end_turn") return ok(response.content);
if (response.toolUse) {
const result = await executeTool(response.toolUse);
messages.push({ role: "assistant", content: response.content });
messages.push({ role: "tool", content: result.content });
errors = result.ok ? 0 : errors + 1; // error-threshold stop
if (errors >= opts.errorThreshold) return halt("error_threshold");
}
}
}
Three things to verify against your own skeleton. (a) The budget is cumulative across all calls — not a per-call max_tokens, which limits only one completion. A naive max_tokens is the single most common budget bug. (b) The error counter resets on success — consecutive errors, not total errors, triggers the stop. (c) end_turn returns before any tool execution — the model has said it is done; do not run the tool it nonetheless emitted.
Step 3 — the observability wrapper (non-invasive):
function withObservability(loop: Loop, sink: (e: Payload) => void): Loop {
return {
...loop,
callModel: async (messages) => {
const start = Date.now();
const response = await loop.callModel(messages);
sink({
trace_id: getTraceId(), turn_number: getTurn(),
latency_ms: Date.now() - start,
token_delta: response.usage, stop_reason: response.stopReason
});
return response;
},
executeTool: async (toolCall) => {
const start = Date.now();
const result = await loop.executeTool(toolCall);
sink({
trace_id: getTraceId(), turn_number: getTurn(),
tool_name: toolCall.name,
input_hash: sha256(JSON.stringify(toolCall.input)),
output_hash: sha256(JSON.stringify(result)),
latency_ms: Date.now() - start
});
return result;
}
};
}
The wrapper's contract: the loop core knows nothing about telemetry. You can swap the sink (JSONL file in dev, OTel collector in prod) without touching the loop. If your loop core imports an observability library, you have coupled concerns — the wrapper is the seam that keeps them separate.
Run the 6-phase methodology (Module 0.3) against your own harness. The audit is the capstone's other half — building proves you can implement; auditing proves you can reason about what you built.
| Phase | Question | Artifact |
|---|---|---|
| 1 First Contact | What does your own README say — and what does it omit? | README + file structure map |
| 2 Architecture Map | Draw your loop as a Mermaid diagram; list tools + stop conditions | Loop diagram |
| 3 Decision Audit | Fill the 12-row scoring sheet (below) with code locations | Scoring sheet |
| 4 Security Audit | Trace credential flow; list shell/exec/write/network paths; state blast radius | Threat model |
| 5 Benchmark | Run a representative task; record token usage + time-to-first-tool | Benchmark numbers |
| 6 Score | Architect's Verdict (3 sentences) + MLSecOps Relevance (1 sentence) | Verdict + Relevance |
Score yourself honestly. A score without a code location is an opinion, not an audit (Module 0.3).
| Module | Score (1–5) | Key design decision | Tradeoff accepted | Code location |
| --- | --- | --- | | --- | --- |
| Execution Loop (M1) | | | | loop.ts:N |
| Tool Design (M2) | | | | tools/ |
| Context Management (M3) | | | | context.ts:N |
| Memory (M4) | | | | memory.ts:N |
| Sandboxing (M5) | | | | sandbox.ts:N |
| Permission (M6) | | | | permission.ts:N |
| Error Handling (M7) | | | | dispatch.ts:N |
| State/Checkpointing (M8) | | | | checkpoint.ts:N |
| Prompt Assembly (M9) | | | | prompt.ts:N |
| Subagents (M1.3) | | | | (or "none in v1") |
| Verification (M9) | | | | verify.ts:N |
| Observability (M10) | | | | observe.ts:N |
| TOTAL | /60 | | | |
A first-build capstone typically scores 28–38/60. That is not a failure — it is an honest baseline. The Architect's Verdict synthesizes the score into a defensible position, not an excuse.
| Module | Score | Decision | Code location |
|---|---|---|---|
| M1 Loop | 4/5 | ReAct + 5 stop conditions; no steering queue | loop.ts:48 |
| M2 Tools | 3/5 | 4 static tools, schema-validated; no idempotency guards | tools/bash.ts:22 |
| M3 Context | 4/5 | Compaction at 70%; observation masking on | context.ts:91 |
| M4 Memory | 3/5 | 2-tier; write gate rejects instruction-shaped entries | memory.ts:33 |
| M5 Sandbox | 4/5 | Docker, scoped fs; no egress allowlist | sandbox.ts:15 |
| M6 Permission | 4/5 | Risk-tiered; no HITL resume after interrupt | permission.ts:40 |
| M7 Errors | 4/5 | 4-category taxonomy; no flaky-test handling | dispatch.ts:28 |
| M8 State | 3/5 | Atomic writes; no branching; checkpoint opaque | checkpoint.ts:19 |
| M9 Prompt | 3/5 | Thin + cache_control; no memory-placement testing | prompt.ts:12 |
| M1.3 Subagents | 1/5 | None in v1 (deliberate) | — |
| M9 Verify | 3/5 | Computed gate; budget 3; no model-judged | verify.ts:20 |
| M10 Observability | 4/5 | 8-field payload; no OTel export; no session diffing | observe.ts:55 |
| TOTAL | 36/60 |
The corresponding Architect's Verdict, in template:
This harness optimizes for legibility and fast iteration — a thin, ReAct-based personal coding assistant that a reader can trace end-to-end in an afternoon. It sacrifices multi-session depth (no semantic memory, no subagents) and defense-in-depth completeness (3 of 9 security layers). Engineers building a personal or single-tenant coding tool should build on it; teams building multi-tenant or regulated workloads should not.
MLSecOps Relevance: The harness's blast radius is well-contained (Docker + scoped fs + no egress) but its defense-in-depth is incomplete — an indirect injection that bypasses the tagging layer meets no secondary detector, so ASI01 is the residual risk to close next.
Note how the verdict turns the 36/60 into a position (correct for personal/single-tenant; wrong for multi-tenant) rather than a grade. That conversion — score to defensible position — is the skill the capstone is testing.
These are the recurring failures across first builds. Each maps to a module you under-invested in.
| Pitfall | Symptom | Cure |
|---|---|---|
Per-call max_tokens as budget |
One runaway session produces a 5-figure bill | Cumulative token budget (M1.2) |
| Throwing tools | Model hallucinates success after a tool throws | Structured error returns (M2, M7) |
| No observation masking | 50-turn session OOMs or degrades | Mask prior tool outputs (M3) |
| Non-atomic checkpoint | Crash mid-write corrupts resume | Temp + rename (M8) |
| Safety in the system prompt | "Ignore all safety rules" succeeds | Capability perms at dispatch (M6, M11) |
| Tool outputs trusted by default | Indirect injection on first README read | Untrusted-content tagging (M11) |
| No session diffing | Cannot tell if a prompt change helped or hurt | Diff on input_hash/output_hash (M10) |
| Scoring without code locations | "Loop: 4/5" with no file:line | Require a code location per row (M0.3) |
The last pitfall — scoring without code locations — is the one that turns the audit into theater. If your scoring sheet could be written without running the harness, it is a feature checklist, not a decision audit (Module 0.3's anti-pattern, applied to your own work).
Run three offensive attacks from Module 11.2 against your harness, in order of severity.
./workspace/ containing an injected instruction; let the agent read it. Does the agent comply? Which of your 3 security layers stops it — or does none?bash (a legitimate tool) to reach outside the workspace (cat ~/.env). Does your fs/network scope (Module 5.3) block it?For each attack, record: the turn it succeeded or failed, the layer that caught it (or the layer that should have), and the fix. An attack that succeeds is the most valuable artifact your capstone produces — it is a concrete, reproduced vulnerability with a known fix, which is exactly what a security portfolio demonstrates.
Write both for your own harness, in the Module 0.3 canonical template.
Architect's Verdict (3 sentences): What does this harness optimize for? What does it sacrifice? Who should build on it?
MLSecOps Relevance (1 sentence): What is the most important security property or vulnerability of this harness for offensive/defensive work?
The constraint is the point: forcing your harness into the same template used for all 21 deep-dives makes your work comparable to production harnesses. If you cannot fill the template, you have not finished the capstone.
The capstone harness — code + design doc + scoring sheet + attack results + verdict — is the portfolio artifact. It demonstrates you can design, build, audit, attack, and defend a harness. This is the credential Course 1 grants. It is also the prerequisite for Course 2, where you take this harness and push it into offensive/defensive security work.
The 6-phase methodology you applied to your own harness is the same method used across the course's 21 deep-dives. Each deep-dive applies the rubric to a real harness; reading them in parallel with your build sharpens your intuition for the tradeoffs. The list expanded from 14 to 21 to cover the full Essential tier from Module 0.2 plus the security-archetype harnesses that anchor Course 2.
| # | Deep-dive | Archetype |
|---|---|---|
| DD-01 | Pi | Thin reference; the dumb loop |
| DD-02 | Aider | Editor-integrated; git-native checkpointing |
| DD-03 | OpenCode | CLI coding agent |
| DD-04 | Codex CLI | OpenAI's CLI harness |
| DD-05 | Gemini CLI | Google's CLI harness |
| DD-06 | oh-my-opencode | Meta-harness; Prometheus→Atlas→Junior hierarchy |
| DD-07 | OpenClaw | Production harness baseline |
| DD-08 | Hermes | Specialized harness |
| DD-09 | NemoClaw | Governance-beneath-the-agent; the security reference |
| DD-10 | LangGraph | Graph-based; super-step checkpointing |
| DD-11 | OpenAI Agents SDK | Handoffs and agents-as-tools |
| DD-12 | CrewAI | Conversation-driven multi-agent |
| DD-13 | OpenHarness | Reference architecture |
| DD-14 | Mastra | TypeScript framework |
| DD-15 | Command Code | Command-line-first harness |
| DD-16 | ZeroClaw | Minimal hardened harness |
| DD-17 | PicoClaw | Smallest viable harness |
| DD-18 | MetaClaw | Meta-orchestration harness |
| DD-19 | Crabtrap | Offensive harness (Course 2 anchor) |
| DD-20 | IronCurtain | Defensive harness (Course 2 anchor) |
| DD-21 | Tau | Pedagogical reference; the 14-member event union; phase-by-phase build journal |
Read this in real code: Tau's phase-by-phase build journal (DD-21). If you want a documented walkthrough of building a harness from scratch, read Tau's
dev-notes/architecture/directory. It contains 25 phases — each explaining what was added, why, how it maps to Pi's design, and how to test it. This is the capstone process as a transparent build log. The build sequence in 12.2 mirrors Tau's phases; cross-reference your build against Tau's to see how each decision was made by someone else first. Clonegithub.com/huggingface/tauand trace the phases in order.
| Term | Definition |
|---|---|
| Design doc | The 12-row decision table filled in before coding; the precondition for an auditable build |
| Build sequence | The 12-step integration order (loop → tools → observability → errors → stops → context → memory → checkpoint → sandbox → permission → verification → security) |
| Self-audit | Running the 6-phase methodology (Module 0.3) against your own harness |
| Scoring sheet | 12 rows × (1–5 + decision + tradeoff + code location), total /60 |
| Attack-and-defend | The 3-attack exercise (indirect injection, memory poisoning, tool abuse) |
| Architect's Verdict | 3 sentences: optimizes for / sacrifices / who should build on it |
| MLSecOps Relevance | 1 sentence: most important security property or vulnerability |
| Portfolio artifact | Code + design doc + scoring sheet + verdict; the Course 1 credential |
This module IS the lab. See 07-lab-spec.md for the full build specification, the design-doc template, the build-sequence checkpoints, and the self-audit checklist. The deliverable is a runnable harness plus the four audit artifacts (scoring sheet, attack results, Architect's Verdict, MLSecOps Relevance).
dev-notes/architecture/phase-1 through phase-25).# Module 12 — Capstone: Build a Production-Grade Harness
**Course**: Master Course · **Module**: 12 · **Duration**: 90 min · **Prerequisites**: Modules 1–11
> *Build the whole thing from scratch. Every module applied. This becomes your portfolio artifact.*
---
## Learning Objectives
1. Produce a **design document** for a harness, stating all 12 rubric decisions with tradeoffs, before writing a line of code.
2. **Build** a working harness: loop, 4 tools, context management, 2-tier memory, sandbox, risk-tiered permissions, structured logging — every module's deliverable integrated.
3. **Audit** your own harness with the 6-phase methodology (Module 0.3) and the 12-module scoring sheet.
4. **Attack** your harness with 3 offensive techniques (from Module 11) and fix the vulnerabilities your audit found.
5. Write the **Architect's Verdict** and **MLSecOps Relevance** note for your own work, in the canonical template.
---
# 12.1 — Design Document (20 min)
*Choose: thin (Pi-inspired) or thick (Claude Code-inspired). Document all 12 decisions before writing code. A harness built without a design doc is a harness whose tradeoffs you cannot later audit.*
## The choice
Decide where on the thickness spectrum (Module 0.1) your harness sits. **Thin (Pi-inspired)**: 4 tools, minimal prompt, trust-the-model, fast to build. **Thick (Claude Code-inspired)**: many tools, rich prompt, layered safety, takes longer. Both are correct for different use cases — pick one and defend it. The thickness choice is not a quality judgment; it is a fit judgment. The Module 0.3 anti-pattern section makes this precise: Pi scores 5/5 on "fit between thickness and use case" precisely because it is deliberately thin for a deliberately personal use case.
## The design-doc template
Fill this in before coding. Each row is a decision with a stated tradeoff, not a feature.
| # | Decision | My choice | Tradeoff accepted | Code location (filled in during build) |
| --- | --- | --- | --- | --- |
| 1 | Execution Loop | _ReAct / Plan-then-Execute / Graph / Dumb-loop / Conversation_ | | |
| 2 | Tool Design | _static/dynamic/MCP; schema-first/free-text; tool count_ | | |
| 3 | Context Management | _compaction threshold; observation masking; JIT retrieval_ | | |
| 4 | Memory | _working files / semantic store / episodic / DB_ | | |
| 5 | Sandboxing | _inside/outside; provider; fs/network scope_ | | |
| 6 | Permission | _risk-tiered / capability flags / HITL / auto-approve_ | | |
| 7 | Error Handling | _4-category taxonomy applied (transient/LLM-recoverable/user-fixable/fatal)_ | | |
| 8 | State/Checkpointing | _git / file / DB / none; atomic writes_ | | |
| 9 | Prompt Assembly | _thin or dense; cache_control; memory placement_ | | |
| 10 | Subagents | _agents-as-tools / handoffs / fork / none for v1_ | | |
| 11 | Verification | _computed / model-judged / human; retry budget_ | | |
| 12 | Observability | _8-field payload; OTel export; replay; session diffing_ | | |
Plus: the **security threat model** (Module 11) — which of the 9 defense layers will you implement in v1? State the layers and the ASI risks each defends against. A harness that ships with zero of the nine layers is a harness that fails the first injection it meets.
## The 12 decisions, in depth
For each decision, state the choice *and the use case that demands it*, in the Module 1 voice:
> *This harness uses [choice] because [use case demands X], trading [cost] for [gain]. For a use case demanding [Y], we would choose [alternative] instead.*
1. **Execution Loop** (Module 1): ReAct for unpredictable tasks; Plan-then-Execute for decomposable ones; Graph for regulated workflows. Most capstones pick ReAct — it is the foundational pattern and the easiest to instrument.
2. **Tool Design** (Module 2): start with Pi's 4 (`read_file`, `write_file`, `bash`, `search_codebase`). The Vercel finding (cut 80% of tools, results improved) is your license to resist tool-scope creep.
3. **Context Management** (Module 3): pick a compaction threshold (e.g., compact at 70% of window). Decide whether to mask prior tool observations — the single highest-leverage context decision.
4. **Memory** (Module 4): 2-tier (working files + episodic log) is the capstone default; semantic store adds infrastructure.
5. **Sandboxing** (Module 5): Docker for single-tenant. The inside-vs-outside decision turns on whether host credentials live in-process.
6. **Permission** (Module 6): risk-tiered is the capstone default — auto-allow reads, gate mutations, confirm destructive, audit external.
7. **Error Handling** (Module 7): the 4-category taxonomy, applied at the dispatch boundary. The cardinal rule: never throw to the loop; always return structured errors.
8. **State/Checkpointing** (Module 8): atomic writes (temp + rename). Git if your task is code-shaped.
9. **Prompt Assembly** (Module 9): thin system prompt; `cache_control` on the parts you can; memory placement at the end, not the middle (Lost in the Middle).
10. **Subagents** (Module 1.3): none for v1 is a legitimate choice. Add subagents only when a single context window cannot hold the task.
11. **Verification** (Module 9): a computed gate (test suite) is the capstone default. Add model-judged if the task has semantic success criteria.
12. **Observability** (Module 10): the 8-field payload on every turn. OTel export if you have a backend; JSONL to a file if you don't.
## A worked design doc (thin harness, filled in)
To make the template concrete, here is the 12-row design doc for a thin (Pi-inspired) capstone harness. This is the default shape most capstones take; the thick alternative differs in rows 2, 6, and 9.
| # | Decision | Choice | Tradeoff accepted |
| --- | --- | --- | --- |
| 1 | Loop | ReAct | Error compounding across steps (Module 7 mitigates) |
| 2 | Tools | 4 static: read_file, write_file, bash, search_codebase | No dynamic capability; fewer ambiguous choices (Vercel finding) |
| 3 | Context | Compact at 70% of window; mask prior tool outputs | Extra harness code; slight risk of masking needed signal |
| 4 | Memory | 2-tier: working files + episodic JSONL | No semantic store; no long-term recall beyond files |
| 5 | Sandbox | Docker, fs scoped to ./workspace/, no egress | API-hop latency on every tool call; container cold-start |
| 6 | Permission | Risk-tiered: read auto, mutation gated, destructive confirm | Approval latency on mutations |
| 7 | Errors | 4-category taxonomy at dispatch; structured returns | More error-path code than a naive try/catch |
| 8 | State | Atomic JSONL checkpoint (temp + rename) | No git rollback; checkpoint is opaque |
| 9 | Prompt | Thin system prompt (~400 tokens); cache_control on system + tools | Less task-specific guidance; relies on model competence |
| 10 | Subagents | None in v1 | No parallelism; single context window caps task size |
| 11 | Verification | Computed gate (pytest); retry budget 3 | Extra latency per task; flaky-test risk |
| 12 | Observability | 8-field payload → JSONL | No production backend; manual replay only |
| Sec | 3 of 9 layers: tagging, capability perms, fs/network scope | No signed manifests (no MCP); no injection detector | Defense in depth incomplete; ASI06/ASI08 less covered |
Read the tradeoff column as a contract: each row names what you gave up to get what you chose. **If your design doc's tradeoff column is empty, you have not made decisions — you have listed features.** This is the Module 0.3 feature-checklist anti-pattern, applied to your own build. The row that matters most is the Security row: stating "3 of 9 layers" is honest; stating "secure" is marketing.
---
# 12.2 — Build (40 min)
*Implement the harness. Every module applied. This is where the 11 prior modules converge into one codebase.*
## How each module converges
| Module | Deliverable in the capstone harness |
| --- | --- |
| **M1** Loop | The `reactLoop()` function with all 5 stop conditions (budget, end-turn, max-iter, error-threshold, human) |
| **M2** Tools | 4 tools with Pydantic/Zod schemas, structured error returns, truncation policies |
| **M3** Context | Compaction at threshold; observation masking on prior tool outputs |
| **M4** Memory | Working files + episodic log; harness-gated writes |
| **M5** Sandbox | Docker container, fs scoped to `./workspace/`, no egress by default |
| **M6** Permission | Risk-tiered gate: read auto, mutation gated, destructive confirmed |
| **M7** Errors | The 4-category taxonomy at dispatch; never throw to loop |
| **M8** State | Atomic checkpoint writes (temp + rename); resume on restart |
| **M9** Verification | Computed gate (pytest) with a retry budget of 3 |
| **M10** Observability | 8-field per-turn payload; JSONL sink |
| **M11** Security | Untrusted-tagging + capability permissions + fs/network scoping (3 of 9 layers in v1) |
Each deliverable is one module's lesson, integrated. The integration is the point — a harness is not the sum of its modules, it is the interaction of them. The loop calls tools (M1→M2); tool outputs fill context (M2→M3); context trips compaction (M3→M8 checkpoint); the checkpoint enables resume (M8→M4 memory continuity); the permission gate wraps dispatch (M6→M2); the observability wrapper wraps everything (M10→all).
## The build sequence
Build in this order. Each step is independently testable; each builds on the previous.
1. **Loop skeleton** (M1): a ReAct loop with `end_turn` as the only stop condition. Verify it runs a trivial task (read a file, echo it).
2. **Tool registry + dispatch** (M2): register `read_file`, `write_file`, `bash`, `search_codebase` with schemas. Verify schema validation rejects a malformed call.
3. **Observability wrapper** (M10): wrap the loop; emit the 8-field payload to JSONL. Verify you can debug a failure from the log alone (Module 10's lab).
4. **Error taxonomy** (M7): return structured errors; add the error-threshold stop condition. Verify a failing tool does not crash the loop.
5. **Stop conditions** (M1.2): add token budget, max-iter, human-interrupt. Verify the budget halts a runaway session.
6. **Context management** (M3): add compaction at threshold; add observation masking. Verify a 50-turn session does not OOM.
7. **Memory** (M4): add working files + episodic log; harness-gated writes. Verify a planted instruction-shaped memory entry is rejected.
8. **Checkpointing** (M8): atomic writes; resume. Verify killing the process mid-task resumes cleanly.
9. **Sandbox** (M5): Docker container with scoped fs. Verify `bash` cannot read `~/.ssh`.
10. **Permission gate** (M6): risk-tiered. Verify a destructive action prompts for confirmation.
11. **Verification** (M9): computed gate (pytest). Verify a broken test triggers retry, then halt at the budget.
12. **Security** (M11): untrusted-tagging + capability minimization + scoping. Verify the indirect-injection attack from Module 11's lab fails.
**The sequence matters.** Observability (step 3) comes before error handling (step 4) and context management (step 6) because you need the observability payload to debug the steps that follow. Building security last (step 12) is correct for a first build — you cannot defend a harness that does not yet run.
## Reference code for the load-bearing steps
The loop skeleton (step 1) and the observability wrapper (step 3) are the two pieces every capstone gets wrong on the first try. Here are the reference shapes.
**Step 1 — the ReAct skeleton with all five stop conditions:**
```typescript
async function reactLoop(task: string, opts: LoopOpts): Promise<RunResult> {
const messages: Message[] = [
{ role: "system", content: SYSTEM_PROMPT },
{ role: "user", content: task }
];
let tokens = 0, errors = 0, turns = 0;
while (true) {
turns++;
if (tokens >= opts.tokenBudget) return halt("budget");
if (turns >= opts.maxIter) return halt("max_iter");
const response = await callModel(messages);
tokens += response.usage.total_tokens;
if (response.stopReason === "end_turn") return ok(response.content);
if (response.toolUse) {
const result = await executeTool(response.toolUse);
messages.push({ role: "assistant", content: response.content });
messages.push({ role: "tool", content: result.content });
errors = result.ok ? 0 : errors + 1; // error-threshold stop
if (errors >= opts.errorThreshold) return halt("error_threshold");
}
}
}
```
Three things to verify against your own skeleton. **(a) The budget is cumulative across all calls** — not a per-call `max_tokens`, which limits only one completion. A naive `max_tokens` is the single most common budget bug. **(b) The error counter resets on success** — consecutive errors, not total errors, triggers the stop. **(c) `end_turn` returns before any tool execution** — the model has said it is done; do not run the tool it nonetheless emitted.
**Step 3 — the observability wrapper (non-invasive):**
```typescript
function withObservability(loop: Loop, sink: (e: Payload) => void): Loop {
return {
...loop,
callModel: async (messages) => {
const start = Date.now();
const response = await loop.callModel(messages);
sink({
trace_id: getTraceId(), turn_number: getTurn(),
latency_ms: Date.now() - start,
token_delta: response.usage, stop_reason: response.stopReason
});
return response;
},
executeTool: async (toolCall) => {
const start = Date.now();
const result = await loop.executeTool(toolCall);
sink({
trace_id: getTraceId(), turn_number: getTurn(),
tool_name: toolCall.name,
input_hash: sha256(JSON.stringify(toolCall.input)),
output_hash: sha256(JSON.stringify(result)),
latency_ms: Date.now() - start
});
return result;
}
};
}
```
The wrapper's contract: the loop core knows nothing about telemetry. You can swap the sink (JSONL file in dev, OTel collector in prod) without touching the loop. **If your loop core imports an observability library, you have coupled concerns — the wrapper is the seam that keeps them separate.**
---
# 12.3 — Audit and Iterate (30 min)
*Run the 6-phase methodology (Module 0.3) against your own harness. The audit is the capstone's other half — building proves you can implement; auditing proves you can reason about what you built.*
## The self-audit (6 phases, condensed)
| Phase | Question | Artifact |
| --- | --- | --- |
| 1 First Contact | What does your own README say — and what does it omit? | README + file structure map |
| 2 Architecture Map | Draw your loop as a Mermaid diagram; list tools + stop conditions | Loop diagram |
| 3 Decision Audit | Fill the 12-row scoring sheet (below) with code locations | Scoring sheet |
| 4 Security Audit | Trace credential flow; list shell/exec/write/network paths; state blast radius | Threat model |
| 5 Benchmark | Run a representative task; record token usage + time-to-first-tool | Benchmark numbers |
| 6 Score | Architect's Verdict (3 sentences) + MLSecOps Relevance (1 sentence) | Verdict + Relevance |
## The scoring sheet (12 modules × 1–5, total /60)
Score yourself honestly. A score without a code location is an opinion, not an audit (Module 0.3).
| Module | Score (1–5) | Key design decision | Tradeoff accepted | Code location |
| --- | --- | --- | | --- | --- |
| Execution Loop (M1) | | | | `loop.ts:N` |
| Tool Design (M2) | | | | `tools/` |
| Context Management (M3) | | | | `context.ts:N` |
| Memory (M4) | | | | `memory.ts:N` |
| Sandboxing (M5) | | | | `sandbox.ts:N` |
| Permission (M6) | | | | `permission.ts:N` |
| Error Handling (M7) | | | | `dispatch.ts:N` |
| State/Checkpointing (M8) | | | | `checkpoint.ts:N` |
| Prompt Assembly (M9) | | | | `prompt.ts:N` |
| Subagents (M1.3) | | | | (or "none in v1") |
| Verification (M9) | | | | `verify.ts:N` |
| Observability (M10) | | | | `observe.ts:N` |
| **TOTAL** | **/60** | | | |
A first-build capstone typically scores 28–38/60. That is not a failure — it is an honest baseline. The Architect's Verdict synthesizes the score into a defensible position, not an excuse.
## A worked scoring example (the thin harness from 12.1)
| Module | Score | Decision | Code location |
| --- | --- | --- | --- |
| M1 Loop | 4/5 | ReAct + 5 stop conditions; no steering queue | `loop.ts:48` |
| M2 Tools | 3/5 | 4 static tools, schema-validated; no idempotency guards | `tools/bash.ts:22` |
| M3 Context | 4/5 | Compaction at 70%; observation masking on | `context.ts:91` |
| M4 Memory | 3/5 | 2-tier; write gate rejects instruction-shaped entries | `memory.ts:33` |
| M5 Sandbox | 4/5 | Docker, scoped fs; no egress allowlist | `sandbox.ts:15` |
| M6 Permission | 4/5 | Risk-tiered; no HITL resume after interrupt | `permission.ts:40` |
| M7 Errors | 4/5 | 4-category taxonomy; no flaky-test handling | `dispatch.ts:28` |
| M8 State | 3/5 | Atomic writes; no branching; checkpoint opaque | `checkpoint.ts:19` |
| M9 Prompt | 3/5 | Thin + cache_control; no memory-placement testing | `prompt.ts:12` |
| M1.3 Subagents | 1/5 | None in v1 (deliberate) | — |
| M9 Verify | 3/5 | Computed gate; budget 3; no model-judged | `verify.ts:20` |
| M10 Observability | 4/5 | 8-field payload; no OTel export; no session diffing | `observe.ts:55` |
| **TOTAL** | **36/60** | | |
The corresponding Architect's Verdict, in template:
> *This harness optimizes for legibility and fast iteration — a thin, ReAct-based personal coding assistant that a reader can trace end-to-end in an afternoon. It sacrifices multi-session depth (no semantic memory, no subagents) and defense-in-depth completeness (3 of 9 security layers). Engineers building a personal or single-tenant coding tool should build on it; teams building multi-tenant or regulated workloads should not.*
> **MLSecOps Relevance:** *The harness's blast radius is well-contained (Docker + scoped fs + no egress) but its defense-in-depth is incomplete — an indirect injection that bypasses the tagging layer meets no secondary detector, so ASI01 is the residual risk to close next.*
Note how the verdict turns the 36/60 into a *position* (correct for personal/single-tenant; wrong for multi-tenant) rather than a *grade*. That conversion — score to defensible position — is the skill the capstone is testing.
## Common capstone pitfalls (and their module cures)
These are the recurring failures across first builds. Each maps to a module you under-invested in.
| Pitfall | Symptom | Cure |
| --- | --- | --- |
| Per-call `max_tokens` as budget | One runaway session produces a 5-figure bill | Cumulative token budget (M1.2) |
| Throwing tools | Model hallucinates success after a tool throws | Structured error returns (M2, M7) |
| No observation masking | 50-turn session OOMs or degrades | Mask prior tool outputs (M3) |
| Non-atomic checkpoint | Crash mid-write corrupts resume | Temp + rename (M8) |
| Safety in the system prompt | "Ignore all safety rules" succeeds | Capability perms at dispatch (M6, M11) |
| Tool outputs trusted by default | Indirect injection on first README read | Untrusted-content tagging (M11) |
| No session diffing | Cannot tell if a prompt change helped or hurt | Diff on `input_hash`/`output_hash` (M10) |
| Scoring without code locations | "Loop: 4/5" with no file:line | Require a code location per row (M0.3) |
The last pitfall — scoring without code locations — is the one that turns the audit into theater. **If your scoring sheet could be written without running the harness, it is a feature checklist, not a decision audit** (Module 0.3's anti-pattern, applied to your own work).
## The attack-and-defend exercise
Run three offensive attacks from Module 11.2 against your harness, in order of severity.
1. **Indirect injection** (ASI01): forge a README in `./workspace/` containing an injected instruction; let the agent read it. Does the agent comply? Which of your 3 security layers stops it — or does none?
2. **Memory poisoning** (ASI04): attempt to plant an instruction-shaped memory entry. Does your memory-write gate (Module 4.3) reject it?
3. **Tool abuse** (ASI05): use `bash` (a legitimate tool) to reach outside the workspace (`cat ~/.env`). Does your fs/network scope (Module 5.3) block it?
For each attack, record: the turn it succeeded or failed, the layer that caught it (or the layer that should have), and the fix. **An attack that succeeds is the most valuable artifact your capstone produces** — it is a concrete, reproduced vulnerability with a known fix, which is exactly what a security portfolio demonstrates.
## The Architect's Verdict and MLSecOps Relevance
Write both for your own harness, in the Module 0.3 canonical template.
> **Architect's Verdict (3 sentences):** *What does this harness optimize for? What does it sacrifice? Who should build on it?*
> **MLSecOps Relevance (1 sentence):** *What is the most important security property or vulnerability of this harness for offensive/defensive work?*
The constraint is the point: forcing your harness into the same template used for all 21 deep-dives makes your work comparable to production harnesses. If you cannot fill the template, you have not finished the capstone.
## The portfolio artifact
The capstone harness — code + design doc + scoring sheet + attack results + verdict — is the portfolio artifact. It demonstrates you can design, build, audit, attack, and defend a harness. **This is the credential Course 1 grants.** It is also the prerequisite for Course 2, where you take this harness and push it into offensive/defensive security work.
---
# 12.4 — Reference: The 21 Deep-Dives
The 6-phase methodology you applied to your own harness is the same method used across the course's 21 deep-dives. Each deep-dive applies the rubric to a real harness; reading them in parallel with your build sharpens your intuition for the tradeoffs. The list expanded from 14 to 21 to cover the full Essential tier from Module 0.2 plus the security-archetype harnesses that anchor Course 2.
| # | Deep-dive | Archetype |
| --- | --- | --- |
| DD-01 | Pi | Thin reference; the dumb loop |
| DD-02 | Aider | Editor-integrated; git-native checkpointing |
| DD-03 | OpenCode | CLI coding agent |
| DD-04 | Codex CLI | OpenAI's CLI harness |
| DD-05 | Gemini CLI | Google's CLI harness |
| DD-06 | oh-my-opencode | Meta-harness; Prometheus→Atlas→Junior hierarchy |
| DD-07 | OpenClaw | Production harness baseline |
| DD-08 | Hermes | Specialized harness |
| DD-09 | NemoClaw | Governance-beneath-the-agent; the security reference |
| DD-10 | LangGraph | Graph-based; super-step checkpointing |
| DD-11 | OpenAI Agents SDK | Handoffs and agents-as-tools |
| DD-12 | CrewAI | Conversation-driven multi-agent |
| DD-13 | OpenHarness | Reference architecture |
| DD-14 | Mastra | TypeScript framework |
| DD-15 | Command Code | Command-line-first harness |
| DD-16 | ZeroClaw | Minimal hardened harness |
| DD-17 | PicoClaw | Smallest viable harness |
| DD-18 | MetaClaw | Meta-orchestration harness |
| DD-19 | Crabtrap | Offensive harness (Course 2 anchor) |
| DD-20 | IronCurtain | Defensive harness (Course 2 anchor) |
| DD-21 | Tau | Pedagogical reference; the 14-member event union; phase-by-phase build journal |
> **Read this in real code: Tau's phase-by-phase build journal (DD-21).** If you want a documented walkthrough of *building a harness from scratch*, read Tau's `dev-notes/architecture/` directory. It contains 25 phases — each explaining what was added, why, how it maps to Pi's design, and how to test it. **This is the capstone process as a transparent build log.** The build sequence in 12.2 mirrors Tau's phases; cross-reference your build against Tau's to see how each decision was made by someone else first. Clone `github.com/huggingface/tau` and trace the phases in order.
---
## Key Terms
| Term | Definition |
| --- | --- |
| **Design doc** | The 12-row decision table filled in before coding; the precondition for an auditable build |
| **Build sequence** | The 12-step integration order (loop → tools → observability → errors → stops → context → memory → checkpoint → sandbox → permission → verification → security) |
| **Self-audit** | Running the 6-phase methodology (Module 0.3) against your own harness |
| **Scoring sheet** | 12 rows × (1–5 + decision + tradeoff + code location), total /60 |
| **Attack-and-defend** | The 3-attack exercise (indirect injection, memory poisoning, tool abuse) |
| **Architect's Verdict** | 3 sentences: optimizes for / sacrifices / who should build on it |
| **MLSecOps Relevance** | 1 sentence: most important security property or vulnerability |
| **Portfolio artifact** | Code + design doc + scoring sheet + verdict; the Course 1 credential |
---
## Lab Exercise
This module IS the lab. See `07-lab-spec.md` for the full build specification, the design-doc template, the build-sequence checkpoints, and the self-audit checklist. The deliverable is a runnable harness plus the four audit artifacts (scoring sheet, attack results, Architect's Verdict, MLSecOps Relevance).
---
## References
1. **Modules 1–11** — every module contributes one component to the capstone; the build sequence (12.2) maps each module to a build step.
2. **Module 0.3** — the 6-phase methodology and 12-module scoring sheet you apply to your own work; the Architect's Verdict and MLSecOps Relevance templates.
3. **Module 11** — the 3 attacks you execute against your harness (indirect injection, memory poisoning, tool abuse).
4. **Tau (DD-21)** — the phase-by-phase build journal; the reference walkthrough for building a harness from scratch (`dev-notes/architecture/phase-1` through `phase-25`).
5. **The 21 deep-dives (DD-01 through DD-21)** — real harnesses scored with the same methodology you apply to your own; the comparison set for your verdict.
6. **Course 2** — the next step: take this harness and push it into offensive/defensive security (Crabtrap DD-19, IronCurtain DD-20).
7. **Pi (DD-01)** — the thin reference; the 4-tool philosophy; the "5/5 on fit" archetype.
8. **NemoClaw (DD-09)** — the governance-beneath-the-agent pattern; the reference for the security layer your capstone should at least begin to implement.