Instrument Development Plan — Winter 2027 Course

The software that runs the course, and doubles as reference implementations for an open research program on governing agentic systems.

The course operates as an instrumented agent deployment: every student agent runs against a budget, acts through sandboxed tools, leaves a complete trace, and is supervisable from an instructor dashboard. These are the same mechanisms the research program studies (budget governance, enforcement placement, failure attribution, oversight at scale), so each course instrument is also a research deliverable.

Instruments

I1. Course gateway (keystone; new build). A model-access proxy through which all student agents call LLM APIs: per-student keys, token metering, hard and soft budget caps, complete request/response logging, and model-agnostic routing. One component provides budgets, traces, and provider independence at once; everything else attaches to it. Build on an existing open-source LLM proxy rather than from scratch. Status (Oct 12): ~80% exists — course-infra skeleton smoke-tested (per-student keys, hard budget caps, auth rejection, both lanes configured including the live ALCF endpoint with Globus token refresher). Remaining: AWS deployment, TLS, model pinning. Security requirements (these are IRB protocol commitments, not preferences): TLS on all connections, encryption at rest, role-based administrative access with University 2FA, students able to access only their own sessions, access logging, no unauthenticated endpoints.

I2. Budget governor (adapt Bursar). Policy layer on the gateway: Allocations, burst behavior, back pressure, and a per-student usage dashboard. Course role: Week-3 lab and quarter-long cost control. Research role: Reference budget-governance implementation. Status: The enforcement primitive (hard cap with refusal) works today via the gateway. Bursar has a ~600-line prototype (broker, leases, Parsl integration) whose concepts transfer but whose code targets HPC campaigns; the token-policy layer (soft warnings, burst, student dashboard) is unbuilt.

I3. Tool sandbox and permissions (adapt Resource Rights). Containerized code execution plus capability-scoped tool access mediated at the tool layer (MCP), so an agent’s permissions are set by policy, not prompt. Course role: All labs from week 2; the week-6 enforcement-placement experiment. Research role: Reference enforcement mechanism. Status: ~half exists — Resource Rights includes a working delegation-service prototype (~1,000 lines, with tests); a governed-tool-use example exists in Capabilities. Unbuilt: the resource-limited container execution service and MCP permission wiring into the gateway path.

I4. Trace store and viewer. A common trace schema (every model call, tool call, and observation, with costs), durable storage keyed by student and session, and a web viewer good enough to scrub through a session in class. Course role: Weekly autopsies. Research role: The atlas ingestion format. Status (Oct 12): WORKING — OpenTelemetry callback to Arize Phoenix, verified with live traffic: full request/response content, usage, and per-student identity in spans. Session grouping wired via request metadata (filter by session id in the viewer); remaining: same convention in the student starter kit, and production deployment.

I5. Oversight dashboard. Instructor view over the fleet: Live usage, recent sessions, budget and anomaly flags, escalations. Version 0 is usage plus trace links; refinement during the quarter is itself an oversight-design exercise. Status: V0 largely exists (the gateway admin UI provides usage by key); fleet view, anomaly flags, and escalations are unbuilt.

I6. Bake-off harness and leaderboard. Task specification, runner, scoring, and a class leaderboard, built on existing open evaluation tooling. Course role: Bake-offs 1–3 and the reproduction challenge. Research role: The fidelity-tier environment pattern. Status: Does not exist — the largest genuine construction item; open eval tooling is the base, but harness, leaderboard, and task specs are all unbuilt (sprint 3’s main event).

I7. Consent and redaction module (phase 2, with IRB). Opt-in flag at the gateway; de-identification at ingest (including conversion of absolute timestamps to session-relative times, per IRB protocol); export path to the open corpus. Built during the quarter; nothing is exported before IRB approval. Status: Fully specified (IRB-committed requirements, including session-relative timestamps); zero code, by design.

I8. Labeling pipeline (phase 2). LLM judges plus a human-validation interface, seeded by autopsy sessions. Built during weeks 3–6, once real traces exist. Status: Does not exist; correctly deferred until real traces accumulate.

Found assets beyond the instrument list. The Capabilities tutorials contain a dozen-plus worked agent examples (local agents, dashboards, multi-agent coordination, long-lived agents, governed tool use, scaled inference) — most of the week 1–2 lab starter material already exists and needs adaptation to the gateway, not authoring.

Model supply: Two lanes

The gateway routes to two model lanes with assigned roles.

Commercial frontier (default lane). Labs, projects, and bake-offs run on frontier commercial models: The course lives or dies on whether student agents work, and long-horizon tool use is where frontier models retain a reliability edge. Funded by sponsored credits or purchase order (initiated in sprint 0).

Argonne inference service, open models (second lane). Four jobs where it is better, not just cheaper: (1) Red-team week—deliberate injection attacks without vendor safety filters or terms-of-service confounds, on premises; (2) bulk background workloads—LLM judges, bake-off scoring, synthetic content generation; (3) a standing open-vs-frontier comparison—bake-offs optionally run on both lanes, with a model column on the leaderboard; (4) overflow if the commercial budget runs hot. Traces on this lane never leave the premises, which simplifies consent language for the research export. Lane usage is shadow-priced against student budgets (~half of frontier rates) even though lab inference is free to the course: Cost-awareness applies on every lane, and the open-vs-frontier tradeoff stays real rather than zero. Constraints: Request the allocation in sprint 0; schedule nothing graded solely on this lane (maintenance windows); the gateway provides a tool-calling shim for open-model API differences.

Accounts and access

Target: One new credential per student. The gateway key is each student’s single course identity—model access on both lanes, budget enforcement, trace attribution, and login for the trace viewer and leaderboard. Students additionally need only a GitHub account (added to a course team in this org) and their CNetID.

Deliberately not required: Commercial provider accounts (the gateway holds the keys; no side door around budgets or trace capture); individual ALCF accounts (the Argonne lane is served through a project-level service credential under the instructor’s allocation—to be confirmed in sprint 0, since individual ALCF account processing for 30 students, including foreign nationals, would not complete by January); sandbox accounts (local containers on student laptops for labs; the course sandbox service authorizes by gateway identity for bake-offs and projects). Client-project teams receive client-provisioned accounts per client commitment #2, before the sprint begins.

Week-0 setup assignment (sent before the quarter; verified before session 1): Gateway key issued and smoke-tested, GitHub handle in the course team, Docker installed and working.

Timeline (October 2026 → March 2027)

Priorities. P0 (class cannot run without): I1, I3, I4. P1: I2, I5, I6. P2: I7, I8.

Staffing. One RSE or TA plus instructor; coding agents are used throughout for implementation, with the development itself instrumented—the toolchain eating its own output is part of the research program.

Risks. (1) Token provisioning logistics (per-student access under an institutional or sponsored account) — resolve in sprint 0. (2) IRB timing — submit now; the course runs regardless, since I7 gates only the research export, not the class. (3) Scope creep — the gateway is the only novel P0 component; everything else adapts existing code or tooling.

Resources (assuming 30 students)

Tokens. Budget roughly 50M tokens per student for the quarter (weekly labs at 2–5M tokens, bake-off bursts, and a long-horizon project tail), enforced by the governor. With ~30% overhead for TA and instructor use, the December dry run, red-team sweeps, LLM judges, and development itself: 2–3B tokens total, roughly $10–20K at blended list prices ($300–500 list-equivalent per student). Hold a 30% reserve unallocated; running dry in week 7 is the most demoralizing failure available. Procurement, not amount, is the risk: Sponsored education credits or a purchase order must be initiated in sprint 0.

People. TA-1 (infrastructure): The committed engineering quarter—but note the build happens in November–December, before winter TA appointments begin, so the autumn gap (~2 person-months) needs RA, RSE, or hourly funding. TA-2 (course operations): At 30 students with weekly deliverables, one person cannot both run infrastructure and grade; a second TA or grader is strongly recommended. Instructor plus two TAs gives a 1:10 ratio during studio time, about the floor for hands-on teaching.

Compute and hosting. Decided: AWS via the University of Chicago’s enterprise agreement. An always-on instance hosts the gateway, trace store, and dashboards (~$100–200/month); burst sandbox capacity runs during labs and bake-offs (~$200–400/month in-quarter); trace storage in S3 (tens of GB—trivial). Under $2K for the quarter. Security configuration follows the IRB commitments above. Sprint-0 confirmations: the enterprise-account route, and whether the University requires a security review for systems holding student records.

Content. The reproduction paper set (15–30 papers with code and data, license-checked, pre-tested by TAs for feasibility—2–3 weeks of curation; artifact-badged papers are the shortcut); the week-6 red-team pack; 4–6 client commitments by mid-December; the IRB protocol (effort, not money, but on the critical path now).

Mapping to the research program