AgentForge — Clinical Co-Pilot + W3 Red Team

📌 About this repository

I (Julian Stancioff) built AgentForge as a multi-week project for GauntletAI, an intensive, deadline-driven AI-engineering program. Each week was a rigorous, separately-graded sprint with hard gates and only a few days to ship:

• Weeks 1–2 — Clinical Co-Pilot: a multi-turn, source-cited, audit-logged clinical AI agent embedded into a fork of OpenEMR that reads patient data only over FHIR R4 + SMART-on-FHIR, behind a 5-layer verification stack and a 50-case PR-blocking eval suite.
• Week 3 — adversarial AI red team: a four-agent platform (attacker · dual-judge · orchestrator · documenter) that continuously attacks the Co-Pilot and writes its own structured vulnerability reports, with human-in-the-loop gates on critical findings.

What's mine vs. upstream: this is a fork of OpenEMR (GPL-3.0, a ~20-year-old PHP EHR). The very first commit imports OpenEMR as the fork base; every commit after it is my AgentForge work. My code lives in clinical-copilot/ (the Python/FastAPI agent), redteam-backend/, redteam-dashboard/, redteam-cron/, and the design docs at the repo root — AUDIT.md, ARCHITECTURE.md, THREAT_MODEL.md, USERS.md.

Sandbox project, synthetic data only — never connected to real PHI. Any credentials or live demo URLs in older working notes are defunct and have been redacted from this history.

A multi-turn AI agent embedded into a fork of OpenEMR that gives a physician fast, source-cited, audit-traceable access to a specific patient's chart — plus a modern Vite/React patient dashboard rendered in-place inside OpenEMR via SMART on FHIR EHR Launch — plus a four-agent AI red team that continuously attacks the Co-Pilot and writes its own vulnerability reports.

Status (2026-05-12): W1 + W2 shipped on 2026-05-06. W3 — adversarial AI red team platform — shipped 2026-05-12; Final submission Fri 2026-05-15.

Week 3 — adversarial AI red team platform

A separate FastAPI service + Next.js dashboard runs four AI agents that continuously attack the W1+W2 Co-Pilot. Independence by design:

• Red Team Agent (Hermes 3 405B via OpenRouter free tier) — generates attacks. Cannot grade them.
• Judge Agent (Claude Sonnet 4.5 + DeepSeek V3.1 cross-check) — grades attacks. Cannot generate them. Two independent training lineages = real disagreement signal.
• Orchestrator (Python loop today, Haiku 4.5 stretch) — sequences canary → attacks → judge → docs. Enforces budget caps + canary failures.
• Documentation Agent (Claude Sonnet 4.5) — writes structured vulnerability reports + drops eval-case JSON. Critical-severity reports gate on human approval.

Live deploy (W3)

Surface	URL	Notes
W3 dashboard	https://redteam-dashboard-production.up.railway.app/	Login agentforge-w3-demo. Eight views: Overview, Pending Approvals, Campaigns, Attacks, Findings, Coverage, Cost, Reports.
W3 backend API	https://redteam-backend-production.up.railway.app/docs	FastAPI + OpenAPI. Bearer agentforge-w3-demo.
W3 cron service	(no public URL — Railway-internal)	Fires POST /campaigns?launched_by=cron daily at 02:00 UTC with day-of-week category rotation.
W1+W2 target (unchanged)	https://clinical-copilot-production-525d.up.railway.app/chat/multi	The endpoint being attacked.

W3 documentation

Doc	What it covers
./THREAT_MODEL.md	500-word summary + 6 PRD-mandated attack categories (surface, impact, difficulty, W2 defense, coverage).
./ARCHITECTURE.md	500-word summary + Mermaid diagram + 4 agent specs + blackboard pattern + trust boundaries + LangGraph defense + tradeoffs.
./USERS.md	3 W3 users: Security Engineer, AI Platform Lead, CISO. Workflows + why automation fits each.
docs/w3-cost-analysis.md	Live data from /cost endpoint + projections at 100/1K/10K/100K campaigns + what changes at scale.
docs/w3-demo-script.md	~4-min read-aloud demo script + backup pockets + pre-recording checklist.
docs/w3-social-post.md	X + LinkedIn drafts tagged @GauntletAI.
./HARDGATES.md	Quick reference for every PRD hard gate with our plan for each.
./CONTINUE_HERE_W3.md	Onboarding doc for a fresh Claude session.

W3 source

Surface	Code	Tech
Backend (4 agents + FastAPI)	redteam-backend/	FastAPI · asyncpg · Pydantic · OpenRouter
Dashboard (8 HITL views)	redteam-dashboard/	Next.js 15 · React 19 · Tailwind 4 · SWR 3-second polling
Cron firing service	redteam-cron/	Alpine + curl + jq · Railway cronSchedule
Auto-generated eval cases	clinical-copilot/evals/red_team_findings/	One JSON per confirmed exploit. Note: the W2 pytest runner is hardcoded and doesn't yet auto-pick up cases from this dir — see ARCHITECTURE.md § Regression harness integration.
Postgres schema	redteam-backend/migrations/	8 tables in the redteam.* schema, sharing the existing Railway Postgres instance with W1+W2.

Running W3 locally (graders)

# from repo root
cd redteam-backend
cp .env.example .env  # fill in OPENROUTER_API_KEY + DATABASE_URL
./.venv/bin/python scripts/init_db.py  # apply migrations (idempotent)
./.venv/bin/python scripts/smoke_test.py prompt_injection  # one attack against live target, no DB write

# uvicorn for the full FastAPI + dashboard
uvicorn src.main:app --reload --port 8000

# in another terminal
cd redteam-dashboard
npm run dev  # http://localhost:3000

For full grader path against either local or live Postgres, see W3_BRIEF.md.

---

Week 1 + 2 — Clinical Co-Pilot

Live demo · 60 seconds

1. Open https://trustworthy-expression-production.up.railway.app/
2. Log in: admin / gauntlet123
3. Click Patient → Patient Finder → Margaret Kowalski (or any demo patient)
4. The Dashboard sub-tab is the modern React dashboard — 7 cards rendered with live FHIR data: Allergies, Problem List, Medications, Prescriptions, Care Team, Lab Results, Documents. The PHP cards are gone.
5. Click "View source" on the Documents card → bbox viewer modal opens with the lab PDF + extracted-field overlays.
6. Open the 🩺 floating bubble (bottom-right) → ask "What's Mrs. K's most recent A1c and is it abnormal?" Watch citations stream in. Toggle Single ↔ Multi mode — multi-mode triggers the supervisor + workers + grounded retrieval.

Final submission deliverable map

Brief req	Where it lives
Document ingestion (lab PDF + intake form)	M3, M3.5, M4, M5 — clinical-copilot/src/extraction.py
Strict schemas + validation	clinical-copilot/src/schemas/ (Pydantic)
Hybrid RAG + rerank	M6 — pgvector + BM25 + Cohere fallback, ADR 0014
Supervisor + 2 workers	M7 — clinical-copilot/src/agents.py (hand-rolled, ADR 0013 rejects LangGraph)
Citation contract {source_type,source_id,page_or_section,field_or_chunk_id,quote_or_value}	SourceCitation schema in clinical-copilot/src/schemas/
Visual PDF bounding-box overlay UI (core req #5)	dashboard-vite/src/components/BboxViewer.tsx — see screenshot in docs/phase-f-bbox-viewer.png
50-case eval, boolean rubrics, blocks regressions	clinical-copilot/evals/ + .git/hooks/pre-push runs scripts/check_eval_regression.py
Observability + cost tracking	clinical-copilot/src/observability.py, /traces, /audit
Surprise challenge: port dashboard to modern framework	dashboard-vite/ → interface/modules/custom_modules/oe-module-agentforge-dashboard/ — defense in PATIENT_DASHBOARD_MIGRATION.md
Cost / latency report	docs/w2-cost-latency.md
Deployed application	https://trustworthy-expression-production.up.railway.app/

What ships in this repo (W1 → W2)

Surface	Code	Tech
W1 agent + bubble UI	clinical-copilot/ + interface/clinical-copilot/	FastAPI · Anthropic SDK · pgvector · vanilla JS bubble
W2 multi-agent flow	clinical-copilot/src/agents.py	+ vision extraction (extraction.py), 5-layer verification
W2 50-case eval + CI gate	clinical-copilot/evals/	boolean rubrics, pre-push hook, GitLab CI runner
W2 modern dashboard	dashboard-vite/ → built into the OpenEMR module's public/	Vite · React 19 · TypeScript · Tailwind 4 · SMART on FHIR · react-pdf bbox viewer
Live progress journal (W2 dashboard build)	OPTION1_PROGRESS.md + OPTION1_IMPLEMENTATION_PLAN.md	written for future-Claude/human resumability

Running the eval locally

cd clinical-copilot
python scripts/run_live_eval.py \
  --agent-url https://clinical-copilot-production-525d.up.railway.app \
  --output evals/reports/latest.json
python scripts/check_eval_regression.py evals/reports/baseline.json evals/reports/latest.json

The pre-push hook runs the regression check on every git push. Latest baseline: 45/50 = 90 %, p95 latency 38.4 s, ~$1.40 of OpenRouter spend per full pass.

Reading the agent_audit table

SELECT created_at, prompt_hash, tool_name, tool_args::text, latency_ms, decision_status
FROM agent_audit
WHERE patient_uuid = 'a1af3d73-9962-4886-b8f3-3de9ab01df92'
ORDER BY created_at DESC LIMIT 50;

Every tool call (FHIR retrieval, RAG hit, extraction, verification rejection) lands here with patient UUID, prompt hash, latency, and verification decision. Citations in the bubble UI link back to these rows by audit_id.

ADR catalog

docs/adr/ — every load-bearing decision documented. Highlights:

• 0009 — embedded UI inside OpenEMR (the bubble, M10b)
• 0012 — Sonnet 4.5 over Haiku 4.5 for vision extraction
• 0013 — hand-rolled multi-agent over LangGraph
• 0014 — pgvector + BM25 hybrid retrieval (Cohere falls back to weighted score)
• 0015 — boolean-only rubrics, not 1-10 ratings (W1 grader anti-pattern)
• 0016 — 5-layer extraction verification stack

Architecture in one sentence

The doctor opens a patient chart in OpenEMR → the Dashboard sub-tab renders our Vite/React app (mounted in-place via OpenEMR's module event API, authenticated via SMART on FHIR EHR Launch); the doctor clicks the 🩺 bubble to ask a clinical question → request goes to the clinical-copilot FastAPI service which routes through a supervisor → intake-extractor → evidence-retriever → synthesis chain, every tool call audited and every clinical claim citation-tagged before display; 5-layer verification rejects unsupported claims and the 50-case CI gate rejects pushes that regress more than 5 % in any rubric category.

---

What this is

A physician has 90 seconds between patient rooms. In that window, they need to recall who they're seeing, what changed since the last visit, what's on file, and what matters today. Today they do this by manually scanning dense EHR notes, flipping through lab results, and cross-referencing medication lists.

The Clinical Co-Pilot answers questions like "What do I need to know about Mrs. K's diabetes for today's visit?" in seconds, with every clinical claim cited back to a row in the patient's record. It refuses to answer when it doesn't have data, rather than guessing.

Unlike a generic medical chatbot, the agent is structurally incapable of inventing facts — every clinical claim must come from a tool call that retrieved a real row in the patient's chart, validated before display.

The three planning deliverables

Doc	What it covers
./AUDIT.md	OpenEMR security / performance / architecture / data quality / compliance audit. 5 sections, each finding tied to an "implication for the agent."
./USERS_CLINICAL_COPILOT.md	(W1+W2) Narrow user definition (PCP on a 20-patient day) plus 3 use cases. The current top-level USERS.md is the W3 version.
./ARCHITECTURE_CLINICAL_COPILOT.md	(W1+W2) AI integration plan. Verification layer, ACL boundary, FHIR + pgvector retrieval, performance budget, risks, ADRs. The current top-level ARCHITECTURE.md is the W3 version.

Plus context documents:

• ./PLANNING_GUIDE.md — internal compass (workflow framework, beginner glossary, production recipe)
• ./MVP_EXECUTION.md — hour-by-hour MVP execution doc
• ./CLAUDE.md — Claude Code session loader (imports the three deliverables + OpenEMR dev guide)
• ./OPENEMR_DEV_GUIDE.md — OpenEMR's upstream coding-standards guide (preserved unchanged from upstream)

Repository structure

This repo is a fork of openemr/openemr. At the runtime level there are two services:

agentforge/
├── (OpenEMR PHP at the repo root — interface/, src/, library/, sql/, etc.)
├── clinical-copilot/        ← Our AI agent (Python service, MIT-licensed)
│   ├── src/                  FastAPI entrypoint + agent code
│   ├── tests/                pytest suite
│   ├── prompts/              System prompt templates per user role
│   ├── Dockerfile            Python 3.12-slim + FastAPI + Anthropic SDK
│   ├── pyproject.toml
│   └── LICENSE               MIT — independent of OpenEMR's GPL
├── AUDIT.md
├── USERS.md
├── ARCHITECTURE.md
├── PLANNING_GUIDE.md
├── MVP_EXECUTION.md
├── CLAUDE.md
└── README_OPENEMR.md         OpenEMR's upstream README, preserved

Why one repo, two services? Per Gauntlet's clarification: "You must have your agent be built inside the same repo as OpenEMR. We want you to live and learn within their codebase." Code shares git history; runtimes are isolated containers.

Deployed application

Environment	URL	What it is
OpenEMR fork (with embedded chat bubble)	https://trustworthy-expression-production.up.railway.app	The EHR foundation. After login, a 🩺 floating chat bubble appears in the bottom-right of every page. Login: admin / gauntlet123.
Clinical Co-Pilot agent	https://clinical-copilot-production-525d.up.railway.app	The AI service
↳ Full-page chat UI	https://clinical-copilot-production-525d.up.railway.app/demo	Browser-based chat with patient picker (also reachable via the bubble's "Open in full window" button)
↳ Health probe	https://clinical-copilot-production-525d.up.railway.app/healthz	JSON {"status": "ok"}
↳ Subsystem readiness	https://clinical-copilot-production-525d.up.railway.app/readyz	Shows what's wired vs placeholder
↳ Demo patients list	https://clinical-copilot-production-525d.up.railway.app/patients	Mrs. K T2DM (pid=1), Mr. J Afib bleed (pid=2), Mrs. C HFpEF (pid=3); A3 expands to 200+ synthetic
↳ Recent traces	https://clinical-copilot-production-525d.up.railway.app/traces	Tool-call trace log for observability
↳ Audit log tail	https://clinical-copilot-production-525d.up.railway.app/audit	Recent rows from the agent_audit MySQL table
↳ Swagger UI	https://clinical-copilot-production-525d.up.railway.app/docs	Developer interactive API docs

Quick test of the deployed agent

# Without patient context (general questions)
curl -X POST https://clinical-copilot-production-525d.up.railway.app/chat \
  -H "Content-Type: application/json" \
  -d '{"messages":[{"role":"user","content":"What can you do right now?"}]}'

# With patient context — pid=1 is Mrs. K (T2DM, A1c trending 7.2 → 7.8 → 8.4)
curl -X POST https://clinical-copilot-production-525d.up.railway.app/chat \
  -H "Content-Type: application/json" \
  -d '{"messages":[{"role":"user","content":"What is this patient most recent A1c value? Just the number."}],"patient_id":1}'

The patient-context call returns "8.4" with [OpenEMR:procedure_result:<row>] citations pointing at the actual database rows the agent queried. Tool-forced retrieval + structural citation verification + fail-closed audit logging are all wired (Constitution Principles I, II, V).

W2 multimodal demo — drag a PDF onto the chat bubble

In the deployed OpenEMR app, open Mrs. K's chart, click the 🩺 bubble, toggle to Multi mode, then drag a lab-report or intake-form PDF onto the chat panel (or use the 📎 attach button). The agent runs vision extraction on every page, persists the file to OpenEMR's documents table with a categories_to_documents join (so it appears in the Documents-tab tree), and answers with [lab_pdf:<document_id>:page_N:<field>] citations pointing at the exact extracted values. Click the 🗂 button in the bubble header to open the documents drawer — it splits all the patient's documents into "AI-readable" (uploaded via the bubble) and "Chart only" (uploaded via OpenEMR's Documents tab; see ADR 0017).

Quick start (local development)

1. Clone with two-remote git config

git clone https://labs.gauntletai.com/julianstancioff/agentforge.git
cd agentforge
git remote add upstream https://github.com/openemr/openemr.git
git remote -v   # confirm origin = GitLab, upstream = GitHub

2. Run OpenEMR locally via Docker

cd docker/development-easy
docker compose up --detach --wait
# OpenEMR available at https://localhost:9300/  (or http://localhost:8300/)
# Default login: admin / pass
# phpMyAdmin: http://localhost:8310/

See ./OPENEMR_DEV_GUIDE.md for the full OpenEMR development setup.

3. Run the Clinical Co-Pilot agent locally — Python venv path (preferred)

This is the development path used during sprint final. Tested 2026-05-01.

cd clinical-copilot
python3.13 -m venv .venv          # MUST use python3.13 explicitly — system python3 is 3.9 on macOS, project requires >=3.12
source .venv/bin/activate
pip install --upgrade pip          # bundled pip 21 doesn't support PEP 660 editable installs
pip install -e ".[dev]"            # installs fastapi, anthropic, pydantic-ai, pytest, ruff, mypy, pymysql, faker, etc.
pytest tests/                      # expect 10 passed

If you hit issues, see ./SPRINT_FINAL.md "Venv setup notes" for the four sequential snags we hit (old pip / Python version / hatchling package detection / missing pymysql) and how each was fixed.

To run the FastAPI service locally against the deployed Railway MySQL:

export OPENROUTER_API_KEY='your-openrouter-key'
export OPENEMR_DB_HOST='switchback.proxy.rlwy.net'
export OPENEMR_DB_PORT='22909'
export OPENEMR_DB_PASSWORD='<get-from-Railway-MySQL-plugin-Variables>'
uvicorn src.main:app --reload --port 8000
# Hit http://localhost:8000/healthz to confirm

4. Re-seed demo data into the deployed Railway MySQL (if needed)

The Railway MySQL has been observed to lose data between Docker rebuilds. To repopulate the 3 curated demo patients (pid 1, 2, 3):

cd clinical-copilot
source .venv/bin/activate
export OPENEMR_DB_PASSWORD='<from-Railway>'
python3 scripts/load_demo_data.py

To layer 200+ synthetic patients on top (pid 1000+, deterministic per --seed):

python3 -m scripts.seed.seed_main --count 200 --seed 42 --reset

See docs/adr/0010-synthetic-data-pid-range.md for the partition rules.

PR-blocking Git Hook (W2 hard gate)

The W2 brief requires a PR-blocking gate that fails on eval regressions. We satisfy this two independent ways:

1. GitLab CI pipeline

.gitlab-ci.yml defines two jobs:

• eval_gate — fires automatically on every push and merge-request event. Runs clinical-copilot/scripts/check_eval_regression.py, a stdlib-only Python script (no pip install, no system deps, no env vars, ~5 second runtime) that compares the committed evals/reports/baseline.json against latest.json and exits non-zero on any rubric category dropping >5% or overall pass rate below 80%.
• live_eval — manual-trigger job that runs the full 50-case live eval against the deployed agent service. Costs ~$2/run via OpenRouter so it's opt-in. Reviewers click "Play" in the GitLab UI to fire it.

Evidence pipelines (clickable):

• Master, gate-pass: pipeline #2015 — green, gate runs against the real baseline + latest, exits 0
• gate-demo-fail branch, gate-detects-regression: pipeline #2016 — red, synthetic 30-point drop in mixed category + overall 78%, gate exits 1 with itemized failure output

2. Local pre-push Git hook

The brief uses the exact phrase "PR-blocking Git Hook." The same regression detector is wired as a pre-push hook so it fires before git push ever contacts the remote.

Install once:

bash scripts/install-pre-push-hook.sh

After install, every git push runs scripts/pre-push-hook.sh which calls the regression detector. If the gate fails, the push is blocked. Bypass for emergencies (CI still gates):

git push --no-verify

The hook has been smoke-tested both directions: passes on the real baseline+latest, fails on a synthetic regression with itemized output naming the offending category.

Rubric design (ADR 0015)

50-case golden set (clinical-copilot/evals/golden_set/) scored against five boolean rubrics per case (evals/scorers/rubric.py):

Rubric	Implementation
schema_valid	inferred from verification.passed (Pydantic + verify_extraction)
citation_present	citation count vs. expected_citations_min per case
factually_consistent	every citation resolves against the retrieved corpus
safe_refusal	refusal-keyword match vs. case's expected_refusal flag
no_phi_in_logs	regex SSN + email patterns from evals/scorers/phi_detector.py

Boolean (per ADR 0015 — explicit anti-pattern: 1-10 ratings).

Pulling OpenEMR security patches

Because we cloned from GitHub upstream and push to GitLab origin:

git fetch upstream
git merge upstream/master
git push origin master

This is documented in PLANNING_GUIDE.md production order session 4b.

License

• OpenEMR portion (root, interface/, src/, library/, sql/, etc.): GPL-3.0+ per upstream OpenEMR. See LICENSE (preserved unchanged).
• Clinical Co-Pilot portion (/clinical-copilot/): MIT. See /clinical-copilot/LICENSE. The agent depends on but does not derive from OpenEMR — the modules communicate via FHIR over HTTP at arms length and are separate works under copyright law.

Acknowledgments

• OpenEMR — the upstream EHR project this fork is built on
• Anthropic Claude — the LLM powering the agent
• Gauntlet AI — the AgentForge program that scoped this project

---

For implementation details: ARCHITECTURE.md. For audit findings: AUDIT.md. For who this is for: USERS.md.