fix: weight:0 un-gates criteria and renders as informational

CLAUDE.md

@@ -35,7 +35,7 @@ coder_eval/
 │   ├── criteria.py                # 14 success criterion types + base + union
 │   ├── experiment.py              # ExperimentDefinition, ExperimentVariant, ResolvedTask, result models
 │   ├── judge_defaults.py          # DEFAULT_JUDGE_MODEL constant (cycle-free leaf)
-│   ├── mutations.py               # PromptMutation variants (prefix/suffix/replace/template/rephrase)
+│   ├── mutations.py               # PromptMutation variants (prefix/suffix/replace/template)
 │   ├── results.py                 # CriterionResult (+ ClassificationCriterionResult), TurnRecord, EvaluationResult, EarlyStopInfo/EarlyStopReason, CriterionAggregate, ThresholdCheck, SuiteRollup
 │   ├── routing.py                 # ApiRoute (DirectRoute/BedrockRoute)
 │   ├── sandbox.py                 # SandboxConfig, ResourceLimits
@@ -294,7 +294,7 @@ Tasks are YAML files. See [docs/TASK_DEFINITION_GUIDE.md](docs/TASK_DEFINITION_G
 
 **Runtime (always)**: pydantic, pydantic-settings, pyyaml, typer, rich, python-dotenv, anthropic, claude-agent-sdk, anyio, radon, tqdm, jmespath, jsonschema
 
-**Runtime (optional, `[uipath]` extra)**: uipath — the in-host `uipath` SDK (handy for local sandbox parity with tasks that invoke `uv run uipath eval ...`). Base installs without this extra still run end-to-end; UiPath-dependent paths fail at dispatch with a clear `pip install 'coder-eval[uipath]'` hint. The LLM judge and the `rephrase` mutation no longer use the LLM Gateway client — they route through the run's backend (Bedrock / Anthropic), so `uipath-llmgw-client` is no longer a dependency.
+**Runtime (optional, `[uipath]` extra)**: uipath — the in-host `uipath` SDK (handy for local sandbox parity with tasks that invoke `uv run uipath eval ...`). Base installs without this extra still run end-to-end; UiPath-dependent paths fail at dispatch with a clear `pip install 'coder-eval[uipath]'` hint. The LLM judge no longer uses the LLM Gateway client — it routes through the run's backend (Bedrock / Anthropic), so `uipath-llmgw-client` is no longer a dependency.
 
 **Dev**: pytest, pytest-asyncio, pytest-mock, pytest-cov, ruff, pyright, pip-audit, bandit, pre-commit, mcp
 

docs/AB_EXPERIMENTS.md

@@ -227,8 +227,8 @@ variants:
         text: "\n\nThink step by step and validate your work before finishing."
 ```
 
-The full mutation catalog (prefix / suffix / replace / template / rephrase) is
-defined in `coder_eval/models/mutations.py`.
+The full mutation catalog (prefix / suffix / replace / template) is defined in
+`coder_eval/models/mutations.py`.
 
 ## Recipe: Smoke vs. e2e Flavors (Early Stop)
 

docs/TASK_DEFINITION_GUIDE.md

@@ -275,11 +275,17 @@ sandbox:
     - "!node_modules"
     - "*.bak"                         #   bare entry adds an extra pattern
   limits:                             # Optional: resource limits
-    timeout: 300                       # Enforced via subprocess timeout
-    max_memory_mb: 512                 # NOT enforced (reserved for future use)
-    max_disk_mb: 1024                  # NOT enforced (reserved for future use)
+    timeout: 300                       # Enforced via subprocess timeout (both drivers)
+    max_memory_mb: 512                 # driver:docker -> `--memory`; ignored under tempdir
+    max_cpus: 2                        # driver:docker -> `--cpus`; ignored under tempdir
+    max_pids: 512                      # driver:docker -> `--pids-limit`; ignored under tempdir
+    max_disk_mb: 1024                  # NOT enforced (reserved: no portable docker knob)
 ```
 
+Under `driver: tempdir` only `timeout` is enforced — the agent can consume
+arbitrary host memory, CPU, and PIDs. Use `driver: docker` when you need the
+container limits above to actually bind.
+
 ## Template Sources
 
 Tasks can start with preset files instead of an empty sandbox. Multiple sources are applied sequentially (last wins for conflicts).
@@ -383,7 +389,7 @@ All criteria share these fields:
 | Field | Default | Description |
 |-------|---------|-------------|
 | `description` | — | Human-readable description (required) |
-| `weight` | 1.0 | Relative importance for weighted score |
+| `weight` | 1.0 | Relative importance for weighted score. `0` = **informational**: excluded from both the score and the pass/fail gate |
 | `pass_threshold` | 0.9 | Minimum score (0.0–1.0) to pass |
 | `stop_when` | `null` | Arms this criterion for early stop (`pass`/`fail`/`decided`); requires `run_limits.stop_early: true` and an observable criterion type (`skill_triggered`, `command_executed`). See [`stop_early`](#stop_early-opt-in-early-stop). |
 
@@ -392,7 +398,20 @@ All criteria share these fields:
 - **Fractional** (0.0–1.0): `file_contains`, `file_check`, `json_check`, `command_executed`, `uipath_eval`
 - **Continuous** (0.0–1.0): `reference_comparison`, `llm_judge`, `agent_judge`
 
-**Task success:** ALL criteria must score >= their `pass_threshold`.
+**Task success:** all *gating* criteria must score >= their `pass_threshold`. A
+criterion with `weight: 0` is informational — it is still checked, stored, and
+rendered in reports, but it neither contributes to the score nor fails the task.
+(A `weight: 0` criterion may not set `stop_when` or `suite_thresholds`: arming a
+non-gating criterion for the early-stop or suite gate would let an
+"informational" check flip a run to failure.)
+
+Every surface labels it as such rather than as a failure: the terminal shows `○`
+instead of `✓`/`✗`, the HTML report tags the row "informational — not gated" and
+excludes it from the *n*/*m* passed header, the evalboard renders an `INFO` pill,
+and a below-threshold informational criterion is never sampled as the reason a
+suite row failed. The persisted `CriterionResult.gating` field carries this to
+every consumer, so no reader needs the original criterion to know whether a low
+score mattered.
 
 **Weighted score:** `weighted_score = sum(score * weight) / sum(weight)` — calculated regardless for quality assessment.
 

evalboard/app/runs/[id]/[...task]/_chips.tsx

@@ -26,15 +26,25 @@ export function Expandable({
     );
 }
 
-export function ResultPill({ passed }: { passed: boolean }) {
-    const cls = passed
-        ? "bg-green-50 text-green-700 border-green-200"
-        : "bg-red-50 text-red-700 border-red-200";
+// `gating: false` (weight: 0) criteria are informational — they cannot fail the
+// task, so rendering PASS/FAIL for them would contradict the task's own status.
+export function ResultPill({
+    passed,
+    gating = true,
+}: {
+    passed: boolean;
+    gating?: boolean;
+}) {
+    const cls = !gating
+        ? "bg-gray-50 text-gray-600 border-gray-200"
+        : passed
+          ? "bg-green-50 text-green-700 border-green-200"
+          : "bg-red-50 text-red-700 border-red-200";
     return (
         <span
             className={`inline-block px-2 py-0.5 text-xs rounded-full border font-medium ${cls}`}
         >
-            {passed ? "PASS" : "FAIL"}
+            {!gating ? "INFO" : passed ? "PASS" : "FAIL"}
         </span>
     );
 }

evalboard/app/runs/[id]/[...task]/_sections.tsx

@@ -108,16 +108,27 @@ export function CriteriaSection({ criteria }: { criteria: CriterionResult[] }) {
         <section className="space-y-2">
             <h2 className="text-sm font-semibold text-gray-900">
                 Success criteria ({criteria.length})
+                {criteria.some((c) => !c.gating) && (
+                    <span className="ml-2 font-normal text-gray-500">
+                        {criteria.filter((c) => !c.gating).length}{" "}
+                        informational
+                    </span>
+                )}
             </h2>
             <div className="space-y-2">
                 {criteria.map((c, i) => {
-                    const passed = c.score === 1;
+                    // Compare against the criterion's own threshold, not === 1:
+                    // fractional criteria pass below 1.0 (default threshold 0.9).
+                    const passed = (c.score ?? 0) >= c.passThreshold;
                     return (
                         <Expandable
                             key={i}
                             header={
                                 <div className="flex items-center gap-3">
-                                    <ResultPill passed={passed} />
+                                    <ResultPill
+                                        passed={passed}
+                                        gating={c.gating}
+                                    />
                                     <span className="text-sm text-gray-900">
                                         {c.description ??
                                             c.criterionType ??

evalboard/lib/runs.ts

@@ -103,6 +103,12 @@ export interface CriterionResult {
     score: number | null;
     details: string | null;
     error: string | null;
+    // Mirrors the Python CriterionResult fields. `gating: false` (weight: 0) means
+    // the criterion is informational — measured, but excluded from the score and
+    // the pass/fail gate, so it must not render as PASS/FAIL. Both default the way
+    // pre-existing task.json files behave: gating, threshold 0.9.
+    passThreshold: number;
+    gating: boolean;
 }
 
 export interface ElementExecution {
@@ -1823,6 +1829,8 @@ export async function readTaskDetail(
             score?: number;
             details?: string;
             error?: string | null;
+            pass_threshold?: number;
+            gating?: boolean;
         }>;
         iterations?: TurnEntry[];
         environment_info?: RawRunJson["environment_info"];
@@ -1836,6 +1844,8 @@ export async function readTaskDetail(
         score: c.score ?? null,
         details: c.details ?? null,
         error: c.error ?? null,
+        passThreshold: c.pass_threshold ?? 0.9,
+        gating: c.gating ?? true,
     }));
 
     const artifactRoot = path.join(contentDir, "artifacts");

src/coder_eval/cli/evaluate_command.py

@@ -130,7 +130,12 @@ async def _setup_and_run() -> EvaluationResult:
         raise typer.Exit(1)
 
     for criterion, cr in zip(task.success_criteria, criteria_results, strict=True):
-        status = "[green]✓[/green]" if cr.score >= criterion.pass_threshold else "[red]✗[/red]"
+        if not criterion.is_gating:
+            # weight=0 is informational: it cannot pass/fail the task, so don't
+            # render it as ✓/✗ (that would contradict the gate and the exit code).
+            status = "[dim]○[/dim]"
+        else:
+            status = "[green]✓[/green]" if cr.score >= criterion.pass_threshold else "[red]✗[/red]"
         console.print(f"{status} {cr.criterion_type}")
         console.print(f"  [dim]{cr.description}[/dim]")
         console.print(f"  [dim]Score: {cr.score:.2f}[/dim]")
@@ -140,15 +145,19 @@ async def _setup_and_run() -> EvaluationResult:
             console.print(f"  [red]Error: {cr.error}[/red]")
         console.print()
 
-    passed = sum(
-        1 for cr, c in zip(criteria_results, task.success_criteria, strict=True) if cr.score >= c.pass_threshold
-    )
-    total = len(task.success_criteria)
+    # Gate over gating criteria only (weight=0 is informational and cannot fail
+    # the task) so this summary + the exit code below match final_status.
+    gating = [(cr, c) for cr, c in zip(criteria_results, task.success_criteria, strict=True) if c.is_gating]
+    passed = sum(1 for cr, c in gating if cr.score >= c.pass_threshold)
+    total = len(gating)
     failed = total - passed
+    informational = len(task.success_criteria) - total
 
     console.print("[bold]Summary:[/bold]")
     console.print(f"  Passed: {passed}/{total}")
     console.print(f"  Failed: {failed}/{total}")
+    if informational:
+        console.print(f"  [dim]Informational (weight=0, not gated): {informational}[/dim]")
     console.print(f"\n[dim]Run directory: {prepared_run_dir}[/dim]")
     if result.sandbox_path:
         console.print(f"[dim]Artifacts: {result.sandbox_path}[/dim]")

src/coder_eval/evaluation/checker.py

@@ -214,12 +214,16 @@ def _check_single(
                 context=context,
             )
             result.pass_threshold = criterion.pass_threshold
+            result.gating = criterion.is_gating
 
             logger.debug(f"Criterion '{criterion_type}' score: {result.score:.2f}")
             if result.score < criterion.pass_threshold:
+                # An informational (weight: 0) criterion cannot fail the task, so
+                # logging it as FAILED contradicts the final status. Say what it is.
                 logger.info(
-                    "Criterion '%s' FAILED (score=%.2f, threshold=%.2f): %s",
+                    "Criterion '%s' %s (score=%.2f, threshold=%.2f): %s",
                     criterion_type,
+                    "FAILED" if criterion.is_gating else "below threshold (informational)",
                     result.score,
                     criterion.pass_threshold,
                     _short_failure_reason(result),
@@ -237,6 +241,7 @@ def _check_single(
                 details=f"No checker registered for criterion type '{criterion_type}'",
                 error=f"Unsupported criterion type: '{criterion_type}'",
                 pass_threshold=criterion.pass_threshold,
+                gating=criterion.is_gating,
             )
         except JudgeInfrastructureError:
             raise  # judge infra failure escalates to FinalStatus.ERROR; do not score it
@@ -250,10 +255,12 @@ def _check_single(
                 details="Error running checker",
                 error=str(e),
                 pass_threshold=criterion.pass_threshold,
+                gating=criterion.is_gating,
             )
             logger.info(
-                "Criterion '%s' FAILED (score=0.00, threshold=%.2f): %s",
+                "Criterion '%s' %s (score=0.00, threshold=%.2f): %s",
                 criterion_type,
+                "FAILED" if criterion.is_gating else "errored (informational)",
                 criterion.pass_threshold,
                 _short_failure_reason(failed),
             )

src/coder_eval/models/criteria.py

@@ -9,7 +9,7 @@
 from __future__ import annotations
 
 from abc import ABC
-from typing import Annotated, Any, ClassVar, Literal
+from typing import Annotated, Any, ClassVar, Literal, Self
 
 from pydantic import BaseModel, ConfigDict, Field, model_validator
 
@@ -101,10 +101,11 @@ class BaseSuccessCriterion(BaseModel, ABC):
         ge=0.0,
         description=(
             "Relative importance of this criterion in the weighted score (default: 1.0). Set to 0 to "
-            "exclude it from the weighted SCORE -- useful for informational or side-effect checks "
-            "(e.g. a setup command). NOTE: weight=0 excludes from the score but NOT from the pass/fail "
-            "gate -- a criterion scoring below its pass_threshold still flips the task to FAILURE "
-            "regardless of weight. To make a criterion truly non-gating, also set pass_threshold=0."
+            "make the criterion purely INFORMATIONAL -- useful for side-effect checks (e.g. a setup "
+            "command): it is excluded from the weighted score AND from the pass/fail gate, so scoring "
+            "below its pass_threshold no longer flips the task to FAILURE. The result is still "
+            "computed, stored, and rendered in reports. A weight=0 criterion may not set stop_when "
+            "or suite_thresholds (arming a non-gating criterion for a pass/fail gate is incoherent)."
         ),
     )
 
@@ -145,6 +146,39 @@ def model_post_init(self, context: Any, /) -> None:
         # the discriminated union rejects the dump with union_tag_not_found.
         self.__pydantic_fields_set__.add("type")
 
+    @model_validator(mode="after")
+    def check_weight_zero_is_not_gating(self) -> Self:
+        """Reject ``weight: 0`` combined with any gate-arming field.
+
+        ``weight: 0`` makes a criterion informational — excluded from the score
+        and from the pass/fail gate (see ``is_gating``). Both ``stop_when`` (arms
+        the per-row early-stop gate) and ``suite_thresholds`` (arms the
+        across-row suite gate, which drives the run's exit code) would let an
+        "informational" criterion flip a run to failure — directly contradicting
+        the field's contract. So both combinations are authoring errors, caught
+        at load time rather than surfacing as a confusing exit code later.
+        """
+        if self.weight == 0.0:
+            for field, name in (("stop_when", "stop_when"), ("suite_thresholds", "suite_thresholds")):
+                if getattr(self, field) is not None:
+                    raise ValueError(
+                        f"criterion {self.type!r}: weight=0 makes the criterion informational (non-gating), "
+                        + f"so it cannot also set {name} (which arms it for a pass/fail gate). "
+                        + f"Give it a non-zero weight, or drop {name}."
+                    )
+        return self
+
+    @property
+    def is_gating(self) -> bool:
+        """True when this criterion participates in the task's pass/fail gate.
+
+        Single source of truth for "does a low score here fail the task". A
+        ``weight: 0`` criterion is informational: it is excluded from the
+        weighted score (it contributes 0 to both numerator and denominator) and,
+        by the same token, from the gate — so the two never disagree.
+        """
+        return self.weight > 0.0
+
     # Business logic (check operations) moved to SuccessChecker in evaluator.py